Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/11063#discussion_r53751420
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala
---
@@ -75,12 +83,63 @@ private[sql] object JDBCRelation {
}
ans.toArray
}
+
+ /**
+ * Returns null value predicate if the column is nullable, otherwise
empty string. Uses
+ * nullable information available in the catalyst schema generated for
the source table.
+ * This method avoids querying the database metadata to find if the
unquoted column names
+ * are interpreted as uppercase or lower case. In rare cases when the
schema has on column
+ * with uppercase name, and another one with lowercase, it will return
null value
+ * partition predicate if either one of the column is nullable.
+ *
+ * @param column name of the column
+ * @param url the jdbc url of the database
+ * @param schema table's Catalyst schema
+ * @return null value predicate or empty string.
+ */
+ private def getParitionNullValuePredicate(column: String,
+ url: String, schema: StructType): String = {
--- End diff --
also the logic in this function is fairly convoluted. can't we always add
the null values to the first partition?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
