Suresh Thalamati created SPARK-13167:
----------------------------------------
Summary: JDBC data source does not include null value partition
columns rows in the result.
Key: SPARK-13167
URL: https://issues.apache.org/jira/browse/SPARK-13167
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 1.6.0, 2.0.0
Reporter: Suresh Thalamati
Reading from am JDBC data source using a partition column that is nullable can
return incorrect number of rows, if there are rows with null value for
partition column.
{code}
val emp =
sqlContext.read.jdbc("jdbc:h2:mem:testdb0;user=testUser;password=testPass",
"TEST.EMP", "theid", 0, 4, 3, new Properties)
emp.count()
{code}
Above jdbc read call sets up the partitions of the following form. It does not
include null predicate.
{code}
JDBCPartition(THEID < 1,0),JDBCPartition(THEID >= 1 AND THEID <
2,1),JDBCPartition(THEID >= 2,2)
{code}
Rows with null values in partition column are not included in the results
because the partition predicate does not specify is null predicates.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
