GitHub user maropu opened a pull request:
https://github.com/apache/spark/pull/21834
[SPARK-22814][SQL] Support Date/Timestamp in a JDBC partition column
## What changes were proposed in this pull request?
This pr supported Date/Timestamp in a JDBC partition column (a numeric
column is only supported in the master). This pr also modified code to verify a
partition column type;
```
val jdbcTable = spark.read
.option("partitionColumn", "text")
.option("lowerBound", "aaa")
.option("upperBound", "zzz")
.option("numPartitions", 2)
.jdbc("jdbc:postgresql:postgres", "t", options)
// with this pr
org.apache.spark.sql.AnalysisException: Partition column type should be
numeric, date, or timestamp, but string found.;
at
org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation$.verifyAndGetNormalizedPartitionColumn(JDBCRelation.scala:165)
at
org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation$.columnPartition(JDBCRelation.scala:85)
at
org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:36)
at
org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:317)
// without this pr
java.lang.NumberFormatException: For input string: "aaa"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:589)
at java.lang.Long.parseLong(Long.java:631)
at
scala.collection.immutable.StringLike$class.toLong(StringLike.scala:277)
```
Closes #19999
## How was this patch tested?
Added tests in `JDBCSuite`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/maropu/spark SPARK-22814
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21834.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21834
----
commit 87f9be3059f3e15b3e9c8769e9b39a06976ca50c
Author: Takeshi Yamamuro <yamamuro@...>
Date: 2018-07-21T07:47:43Z
Fix
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
