cloud-fan commented on a change in pull request #30902:
URL: https://github.com/apache/spark/pull/30902#discussion_r561601546
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala
##########
@@ -408,6 +421,23 @@ object JdbcUtils extends Logging {
(rs: ResultSet, row: InternalRow, pos: Int) =>
row.setFloat(pos, rs.getFloat(pos + 1))
+
+ // SPARK-33888 - sql TIME type represents as physical int in millis
+ // Represents a time of day, with no reference to a particular calendar,
+ // time zone or date, with a precision of one millisecond.
+ // It stores the number of milliseconds after midnight, 00:00:00.000.
+ case IntegerType if metadata.contains("logical_time_type") =>
+ (rs: ResultSet, row: InternalRow, pos: Int) => {
+ val rawTime = rs.getTime(pos + 1)
Review comment:
@sarutak do you mean what returns here is seconds (with certain
precision) from a starting timestamp, while the timestamp is different between
databases? I'm a bit surprised if the JDBC protocol was design this way, but if
this is true, then this PR doesn't make sense...
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala
##########
@@ -408,6 +421,23 @@ object JdbcUtils extends Logging {
(rs: ResultSet, row: InternalRow, pos: Int) =>
row.setFloat(pos, rs.getFloat(pos + 1))
+
+ // SPARK-33888 - sql TIME type represents as physical int in millis
+ // Represents a time of day, with no reference to a particular calendar,
+ // time zone or date, with a precision of one millisecond.
+ // It stores the number of milliseconds after midnight, 00:00:00.000.
+ case IntegerType if metadata.contains("logical_time_type") =>
+ (rs: ResultSet, row: InternalRow, pos: Int) => {
+ val rawTime = rs.getTime(pos + 1)
Review comment:
What returns here is `java.sql.Time`, and its doc says
```
The date components should be set to the "zero epoch"
value of January 1, 1970 and should not be accessed.
```
Maybe some databases don't follow the requirement, but it doesn't matter, as
we call `rawTime.toLocalTime` which only access the hour:minute:second
components.
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala
##########
@@ -408,6 +421,23 @@ object JdbcUtils extends Logging {
(rs: ResultSet, row: InternalRow, pos: Int) =>
row.setFloat(pos, rs.getFloat(pos + 1))
+
+ // SPARK-33888 - sql TIME type represents as physical int in millis
+ // Represents a time of day, with no reference to a particular calendar,
+ // time zone or date, with a precision of one millisecond.
+ // It stores the number of milliseconds after midnight, 00:00:00.000.
+ case IntegerType if metadata.contains("logical_time_type") =>
+ (rs: ResultSet, row: InternalRow, pos: Int) => {
+ val rawTime = rs.getTime(pos + 1)
Review comment:
That said, I think reading SQL TIME type as integer has a well-define
semantic in Spark (after this PR): the integer represents the milliseconds of
the time from 00:00:00.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
