vinodkc commented on PR #53010:
URL: https://github.com/apache/spark/pull/53010#issuecomment-3524573710
@pan3793 ,
> does it work properly with spark.sql.datetime.java8API.enabled
enabled/disabled? IIRC, STS forcibly enables it, but the Connect Client may
have a different situation.
I think, the Spark Connect JDBC client is independent of the java8API
configuration.It always return java.sql.Date . The java8API setting only
affects regular DataFrame operations (whether Row.getAs[Date] returns
java.time.LocalDate vs java.sql.Date)
The Connect JDBC client always calls the Row API in a way that returns
java.sql.Date
> does it work properly with days after 9999-12-31? e.g. 10000-01-01
No, I see this output from spark connect based Beeline
```
0: jdbc:sc://localhost:15002> SELECT date '10000-01-01';
+----------------------+
| DATE '+10000-01-01' |
+----------------------+
| 0000-01-01 |
+----------------------+
```
This is because java.sql.Date.toString() fails for years > 9999 (returns
wrong strings like "0000-01-01" instead of "10000-01-01"). This is a limitation
of `java.sql.Date.toString()`, not Spark Connect. In `java.sql.Date`, he
underlying milliseconds value is preserved correctly. I can add a follow up PR
with additional testcases like below
```
test("get date type beyond year 9999") {
// Test dates after 9999-12-31 (e.g., 10000-01-01)
// Spark stores dates as days since epoch (Int), which can represent
dates far beyond 9999
// java.sql.Date can store the value, but toString() doesn't format
years > 9999 correctly
// due to SimpleDateFormat limitations. We verify correctness using
Calendar instead.
withExecuteQuery("SELECT date '10000-01-01'") { rs =>
assert(rs.next())
val date = rs.getDate(1)
assert(date !== null)
assert(!rs.wasNull)
// Verify date components using Calendar (toString() doesn't work for
years > 9999)
val calendar = Calendar.getInstance()
calendar.setTime(date)
assert(calendar.get(Calendar.YEAR) === 10000)
assert(calendar.get(Calendar.MONTH) === Calendar.JANUARY)
assert(calendar.get(Calendar.DAY_OF_MONTH) === 1)
assert(!rs.next())
}
}
```
> BTW, would you like to add TIMESTAMP/TIMESTAMP_NTZ support? Since we want
to include a few more features in 4.1 (feature freeze is planned in Nov. 15),
we can prioritize the feature implementation and defer some closer reviewing
and bug fixing to the 4.1 QA period.
Sure, I'll raise a PR for TIMESTAMP.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
