sadikovi commented on PR #36726: URL: https://github.com/apache/spark/pull/36726#issuecomment-1154747718
I think we can. JDBC dialects can configure how they map TimestampNTZ type. In the case you mentioned, both timestamps will be read as timestamp_ntz in MySQL and Postgres. In fact, the current timestamp type is stored as timestamp_ntz in those database systems. Even with dialects managing timestamp_ntz writes and reads, this would be the same problem unless you store them as different types. Also, the test passes in master: ``` [ivan.sadikov@C02DV1TGMD6R spark-oss (master)]$ git log -n1 commit 2349175e1b81b0a61e1ed90c2d051c01cf78de9b (HEAD -> master, upstream/master) Author: Ivan Sadikov <[email protected]> Date: Mon Jun 13 21:22:15 2022 -0700 [SPARK-39339][SQL] Support TimestampNTZ type in JDBC data source [info] JDBCSuite: 05:53:04.233 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 05:53:07.936 ERROR org.apache.spark.sql.execution.datasources.jdbc.connection.ConnectionProvider: Failed to load built-in provider. [info] - SPARK-39339: Handle TimestampNTZType null values (1 second, 555 milliseconds) [info] - SPARK-39339: TimestampNTZType with different local time zones (4 seconds, 48 milliseconds) 05:53:14.022 WARN org.apache.spark.sql.jdbc.JDBCSuite: ===== POSSIBLE THREAD LEAK IN SUITE o.a.s.sql.jdbc.JDBCSuite, threads: Timer-2 (daemon=true), rpc-boss-3-1 (daemon=true), shuffle-boss-6-1 (daemon=true) ===== [info] Run completed in 11 seconds, 724 milliseconds. [info] Total number of tests run: 2 [info] Suites: completed 1, aborted 0 [info] Tests: succeeded 2, failed 0, canceled 0, ignored 0, pending 0 [info] All tests passed. [success] Total time: 81 s (01:21), completed Jun 14, 2022 5:53:14 AM ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
