sadikovi commented on pull request #34596: URL: https://github.com/apache/spark/pull/34596#issuecomment-970897755
I would like to discuss how to handle the following case: - User writes TIMESTAMP_NTZ value in CSV. - User set `spark.sql.legacy.timeParserPolicy` to "legacy" This case is ambiguous: default timestamp pattern for legacy requires timezone but the value does not have any. However, the code works just fine when the pattern does not have timezone. In that case we would write the value successfully. To resolve ambiguity, I identified 2 options: - throw an exception saying that TIMESTAMP_NTZ values are not supported in "legacy" time parsing, regardless of whether timestamp format has the timezone component or not. - Write timestamp value with the default timezone, basically converting `LocalDateTime` into `java.sql.Timestamp` (currently implemented in the PR). @gengliangwang Do you have any preferences here? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
