[
https://issues.apache.org/jira/browse/SPARK-28955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17456539#comment-17456539
]
Bill Schneider commented on SPARK-28955:
----------------------------------------
I see a bunch of related tickets with fix version 3.3.0
Is this effectively going to be addressed by SPARK-35720
The goal being that if you see "2022-01-01 00:00" in a text file it can get
parsed as "2022-01-01 00:00" regardless of system timezone, and doesn't
accidentally end up "2021-12-31 17:00" for downstream consumers (e.g., Athena)
if your Spark job happened to run in EDT
> Support for LocalDateTime semantics
> -----------------------------------
>
> Key: SPARK-28955
> URL: https://issues.apache.org/jira/browse/SPARK-28955
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Affects Versions: 2.3.0
> Reporter: Bill Schneider
> Priority: Major
>
> It would be great if Spark supported local times in DataFrames, rather than
> only instants.
> The specific use case I have in mind is something like
> * parse "2019-01-01 17:00" (no timezone) from CSV -> LocalDateTime in
> dataframe
> * save to Parquet: LocalDateTime is stored with same integer value as
> 2019-01-01 17:00 UTC, but with isAdjustedToUTC=false. (Currently Spark saves
> either INT96 or TIME_MILLIS/TIME_MICROS which has isAdjustedToUTC=true)
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
