[
https://issues.apache.org/jira/browse/SPARK-51753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-51753.
----------------------------------
Resolution: Invalid
Resolving as Invalid: this is a usage/how-to question rather than a report of a
specific Spark defect or a concrete change proposal. Apache Spark uses JIRA to
track bugs and improvements; usage questions are best asked on the
[email protected] mailing list (https://spark.apache.org/community.html) or
Stack Overflow (tag apache-spark), where a wider audience can help. Please
re-open with a concrete reproducer or a specific proposed code/doc change if
this is actually a bug or an actionable improvement. Thanks!
> Spark , Avro and Iceberg Timestamp , Timestamp NTZ definition need more
> clarifications
> --------------------------------------------------------------------------------------
>
> Key: SPARK-51753
> URL: https://issues.apache.org/jira/browse/SPARK-51753
> Project: Spark
> Issue Type: Question
> Components: Documentation, Java API, Spark Core
> Affects Versions: 3.5.5
> Reporter: Shekhar Prasad Rajak
> Priority: Major
>
> * There is no definition added in spark for tiemstamp datatype variants :
> [https://spark.apache.org/docs/3.5.0/sql-data-sources-avro.html#supported-types-for-spark-sql---avro-conversion]
> * Avro spec : [https://avro.apache.org/docs/1.12.0/specification/]
>
> {quote}
> {quote}
> {quote}{{Timestamps
> The timestamp-\{millis,micros,nanos} logical type represents an instant on
> the global timeline, independent of a particular time zone or calendar. Upon
> reading a value back, we can only reconstruct the instant, but not the
> original representation. In practice, such timestamps are typically displayed
> to users in their local time zones, therefore they may be displayed
> differently depending on the execution environment.
> timestamp-millis: logical type annotates an Avro long, where the long stores
> the number of milliseconds from the unix epoch, 1 January 1970 00:00:00.000.
> timestamp-micros: logical type annotates an Avro long, where the long stores
> the number of microseconds from the unix epoch, 1 January 1970
> 00:00:00.000000.
> timestamp-nanos: logical type annotates an Avro long, where the long stores
> the number of nanoseconds from the unix epoch, 1 January 1970
> 00:00:00.000000000.
> Example: Given an event at noon local time (12:00) on January 1, 2000, in
> Helsinki where the local time was two hours east of UTC (UTC+2). The
> timestamp is first shifted to UTC 2000-01-01T10:00:00 and that is then
> converted to Avro long 946720800000 (milliseconds) and written.
> Local Timestamps
> The local-timestamp-\{millis,micros,nanos} logical type represents a
> timestamp in a local timezone, regardless of what specific time zone is
> considered local.
> local-timestamp-millis: logical type annotates an Avro long, where the long
> stores the number of milliseconds, from 1 January 1970 00:00:00.000.
> local-timestamp-micros: logical type annotates an Avro long, where the long
> stores the number of microseconds, from 1 January 1970 00:00:00.000000.
> local-timestamp-nanos: logical type annotates an Avro long, where the long
> stores the number of nanoseconds, from 1 January 1970 00:00:00.000000000.
> Example: Given an event at noon local time (12:00) on January 1, 2000, in
> Helsinki where the local time was two hours east of UTC (UTC+2). The
> timestamp is converted to Avro long 946728000000 (milliseconds) and then
> written.}}
> {quote} * Iceberg Spec for all the timestamp types :
> [https://iceberg.apache.org/spec/#avro]
>
> {quote}
> {quote}
> {quote}{{timestamp | Timestamp, microsecond precision, without timezone
> timestamptz | Timestamp, microsecond precision, with timezone}}
> {quote}
> But spark 3 treat non timezone as TimestampNTZType but the avro serialiser
> will treat TimestampTZType as logicaltype *timestamp-millis* and
> TimestampNTZType as *local-timestamp-millis*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
