MaxGekk commented on issue #27953: [WIP][SPARK-31183][SQL] Rebase date/timestamp from/to Julian calendar in avro URL: https://github.com/apache/spark/pull/27953#issuecomment-601092719 @cloud-fan I cannot test the `timestamp_millis` saved by Spark 2.4.5 because it doesn't allow to save timestamps of the type, see: ```scala scala> :paste // Entering paste mode (ctrl-D to finish) val timestampSchema = s""" { "namespace": "logical", "type": "record", "name": "test", "fields": [ {"name": "ts", "type": {"type": "long","logicalType": "timestamp-millis"}} ] } """ // Exiting paste mode, now interpreting. timestampSchema: String = " { "namespace": "logical", "type": "record", "name": "test", "fields": [ {"name": "ts", "type": {"type": "long","logicalType": "timestamp-millis"}} ] } " scala> val df3 = Seq("1001-01-01 01:02:03.123456").toDF("tsS").select($"tsS".cast("timestamp").as("ts")) df3: org.apache.spark.sql.DataFrame = [ts: timestamp] scala> df3.write.format("avro").option("avroSchema", timestampSchema).save("/Users/maxim/tmp/before_1582/2_4_5_ts_millis_avro") 20/03/19 03:00:37 ERROR Utils: Aborting task org.apache.avro.AvroRuntimeException: Not a union: {"type":"long","logicalType":"timestamp-millis"} ``` The same works on the master.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
