Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21935#discussion_r207088878
--- Diff:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroOptions.scala ---
@@ -79,4 +80,16 @@ class AvroOptions(
val compression: String = {
parameters.get("compression").getOrElse(SQLConf.get.avroCompressionCodec)
}
+
+ /**
+ * The `outputTimestampType` option sets which Avro timestamp type to
use when Spark writes
+ * data to Avro files. Currently supported types are `TIMESTAMP_MICROS`
and `TIMESTAMP_MILLIS`.
+ * TIMESTAMP_MICROS is a logical timestamp type in Avro, which stores
number of microseconds
+ * from the Unix epoch. TIMESTAMP_MILLIS is also logical, but with
millisecond precision,
+ * which means Spark has to truncate the microsecond portion of its
timestamp value.
+ */
+ val outputTimestampType: AvroOutputTimestampType.Value = {
--- End diff --
Hm, I wouldn't expose this as an option for now - that at least matches to
Parquet's.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
