RussellSpitzer commented on a change in pull request #2282:
URL: https://github.com/apache/iceberg/pull/2282#discussion_r584890347
##########
File path: spark2/src/main/java/org/apache/iceberg/spark/source/Reader.java
##########
@@ -170,6 +174,13 @@
this.batchSize =
options.get(SparkReadOptions.VECTORIZATION_BATCH_SIZE).map(Integer::parseInt).orElseGet(()
->
PropertyUtil.propertyAsInt(table.properties(),
TableProperties.PARQUET_BATCH_SIZE,
TableProperties.PARQUET_BATCH_SIZE_DEFAULT));
+ // Allow reading timestamp without time zone as timestamp with time zone.
Generally, this is not safe as timestamp
+ // without time zone is supposed to represent wall clock time semantics,
i.e. no matter the reader/writer timezone
+ // 3PM should always be read as 3PM, but timestamp with time zone
represents instant semantics, i.e the timestamp
+ // is adjusted so that the corresponding time in the reader timezone is
displayed.
+ // When set to false (default), we throw an exception at runtime
+ // "Spark does not support timestamp without time zone fields" if reading
timestamp without time zone fields
+ this.readTimestampWithoutZone =
options.get("read-timestamp-without-zone").map(Boolean::parseBoolean).orElse(false);
Review comment:
Maybe i'm being silly, but I would like "spark" to be in this flag like
"spark handle timestamp without zone" and have it handle reading and writing
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
