rdblue commented on a change in pull request #3132:
URL: https://github.com/apache/iceberg/pull/3132#discussion_r725394379
##########
File path: spark/src/main/java/org/apache/iceberg/spark/SparkUtil.java
##########
@@ -135,43 +130,16 @@ public static boolean hasTimestampWithoutZone(Schema
schema) {
}
/**
- * Allow reading/writing timestamp without time zone as timestamp with time
zone. Generally,
- * this is not safe as timestamp without time zone is supposed to represent
wall clock time semantics,
- * i.e. no matter the reader/writer timezone 3PM should always be read as
3PM,
- * but timestamp with time zone represents instant semantics, i.e the
timestamp
- * is adjusted so that the corresponding time in the reader timezone is
displayed.
- * When set to false (default), we throw an exception at runtime
- * "Spark does not support timestamp without time zone fields" if reading
timestamp without time zone fields
+ * Checks whether timestamp types for new tables should be stored with
timezone info.
+ * <p>
+ * The default value is false and all timestamp fields are stored as {@link
Types.TimestampType#withZone()}.
+ * If enabled, all timestamp fields in new tables will be stored as {@link
Types.TimestampType#withoutZone()}.
*
- * @param readerConfig table read options
- * @param sessionConf spark session configurations
- * @return boolean indicating if reading timestamps without timezone is
allowed
- */
- public static boolean canHandleTimestampWithoutZone(Map<String, String>
readerConfig, RuntimeConfig sessionConf) {
- String readerOption = readerConfig.get(HANDLE_TIMESTAMP_WITHOUT_TIMEZONE);
- if (readerOption != null) {
- return Boolean.parseBoolean(readerOption);
- }
- String sessionConfValue =
sessionConf.get(HANDLE_TIMESTAMP_WITHOUT_TIMEZONE, null);
- if (sessionConfValue != null) {
- return Boolean.parseBoolean(sessionConfValue);
- }
- return false;
- }
-
- /**
- * Check whether the spark session config contains a {@link
SparkUtil#USE_TIMESTAMP_WITHOUT_TIME_ZONE_IN_NEW_TABLES}
- * property.
- * Default value - false
- * If true in new table all timestamp fields will be stored as {@link
Types.TimestampType#withoutZone()},
- * otherwise {@link Types.TimestampType#withZone()} will be used
- *
- * @param sessionConf a spark runtime config
- * @return true if the session config has {@link
SparkUtil#USE_TIMESTAMP_WITHOUT_TIME_ZONE_IN_NEW_TABLES} property
- * and this property is set to true
+ * @param sessionConf a Spark runtime config
+ * @return true if timestamp types for new tables should be stored with
timezone info
*/
public static boolean useTimestampWithoutZoneInNewTables(RuntimeConfig
sessionConf) {
- String sessionConfValue =
sessionConf.get(USE_TIMESTAMP_WITHOUT_TIME_ZONE_IN_NEW_TABLES, null);
+ String sessionConfValue =
sessionConf.get(SparkSQLConfigs.USE_TIMESTAMP_WITHOUT_TIME_ZONE_IN_NEW_TABLES,
null);
Review comment:
Any plan to make `SparkConfParser` handle these one-off cases?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
