sshkvar commented on pull request #2282:
URL: https://github.com/apache/iceberg/pull/2282#issuecomment-853851066
Hi, I am working on write support together with @daniil-dubin
@bkahloon @RussellSpitzer in current implementation we have
`spark-handle-timestamp-without-timezone` flag, which can be used as read or
write(after implementation) option, like
```scala
spark
.read
.format("iceberg")
.option("spark-handle-timestamp-without-timezone", "true")
.load("some.table.name")
```
but we do not have ability to set this flag when using spark sql
```scala
spark.sql("select * from some.table.name") //<- will fail in current
implementation
```
Our proposal is move this flag to spark session property, @bkahloon
@RussellSpitzer what do you think about it?
This solution will also allow us to add additional check in
[TypeToSparkType.java](https://github.com/bkahloon/iceberg/blob/022dde6d4e56c373194a78a52e3f61df8d582026/spark/src/main/java/org/apache/iceberg/spark/TypeToSparkType.java#L112)
Something like it
```scala
case TIMESTAMP:
RuntimeConfig runtimeConfig = SparkSession.active().conf();
boolean canHandleTimestampWithoutZone =
SparkUtil.canHandleTimestampWithoutZone(runtimeConfig);
Types.TimestampType timestamp = (Types.TimestampType) primitive;
if (timestamp.shouldAdjustToUTC() || canHandleTimestampWithoutZone) {
return TimestampType$.MODULE$;
}
throw new
UnsupportedOperationException(SparkUtil.TIMESTAMP_WITHOUT_TIMEZONE_ERROR);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
