wangmiao1981 commented on issue #475: TimestampWriter isn't being used URL: https://github.com/apache/incubator-iceberg/issues/475#issuecomment-607658305 @rdblue Just hit the same issue. In TypeToMessageType.java, https://github.com/apache/incubator-iceberg/blob/master/parquet/src/main/java/org/apache/iceberg/parquet/TypeToMessageType.java#L82, calls https://github.com/apache/incubator-iceberg/blob/master/parquet/src/main/java/org/apache/iceberg/parquet/TypeToMessageType.java#L133 It can't store the `shouldAdjustToUTC ` field. This is the reason it can't call the correct writer. So it fails in the `if` branch, https://github.com/apache/incubator-iceberg/blob/master/parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java#L212 There are few other existing tests pass, which go to https://github.com/apache/incubator-iceberg/blob/master/parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java#L235, which ignores the boolean flag and write it as Timestamp.withZone(). So it passes. In Spark parquet writer, it actually throws an exception: `java.lang.UnsupportedOperationException: Spark does not support timestamp without time zone field` If I understand correctly, it may not be fixed by using `TypeWithSchemaVisitor ` only. I will give it a try. I have made changes to throw similar exception as Spark does and comment out some tests. That was done before I found this opened issue. If it is not an easy fix, I will submit the exception PR. I also did a quick test on UUID and it also fails. I haven't debug that issue yet. I suspect similar issue of Logical Type support.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
