izchen opened a new pull request #3368: URL: https://github.com/apache/iceberg/pull/3368
Currently, directly register iceberg api `org.apache.iceberg.transforms.Bucket#apply` as spark UDF in `org.apache.iceberg.spark#IcebergSpark`. For byte, short, date, timestamp, and binary, Spark value of these types is different from iceberg‘s internal representation, which will cause a `ClassCastException` error. | SPARK TYPE | SPARK VALUE | ICEBERG TYPE | ICEBERG VALUE | | ------------- | ------------------ | ------------- | ------------------- | | ByteType | java.lang.Byte | IntegerType | java.lang.Integer | | ShortType | java.lang.Short | IntegerType | java.lang.Integer | | DateType | java.sql.Date | DateType | java.lang.Integer | | TimestampType | java.sql.Timestamp | TimestampType | java.lang.Long | | BinaryType | byte array | BinaryType | java.nio.ByteBuffer | We should first convert the spark value to iceberg's internal representation, and then use the converted value as the input of iceberg api `org.apache.iceberg.spark#IcebergSpark`. In addition, add more ut in this PR to cover all spark atom types. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
