Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20954 )
Change subject: IMPALA-12742: DELETE/UPDATE Iceberg table partitioned by DATE fails with error ...................................................................... IMPALA-12742: DELETE/UPDATE Iceberg table partitioned by DATE fails with error Iceberg tables can be identity partitioned by any type, e.g. int, date and even float. If a table is partitioned, the file path contains the partition value in human readable form, and this form is expected to be passed to CatalogD. When an UPDATE or DELETE command is executed, we don't transform the integer date value to human readable format, which causes errors in CatalogD. With this patch, we transform identity-partitioned date values to human-readable format. Note on floating point numbers: When users partition their data via floating point values (users should not do that), then the file paths created for delete files might not correspond to the data files (e.g. '1.1' vs '1.100000023841858'). Though the values are the same in the Iceberg metadata layer, so it doesn't cause correctness issues. Testing: * added e2e tests for DELETEs * added e2e tests for UPDATEs Change-Id: I506f95527e741efe18c71706e2cdea51b45958b8 Reviewed-on: http://gerrit.cloudera.org:8080/20954 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/exec/iceberg-delete-sink-base.cc M be/src/exec/iceberg-delete-sink-base.h M be/src/exec/table-sink-base.cc M be/src/runtime/descriptors.cc M be/src/runtime/descriptors.h M common/thrift/CatalogObjects.thrift M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/catalog/ScalarType.java M fe/src/main/java/org/apache/impala/catalog/Type.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java M fe/src/test/java/org/apache/impala/util/IcebergUtilTest.java M testdata/workloads/functional-query/queries/QueryTest/iceberg-delete-partitioned.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-update-partitions.test 15 files changed, 160 insertions(+), 74 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/20954 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I506f95527e741efe18c71706e2cdea51b45958b8 Gerrit-Change-Number: 20954 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]> Gerrit-Reviewer: Gabor Kaszab <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
