Hello Gabor Kaszab, Noemi Pap-Takacs, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20954
to look at the new patch set (#2).
Change subject: IMPALA-12742: DELETE/UPDATE Iceberg table partitioned by DATE
fails with error
......................................................................
IMPALA-12742: DELETE/UPDATE Iceberg table partitioned by DATE fails with error
Iceberg tables can be identity partitioned by any type, e.g. int, date
and even float. If a table is partitioned, the file path contains the
partition value in human readable form, and this form is expected to
be passed to CatalogD. When an UPDATE or DELETE command is executed,
we don't transform the integer date value to human readable format,
which causes errors in CatalogD.
With this patch, we transform identity-partitioned date values to
human-readable format.
Note on floating point numbers:
When users partition their data via floating point values (users should
not do that), then the file paths created for delete files might not
correspond to the data files (e.g. '1.1' vs '1.100000023841858'). Though
the values are the same in the Iceberg metadata layer, so it doesn't
cause correctness issues.
Testing:
* added e2e tests for DELETEs
* added e2e tests for UPDATEs
Change-Id: I506f95527e741efe18c71706e2cdea51b45958b8
---
M be/src/exec/iceberg-delete-sink-base.cc
M be/src/exec/iceberg-delete-sink-base.h
M be/src/exec/table-sink-base.cc
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M common/thrift/CatalogObjects.thrift
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/ScalarType.java
M fe/src/main/java/org/apache/impala/catalog/Type.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java
M fe/src/test/java/org/apache/impala/util/IcebergUtilTest.java
M
testdata/workloads/functional-query/queries/QueryTest/iceberg-delete-partitioned.test
M
testdata/workloads/functional-query/queries/QueryTest/iceberg-update-partitions.test
15 files changed, 160 insertions(+), 74 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/20954/2
--
To view, visit http://gerrit.cloudera.org:8080/20954
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I506f95527e741efe18c71706e2cdea51b45958b8
Gerrit-Change-Number: 20954
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <[email protected]>
Gerrit-Reviewer: Gabor Kaszab <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]>
