[
https://issues.apache.org/jira/browse/IMPALA-15065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18088390#comment-18088390
]
ASF subversion and git services commented on IMPALA-15065:
----------------------------------------------------------
Commit 548df5824879fea3ec7476437fd9df1e7ca9a8d5 in impala's branch
refs/heads/master from Peter Rozsa
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=548df5824 ]
IMPALA-15065: Fix incorrect partition spec identification when removing
deletion vectors from Iceberg V3 tables
When Impala removes an old deletion vector via rowDelta.removeDeletes(),
it previously passed a DeleteFile reconstructed from the C++ flatbuffer
using FileMetadata.Builder. For tables whose partition spec is a void
transform (partitioned-to-unpartitioned evolution), FileMetadata.Builder
produces partition()=null. The manifest-resident entry for the same DV
was written with a non-null (empty) PartitionData. Iceberg's
ManifestFilterManager.canContainAny() then fails to match the manifest.
This change uses direct manifest entry lookup to locate the old deletion
vectors based on the referenced data file path, it guarantees that the
DeleteFile is removable in the row delta.
Tests:
- added regression test to iceberg-v3-merge.test
Change-Id: Iba280972e2089e92771c8b3f7355d916d46ad968
Reviewed-on: http://gerrit.cloudera.org:8080/24394
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Incorrect partition identification when removing deletion vectors from
> void-partitioned Iceberg tables
> -------------------------------------------------------------------------------------------------------
>
> Key: IMPALA-15065
> URL: https://issues.apache.org/jira/browse/IMPALA-15065
> Project: IMPALA
> Issue Type: Sub-task
> Reporter: Peter Rozsa
> Assignee: Peter Rozsa
> Priority: Major
> Labels: impala-iceberg
>
> When Impala removes an old DV via rowDelta.removeDeletes(), it passes a
> DeleteFile reconstructed from the C++ flatbuffer using FileMetadata.Builder.
> For tables whose partition spec is just a void transform (partitioned to
> unpartitioned evolution), FileMetadata.Builder produces partition() = null.
> The manifest-resident entry for the same DV was written with a non-null
> (empty) PartitionData. Iceberg's ManifestFilterManager.canContainAny() then
> fails to match the manifest.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
