[
https://issues.apache.org/jira/browse/IMPALA-12860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17823527#comment-17823527
]
ASF subversion and git services commented on IMPALA-12860:
----------------------------------------------------------
Commit 4428db37b3884373482071fc918936b0c080e47c in impala's branch
refs/heads/master from Zoltan Borok-Nagy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=4428db37b ]
IMPALA-12860: Invoke validateDataFilesExist for RowDelta operations
We must invoke validateDataFilesExist for RowDelta operations (DELETE/
UPDATE/MERGE). Without this a concurrent RewriteFiles (compaction) and
RowDelta can corrupt a table.
IcebergBufferedDeleteSink now also collects the filenames of the data
files that are referenced in the position delete files. It adds them to
the DML exec state which is then collected by the Coordinator. The
Coordinator passes the file paths to CatalogD which executes Iceberg's
RowDelta operation and now invokes validateDataFilesExist() with the
file paths. Additionally it also invokes validateDeletedFiles().
This patch set also resolves IMPALA-12640 which is about replacing
IcebergDeleteSink with IcebergBufferedDeleteSink, as from now on
we use the buffered version for all DML operations that write
position delete files.
Testing:
* adds new stress test with DELETE + UPDATE + OPTIMIZE
Change-Id: I4869eb863ff0afe8f691ccf2fd681a92d36b405c
Reviewed-on: http://gerrit.cloudera.org:8080/21099
Tested-by: Impala Public Jenkins <[email protected]>
Reviewed-by: Gabor Kaszab <[email protected]>
> Invoke validateDataFilesExist for RowDelta operations
> -----------------------------------------------------
>
> Key: IMPALA-12860
> URL: https://issues.apache.org/jira/browse/IMPALA-12860
> Project: IMPALA
> Issue Type: Bug
> Reporter: Zoltán Borók-Nagy
> Assignee: Zoltán Borók-Nagy
> Priority: Major
> Labels: impala-iceberg
>
> We must invoke validateDataFilesExist for RowDelta operations
> (DELETE/UPDATE/MERGE).
> Without this a concurrent RewriteFiles (compaction) and RowDelta can corrupt
> a table.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
