Noemi Pap-Takacs created IMPALA-13501:
-----------------------------------------
Summary: Conflicting commits to Iceberg tables leave uncommitted
orphan files
Key: IMPALA-13501
URL: https://issues.apache.org/jira/browse/IMPALA-13501
Project: IMPALA
Issue Type: Improvement
Components: Catalog
Reporter: Noemi Pap-Takacs
Iceberg supports multiple writers with optimistic concurrency. Each writer can
write new files which are then added to the table after a validation check to
ensure that the commit does not conflict with other modifications made during
the execution.
When there was a conflicting change and the newly written files cannot be
committed, there are 2 ways to proceed: the commit can be retried and rebased
on top of the latest snapshot. If this cannot resolve the conflict, the change
cannot be committed and the files become orphan files in the file system.
It would be nice to remove the remaining files from an unsuccessful commit in
one step. Deleting orphan files later as a table maintenance step is also a
possible resolution.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
