Zoltán Borók-Nagy created IMPALA-11331:
------------------------------------------
Summary: Create Iceberg transactions earlier
Key: IMPALA-11331
URL: https://issues.apache.org/jira/browse/IMPALA-11331
Project: IMPALA
Issue Type: Bug
Components: Frontend
Reporter: Zoltán Borók-Nagy
Currently we create Iceberg transactions via
IcebergUtil.getIcebergTransaction() in CatalogOpExecutor.
This is problematic in some cases, especially for INSERT OVERWRITEs, because in
this case we open the transaction too late, when the data files are already
written, then we open the transaction and commit it. INSERT statements in the
meantime get overwritten instead of failing the INSERT OVERWRITE operation.
This can be problematic when we try to use INSERT OVERWRITE for compacting a
table. In that case we definitely don't want to loose INSERTed data.
Moving transaction open/close to the coordinator requires a lot of work, and
the handling of self-events would become even more complicated.
Alternatively, we could initiate an open transaction from the Coordinator, i.e.
asking CatalogD to open one, then at the end CatalogD would commit the opened
transaction.
We also need to abort transactions of failed queries. We also need ways of
aborting transactions of crashed Coordinators.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
