[
https://issues.apache.org/jira/browse/IMPALA-15222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18115775#comment-18115775
]
ASF subversion and git services commented on IMPALA-15222:
----------------------------------------------------------
Commit f89b30507ab9ef5c85ba7ce4cf1f8d7f3f941536 in impala's branch
refs/heads/master from Noemi Pap-Takacs
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=f89b30507 ]
IMPALA-15222: Short-circuit OPTIMIZE when the table is compact
OPTIMIZE TABLE compacts the Iceberg tables by rewriting small files.
When given a file size threshold, it can automatically detect when
the table is already in a compact form. It flags the case with NO_OP
mode internally and does not create a new commit.
However, the executors still got the QueryExecRequest to scan the
table and rewrote the files, just silently left the newly written
files on the file system as orphan files.
This patch fixes this issue building on the NO_OP request handling
logic introduced by IMPALA-14536. When Impala detects that no
compaction is needed based on the given parameters, it converts the
Optimize statement to TStmtType.NO_OP and returns without creating
an execution plan with the following analysis result:
"No files selected for optimization."
This short-circuiting is enabled when
- the table is empty,
- there is a FILE_SIZE_THRESHOLD_MB set and there is at most 1
file per partition that fits the rewrite criteria. This means
compaction would not reduce file count.
Note that full-table OPTIMIZE always rewrites the entire table if
it contains any files, regardless of file size and file count per
partition to enable rewriting the table to the latest partition spec.
Testing:
- added e2e test for empty table and single file cases
- added Analysis and Planner test for table with 1 file.
Assisted-by: Claude Opus 4.8 (Claude Code)
Change-Id: I795967ca18231ef1fefdbc71a9b4505905906f95
Reviewed-on: http://gerrit.cloudera.org:8080/24633
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> OPTIMIZE can create orphan files
> --------------------------------
>
> Key: IMPALA-15222
> URL: https://issues.apache.org/jira/browse/IMPALA-15222
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Reporter: Noémi Pap-Takács
> Assignee: Noémi Pap-Takács
> Priority: Major
> Labels: impala-iceberg
>
> OPTIMIZE TABLE compacts the Iceberg tables by rewriting small files. When
> given a file size threshold, it can automatically detect when the table is
> already in a compact form. It flags the case with NO-OP mode internally and
> does not create a new commit. However, the executors still get the
> ExecRequest to scan the table and rewrite the files, and just silently leave
> the new files on the file system as orphan files.
> We should short-circuit OPTIMIZE operation on the Frontend when it is NO-OP.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
