Noemi Pap-Takacs has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24633
Change subject: IMPALA-15222: Short-circuit OPTIMIZE when the table is compact ...................................................................... IMPALA-15222: Short-circuit OPTIMIZE when the table is compact OPTIMIZE TABLE compacts the Iceberg tables by rewriting small files. When given a file size threshold, it can automatically detect when the table is already in a compact form. It flags the case with NO_OP mode internally and does not create a new commit. However, the executors still got the QueryExecRequest to scan the table and rewrote the files, just silently left the newly written files on the file system as orphan files. This patch fixes this issue building on the NO_OP request handling logic introduced by IMPALA-14536. When Impala detects that no compaction is needed based on the given parameters, it converts the Optimize statement to TStmtType.NO_OP and returns without creating an execution plan with the following analysis result: "No files selected for optimization." This short-circuiting is enabled when - the table is empty, - there is a FILE_SIZE_THRESHOLD_MB set and there is at most 1 file per partition that fits the rewrite criteria. This means compaction would not reduce file count. Note that full-table OPTIMIZE always rewrites the entire table if it contains any files, regardless of file size and file count per partition to enable rewriting the table to the latest partition spec. Testing: - added e2e test for empty table - added Analysis and Planner test for table with 1 file. Assisted-by: Claude Opus 4.6 Change-Id: I795967ca18231ef1fefdbc71a9b4505905906f95 --- M be/src/service/client-request-state.cc M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/OptimizeStmt.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/IcebergOptimizeFileFilter.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M fe/src/test/java/org/apache/impala/util/IcebergFileFilterTest.java M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-optimize.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-optimize.test 10 files changed, 59 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/24633/1 -- To view, visit http://gerrit.cloudera.org:8080/24633 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I795967ca18231ef1fefdbc71a9b4505905906f95 Gerrit-Change-Number: 24633 Gerrit-PatchSet: 1 Gerrit-Owner: Noemi Pap-Takacs <[email protected]>
