Noemi Pap-Takacs has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/21388 )
Change subject: IMPALA-12867: Filter files to OPTIMIZE based on file size (WIP) ...................................................................... IMPALA-12867: Filter files to OPTIMIZE based on file size (WIP) OPTIMIZE TABLE statement is currently used to rewrite the entire Iceberg table. With 'FILE_SIZE_THRESHOLD' option, the user can specify a file size limit to rewrite only small files. Syntax: OPTIMIZE TABLE <table_name> [(FILE_SIZE_THRESHOLD_MB=100)]; The value of the threshold is the file size in MBs. Data files larger than the given limit will only be rewritten if they are referenced from delete deltas. If only 1 file is selected in a partition, it will not be rewritten. IMPALA-12839: 'Optimizing empty table should be no-op' is also resolved in this patch. Testing: - Parser test - FE unit tests - TODO: E2E tests Change-Id: Icfbb589513aacdb68a86c1aec4a0d39b12091820 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M common/thrift/Query.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/OptimizeStmt.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java A fe/src/main/java/org/apache/impala/util/IcebergFileFilter.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java A fe/src/test/java/org/apache/impala/util/IcebergFileFilterTest.java M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-optimize.test 17 files changed, 482 insertions(+), 25 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/21388/2 -- To view, visit http://gerrit.cloudera.org:8080/21388 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icfbb589513aacdb68a86c1aec4a0d39b12091820 Gerrit-Change-Number: 21388 Gerrit-PatchSet: 2 Gerrit-Owner: Noemi Pap-Takacs <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
