Noemi Pap-Takacs has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21388
Change subject: IMPALA-12867: Filter files to OPTIMIZE based on file size (WIP) ...................................................................... IMPALA-12867: Filter files to OPTIMIZE based on file size (WIP) OPTIMIZE TABLE statement is currently used to rewrite the entire Iceberg table. With 'FILE_SIZE_THRESHOLD' option, the user can specify a file size limit to rewrite only small files. Syntax: OPTIMIZE TABLE <table_name> [(FILE_SIZE_THRESHOLD_MB=100)]; The value of the threshold is the file size in MBs. Data files larger than the given limit will only be rewritten if they are referenced from delete deltas. If only 1 file is selected in a partition, it will not be rewritten. IMPALA-12839: 'Optimizing empty table should be no-op' is also resolved in this patch. Testing: - Parser test - FE unit tests - TODO: E2E tests Change-Id: Icfbb589513aacdb68a86c1aec4a0d39b12091820 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M common/thrift/Query.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/OptimizeStmt.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java A fe/src/main/java/org/apache/impala/util/IcebergFileFilter.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/ParserTest.java A fe/src/test/java/org/apache/impala/util/IcebergFileFilterTest.java M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-optimize.test 15 files changed, 454 insertions(+), 16 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/21388/1 -- To view, visit http://gerrit.cloudera.org:8080/21388 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Icfbb589513aacdb68a86c1aec4a0d39b12091820 Gerrit-Change-Number: 21388 Gerrit-PatchSet: 1 Gerrit-Owner: Noemi Pap-Takacs <[email protected]>
