Steve Carlin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24309 )
Change subject: IMPALA-14999: Calcite planner: support Iceberg tables (part 1) ...................................................................... IMPALA-14999: Calcite planner: support Iceberg tables (part 1) This commit is part 1 of changes needed to support querying Iceberg table through the Calcite planner. The change in this commit is a refactor to support count star optimization. The Calcite planner populates the HdfsScanNode.countStarSlot_ differently from the original planner. The original planner does analysis through the physical AggregationNode and HdfsScanNode at "init" time based on the columns and conjuncts present in these nodes. The Calcite planner does this analysis before the physical nodes are created. The TupleDescriptor is already created before the PlanNode.init() method is called. The Calcite planner determines the SlotDescriptor for the countStarSlot_ before the PlanNodes are instantiated, so it simply populates the variable, avoiding the PlanNode.init code. The code before this commit places this variable in the extended ImpalaHdfsScanNode class. However, this does not work for the Iceberg planner since it uses its own class for IcebergScanNode extended off of HdfsScanNode. Creating an ImpalaIcebergScanNode class seemed too hacky, so this commit contains refactored code which will allow the Calcite planner to populate the countStarSlot_. The ScanNodeHelper interface has been created to handle the different ways the countStarSlot_ gets populated. The implementation for the original planner is used by KuduScanNode, HdfsScanNode, and IcebergScanNode which calls the ScanNodeHelperImpl.getCountStarOptimizationDescriptor method. The whole block of code containing "canApply..." and "apply..." has been moved. Kudu uses a slightly different conjuncts variable so it is passed into the common method. For simplification, a small change was made to HdfsScanNode. The fileFormats was passed into the "canApply" method. Instead of passing it in, the "canApply..." now matches the parent "ScanNode.canApply..." method, uses One of the ScanNode.apply... methods was unused and removed. Other planners that extend the ScanNode also pass in the default original ScanNodeHelperImpl class, but it is not used for these classes. The one important change for the IcebergScanNode creation is that the ScanNodeHelper class is instantiated from SingleNodePlanner and passed into the IcebergScanPlanner. When the Calcite planner code gets implemented, it will also call the IcebergScanPlanner, but with its own implementation of ScanNodeHelper. Testing: This is just a refactor, so no new tests are needed. Change-Id: Iba7260194e14d8aeb2df9fe3cb875a0ee8450fb1 Reviewed-on: http://gerrit.cloudera.org:8080/24309 Reviewed-by: Zoltan Borok-Nagy <[email protected]> Reviewed-by: Peter Rozsa <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> Reviewed-by: Steve Carlin <[email protected]> --- M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/IcebergMetadataScanNode.java M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java M fe/src/main/java/org/apache/impala/planner/PaimonScanNode.java M fe/src/main/java/org/apache/impala/planner/ScanNode.java A fe/src/main/java/org/apache/impala/planner/ScanNodeHelper.java A fe/src/main/java/org/apache/impala/planner/ScanNodeHelperImpl.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/planner/SystemTableScanNode.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaHdfsScanRel.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/phys/ImpalaHdfsScanNode.java 15 files changed, 172 insertions(+), 57 deletions(-) Approvals: Zoltan Borok-Nagy: Looks good to me, but someone else must approve Peter Rozsa: Looks good to me, but someone else must approve Impala Public Jenkins: Verified Steve Carlin: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/24309 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iba7260194e14d8aeb2df9fe3cb875a0ee8450fb1 Gerrit-Change-Number: 24309 Gerrit-PatchSet: 9 Gerrit-Owner: Steve Carlin <[email protected]> Gerrit-Reviewer: Aman Sinha <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]> Gerrit-Reviewer: Peter Rozsa <[email protected]> Gerrit-Reviewer: Steve Carlin <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
