----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/19665/ -----------------------------------------------------------
Review request for drill. Repository: drill-git Description ------- This patch contains mainly the following changes: 1. Add a set of physical RelNode for the physical operators in Drill. 2. Add a set of new optiq rules to insert exchange operators into the physical plan. 3. A new RelTrait class to support the distribution property in Drill. 4. The existing Drill Logical Rel and the corresponding rules are modified. 5. Storage engine change. Diffs ----- common/src/main/antlr3/org/apache/drill/common/expression/parser/ExprLexer.g be2a3f2 common/src/main/java/org/apache/drill/common/logical/data/Order.java 3c864b0 common/src/main/java/org/apache/drill/common/logical/data/Scan.java c36c2d1 distribution/src/resources/sqlline ff83ab1 exec/java-exec/pom.xml bb044ad exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/HashFunctions.java 5a6943e exec/java-exec/src/main/java/org/apache/drill/exec/opt/BasicOptimizer.java 42de28b exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/AbstractBase.java f7a6f51 exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/AbstractSubScan.java c6e11d1 exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/GroupScan.java e51ebb6 exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/PhysicalOperator.java 6872595 exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/Filter.java aba1747 exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/HashToRandomExchange.java 9d5f3d6 exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/MergeJoinPOP.java fde88a9 exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/Project.java 7ab0620 exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/Screen.java 654d1f7 exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/SingleMergeExchange.java b8073c1 exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/Sort.java 82aa830 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/StreamingAggBatch.java fccdbd6 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/mergereceiver/MergingRecordBatch.java d6d42b6 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/orderedpartitioner/OrderedPartitionRecordBatch.java 4eee6af exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/sort/SortBatch.java e2ae823 exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillAggregateRelBase.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillFilterRelBase.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillJoinRelBase.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillLimitRelBase.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillProjectRelBase.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillRelNode.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillScanRelBase.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillScreenRelBase.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillStoreRelBase.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillUnionRelBase.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillAggregateRel.java e1337ef exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillAggregateRule.java 77f1ba6 exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillFilterRel.java cc147e3 exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillFilterRule.java d4fb239 exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillJoinRel.java 16e9d47 exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillJoinRule.java f32fa59 exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillLimitRel.java 54be052 exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillLimitRule.java 85c594e exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillOptiq.java 7398d78 exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillProjectRel.java ee3a0f4 exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillProjectRule.java bf5bcff exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillRel.java 63a7207 exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillRuleSets.java 53da67f exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillScanRel.java afc2d1b exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillScanRule.java 58e648a exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillScreenRel.java 829947a exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillSortRule.java c968e85 exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillStoreRel.java 30c7810 exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillTable.java bad6f74 exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillUnionRel.java 1be9caf exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillUnionRule.java bc1e6f4 exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillValuesRel.java e770181 exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillValuesRule.java fae18ca exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DynamicDrillTable.java 36b554f exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/EnumerableDrillRule.java 2fcf660 exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/RelDataTypeHolder.java 038f3de exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/DrillDistributionTrait.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/DrillDistributionTraitDef.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/FilterPrel.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/FilterPrule.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/HashToRandomExchangePrel.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/LimitPrel.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/LimitPrule.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/MergeJoinPrel.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/MergeJoinPrule.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/OrderedPartitionExchangePrel.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/PhysicalPlanCreator.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/Prel.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/PrelUtil.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ProjectPrel.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ProjectPrule.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ScanPrel.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ScanPrule.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ScreenPrel.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ScreenPrule.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/SingleMergeExchangePrel.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/SortConvertPrule.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/SortPrel.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/SortPrule.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/StreamAggPrel.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/StreamAggPrule.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/UnionExchangePrel.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java bdbb642 exec/java-exec/src/main/java/org/apache/drill/exec/planner/torel/ConversionContext.java d4aabb4 exec/java-exec/src/main/java/org/apache/drill/exec/schema/Field.java 0db3221 exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractStoragePlugin.java 8baa72a exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePlugin.java 2e54b0d exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java 3762c1a exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FormatPlugin.java a37142e exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/WorkspaceSchemaFactory.java c69edb7 exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyFormatPlugin.java d7949c3 exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyGroupScan.java a7f556e exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasySubScan.java 72d1fe6 exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/shim/fallback/FallbackFileSystem.java 5743ca1 exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/JSONFormatPlugin.java 82bf3bf exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/JSONRecordReader.java 66c5ccf exec/java-exec/src/main/java/org/apache/drill/exec/store/hive/HiveStoragePlugin.java a1d575c exec/java-exec/src/main/java/org/apache/drill/exec/store/hive/schema/DrillHiveTable.java abec2c5 exec/java-exec/src/main/java/org/apache/drill/exec/store/hive/schema/HiveSchemaFactory.java 091381f exec/java-exec/src/main/java/org/apache/drill/exec/store/ischema/InfoSchemaDrillTable.java 20609b8 exec/java-exec/src/main/java/org/apache/drill/exec/store/ischema/InfoSchemaStoragePlugin.java 891419d exec/java-exec/src/main/java/org/apache/drill/exec/store/mock/MockStorageEngine.java 0578b06 exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetFormatPlugin.java bfaaa45 exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetGroupScan.java f76e59a exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordReader.java 9e1cc66 exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRowGroupScan.java 0e672d0 exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetScanBatchCreator.java 17e7da2 exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java 08cb599 exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/QueryManager.java eaf921d exec/java-exec/src/test/java/org/apache/drill/TestExampleQueries.java be51276 exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/mergereceiver/MyJUnitTest.java PRE-CREATION exec/java-exec/src/test/java/org/apache/drill/exec/planner/physical/TestPhysicalPlanning.java PRE-CREATION exec/java-exec/src/test/java/org/apache/drill/exec/store/json/JSONRecordReaderTest.java e812997 exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/ParquetRecordReaderTest.java f6a8aa4 exec/java-exec/src/test/resources/storage-engines.json 6e4d23e pom.xml c4bc9c9 sample-data/nationsMF/nation.parquet PRE-CREATION sample-data/nationsMF/nation2.parquet PRE-CREATION sample-data/nationsSF/nation.parquet PRE-CREATION sample-data/regionsMF/region.parquet PRE-CREATION sample-data/regionsMF/region2.parquet PRE-CREATION sample-data/regionsSF/region.parquet PRE-CREATION sqlparser/src/main/java/org/apache/drill/jdbc/ConnectionConfig.java 5a81035 sqlparser/src/main/java/org/apache/drill/jdbc/DrillConnection.java ab1259a sqlparser/src/main/java/org/apache/drill/jdbc/DrillConnectionConfig.java PRE-CREATION sqlparser/src/main/java/org/apache/drill/jdbc/DrillConnectionImpl.java 11a1157 sqlparser/src/test/java/org/apache/drill/jdbc/test/JdbcTest.java 8cd682b sqlparser/src/test/java/org/apache/drill/jdbc/test/MyFullEngineTest.java PRE-CREATION sqlparser/src/test/java/org/apache/drill/jdbc/test/TestJdbcDistQuery.java PRE-CREATION sqlparser/src/test/java/org/apache/drill/jdbc/test/TestJdbcQuery.java 21de9f4 Diff: https://reviews.apache.org/r/19665/diff/ Testing ------- A new testJdbcDistQuery unit testcase is added. The distributed physical plan for each query could be seen in the console or in the log. With the exchange operators inserted, some queries expose memory leak issues, which will be addressed in a different JIRA. For those impacted queries, temporarily disable the unit tests. Thanks, Jinfeng Ni
