[ https://issues.apache.org/jira/browse/DRILL-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973317#comment-14973317 ]
ASF GitHub Bot commented on DRILL-3975: --------------------------------------- GitHub user jacques-n opened a pull request: https://github.com/apache/drill/pull/217 DRILL-3975: Make sure to strip scheme and authority from partition lo… …cation. I didn't add tests as I've been unable to reproduce locally as it seems to require a full HDFS cluster. The extended test suite identifies this problem when run against HDFS so that should provide a sufficient test (~150 tests fails without this fix). @StevenMPhillips, can you take a look. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jacques-n/drill DRILL-3975 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/217.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #217 ---- commit f97247999931ea2a01e927318dd903f36358f413 Author: Jacques Nadeau <jacq...@apache.org> Date: 2015-10-25T15:59:39Z DRILL-3975: Make sure to strip scheme and authority from partition location. ---- > Partition Planning rule causes query failure due to IndexOutOfBoundsException > on HDFS > ------------------------------------------------------------------------------------- > > Key: DRILL-3975 > URL: https://issues.apache.org/jira/browse/DRILL-3975 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning & Optimization > Reporter: Jacques Nadeau > > In attempting to run the extended test suite provided by MapR, there are a > large number of queries that fail due to issues in the PruneScanRule and > specifically the DFSPartitionLocation constructor line 31. It is likely due > to issues with the code that are related to running on HDFS where this code > path has apparently not been tested. > An example test query this type of failure occurred: > /src/drill-test-framework/resources/Functional/ctas/ctas_auto_partition/tpch0.01_multiple_partitions/data/q11.q > Example stack trace below: > {code} > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: > StringIndexOutOfBoundsException: String index out of range: -12 > [Error Id: f2941267-49b1-4f67-a17f-610ffb13fcb7 on > ip-172-31-30-32.us-west-2.compute.internal:31010] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:534) > ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:742) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:841) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:786) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:73) > [drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.foreman.Foreman$StateSwitch.moveToState(Foreman.java:788) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:894) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:255) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_85] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_85] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_85] > Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected > exception during fragment initialization: Internal error: Error while > applying rule PruneScanRule:Filter_On_Scan_Parquet, args > [rel#43148:DrillFilterRel.LOGICAL.ANY([]).[](input=rel#43147:Subset#4.LOGICAL.ANY([]).[],condition==($0, > 1)), rel#43241:DrillScanRel.LOGICAL.ANY([]).[](table=[dfs, > ctasAutoPartition, > tpch_multiple_partitions/lineitem_twopart_ordered2],groupscan=ParquetGroupScan > [entries=[ReadEntryWithPath > [path=hdfs://ip-172-31-30-32:54310/drill/testdata/ctas_auto_partition/tpch_multiple_partitions/lineitem_twopart_ordered2]], > > selectionRoot=hdfs://ip-172-31-30-32:54310/drill/testdata/ctas_auto_partition/tpch_multiple_partitions/lineitem_twopart_ordered2, > numFiles=1, usedMetadataFile=false, columns=[`l_modline`, `l_moddate`]])] > ... 4 common frames omitted > Caused by: java.lang.AssertionError: Internal error: Error while applying > rule PruneScanRule:Filter_On_Scan_Parquet, args > [rel#43148:DrillFilterRel.LOGICAL.ANY([]).[](input=rel#43147:Subset#4.LOGICAL.ANY([]).[],condition==($0, > 1)), rel#43241:DrillScanRel.LOGICAL.ANY([]).[](table=[dfs, > ctasAutoPartition, > tpch_multiple_partitions/lineitem_twopart_ordered2],groupscan=ParquetGroupScan > [entries=[ReadEntryWithPath > [path=hdfs://ip-172-31-30-32:54310/drill/testdata/ctas_auto_partition/tpch_multiple_partitions/lineitem_twopart_ordered2]], > > selectionRoot=hdfs://ip-172-31-30-32:54310/drill/testdata/ctas_auto_partition/tpch_multiple_partitions/lineitem_twopart_ordered2, > numFiles=1, usedMetadataFile=false, columns=[`l_modline`, `l_moddate`]])] > at org.apache.calcite.util.Util.newInternal(Util.java:792) > ~[calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6] > at > org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:251) > ~[calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6] > at > org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:808) > ~[calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6] > at > org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:303) > ~[calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6] > at > org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:303) > ~[calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6] > at > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.logicalPlanningVolcanoAndLopt(DefaultSqlHandler.java:545) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:213) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:248) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:164) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:178) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:905) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:244) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > ... 3 common frames omitted > Caused by: java.lang.StringIndexOutOfBoundsException: String index out of > range: -12 > at java.lang.String.substring(String.java:1875) ~[na:1.7.0_85] > at > org.apache.drill.exec.planner.DFSPartitionLocation.<init>(DFSPartitionLocation.java:31) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.planner.ParquetPartitionDescriptor.createPartitionSublists(ParquetPartitionDescriptor.java:126) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.planner.AbstractPartitionDescriptor.iterator(AbstractPartitionDescriptor.java:53) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.planner.logical.partition.PruneScanRule.doOnMatch(PruneScanRule.java:190) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.planner.logical.partition.ParquetPruneScanRule$2.onMatch(ParquetPruneScanRule.java:87) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:228) > ~[calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6] > ... 13 common frames omitted > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)