[Impala-ASF-CR] IMPALA-4270: Gracefully fail unsupported queries with mt dop > 0.
Internal Jenkins has submitted this change and it was merged. Change subject: IMPALA-4270: Gracefully fail unsupported queries with mt_dop > 0. .. IMPALA-4270: Gracefully fail unsupported queries with mt_dop > 0. MT_DOP > 0 is only supported for plans without distributed joins or table sinks. Adds validation to fail unsupported queries gracefully in planning. For scans in queries that are executable with MT_DOP > 0 we either use the optimized MT scan node BE implementation (only Parquet), or we use the conventional scan node with num_scanner_threads=1. TODO: Still need to add end-to-end tests. Change-Id: I91a60ea7b6e3ae4ee44be856615ddd3cd0af476d Reviewed-on: http://gerrit.cloudera.org:8080/4677 Reviewed-by: Alex Behm Tested-by: Internal Jenkins --- M be/src/exec/exec-node.cc M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/planner/PlannerContext.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java A testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test 10 files changed, 450 insertions(+), 34 deletions(-) Approvals: Internal Jenkins: Verified Alex Behm: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/4677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: I91a60ea7b6e3ae4ee44be856615ddd3cd0af476d Gerrit-PatchSet: 7 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Alex Behm Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Marcel Kornacker
[Impala-ASF-CR] IMPALA-4270: Gracefully fail unsupported queries with mt dop > 0.
Internal Jenkins has posted comments on this change. Change subject: IMPALA-4270: Gracefully fail unsupported queries with mt_dop > 0. .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/4677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I91a60ea7b6e3ae4ee44be856615ddd3cd0af476d Gerrit-PatchSet: 6 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Alex Behm Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Marcel Kornacker Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4270: Gracefully fail unsupported queries with mt dop > 0.
Alex Behm has posted comments on this change. Change subject: IMPALA-4270: Gracefully fail unsupported queries with mt_dop > 0. .. Patch Set 6: Code-Review+2 Rebase + trival fix. -- To view, visit http://gerrit.cloudera.org:8080/4677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I91a60ea7b6e3ae4ee44be856615ddd3cd0af476d Gerrit-PatchSet: 6 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Alex Behm Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Marcel Kornacker Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4270: Gracefully fail unsupported queries with mt dop > 0.
Hello Marcel Kornacker, Internal Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/4677 to look at the new patch set (#6). Change subject: IMPALA-4270: Gracefully fail unsupported queries with mt_dop > 0. .. IMPALA-4270: Gracefully fail unsupported queries with mt_dop > 0. MT_DOP > 0 is only supported for plans without distributed joins or table sinks. Adds validation to fail unsupported queries gracefully in planning. For scans in queries that are executable with MT_DOP > 0 we either use the optimized MT scan node BE implementation (only Parquet), or we use the conventional scan node with num_scanner_threads=1. TODO: Still need to add end-to-end tests. Change-Id: I91a60ea7b6e3ae4ee44be856615ddd3cd0af476d --- M be/src/exec/exec-node.cc M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/planner/PlannerContext.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java A testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test 10 files changed, 450 insertions(+), 34 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/4677/6 -- To view, visit http://gerrit.cloudera.org:8080/4677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I91a60ea7b6e3ae4ee44be856615ddd3cd0af476d Gerrit-PatchSet: 6 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Alex Behm Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Marcel Kornacker
[Impala-ASF-CR] IMPALA-4270: Gracefully fail unsupported queries with mt dop > 0.
Internal Jenkins has posted comments on this change. Change subject: IMPALA-4270: Gracefully fail unsupported queries with mt_dop > 0. .. Patch Set 5: Verified-1 Build failed: http://sandbox.jenkins.cloudera.com/job/impala-external-gerrit-verify-merge-ASF/298/ -- To view, visit http://gerrit.cloudera.org:8080/4677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I91a60ea7b6e3ae4ee44be856615ddd3cd0af476d Gerrit-PatchSet: 5 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Alex Behm Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Marcel Kornacker Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4270: Gracefully fail unsupported queries with mt dop > 0.
Hello Marcel Kornacker, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/4677 to look at the new patch set (#5). Change subject: IMPALA-4270: Gracefully fail unsupported queries with mt_dop > 0. .. IMPALA-4270: Gracefully fail unsupported queries with mt_dop > 0. MT_DOP > 0 is only supported for plans without distributed joins or table sinks. Adds validation to fail unsupported queries gracefully in planning. For scans in queries that are executable with MT_DOP > 0 we either use the optimized MT scan node BE implementation (only Parquet), or we use the conventional scan node with num_scanner_threads=1. TODO: Still need to add end-to-end tests. Change-Id: I91a60ea7b6e3ae4ee44be856615ddd3cd0af476d --- M be/src/exec/exec-node.cc M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/planner/PlannerContext.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java A testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test 10 files changed, 431 insertions(+), 33 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/4677/5 -- To view, visit http://gerrit.cloudera.org:8080/4677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I91a60ea7b6e3ae4ee44be856615ddd3cd0af476d Gerrit-PatchSet: 5 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Alex Behm Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Marcel Kornacker
[Impala-ASF-CR] IMPALA-4270: Gracefully fail unsupported queries with mt dop > 0.
Alex Behm has posted comments on this change. Change subject: IMPALA-4270: Gracefully fail unsupported queries with mt_dop > 0. .. Patch Set 4: (3 comments) http://gerrit.cloudera.org:8080/#/c/4677/4/testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test File testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test: Line 41: PARALLELPLANS > let's also check 'plan' here, because these should also run in single-node Done Line 115: PARALLELPLANS > same here Done Line 170: PARALLELPLANS > and here Done -- To view, visit http://gerrit.cloudera.org:8080/4677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I91a60ea7b6e3ae4ee44be856615ddd3cd0af476d Gerrit-PatchSet: 4 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Alex Behm Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Marcel Kornacker Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4270: Gracefully fail unsupported queries with mt dop > 0.
Alex Behm has posted comments on this change. Change subject: IMPALA-4270: Gracefully fail unsupported queries with mt_dop > 0. .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/4677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I91a60ea7b6e3ae4ee44be856615ddd3cd0af476d Gerrit-PatchSet: 5 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Alex Behm Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Marcel Kornacker Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4270: Gracefully fail unsupported queries with mt dop > 0.
Marcel Kornacker has posted comments on this change. Change subject: IMPALA-4270: Gracefully fail unsupported queries with mt_dop > 0. .. Patch Set 4: Code-Review+2 (3 comments) http://gerrit.cloudera.org:8080/#/c/4677/4/testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test File testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test: Line 41: PARALLELPLANS let's also check 'plan' here, because these should also run in single-node mode Line 115: PARALLELPLANS same here Line 170: PARALLELPLANS and here -- To view, visit http://gerrit.cloudera.org:8080/4677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I91a60ea7b6e3ae4ee44be856615ddd3cd0af476d Gerrit-PatchSet: 4 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Alex Behm Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Marcel Kornacker Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4270: Gracefully fail unsupported queries with mt dop > 0.
Alex Behm has posted comments on this change. Change subject: IMPALA-4270: Gracefully fail unsupported queries with mt_dop > 0. .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/4677/1/testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test File testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test: Line 5: PLAN > the build registration, etc., isn't expected to get in before code freeze, Ahh right. Done. -- To view, visit http://gerrit.cloudera.org:8080/4677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I91a60ea7b6e3ae4ee44be856615ddd3cd0af476d Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Alex Behm Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Marcel Kornacker Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4270: Gracefully fail unsupported queries with mt dop > 0.
Alex Behm has uploaded a new patch set (#4). Change subject: IMPALA-4270: Gracefully fail unsupported queries with mt_dop > 0. .. IMPALA-4270: Gracefully fail unsupported queries with mt_dop > 0. MT_DOP > 0 is only supported for plans without distributed joins or table sinks. Adds validation to fail unsupported queries gracefully in planning. For scans in queries that are executable with MT_DOP > 0 we either use the optimized MT scan node BE implementation (only Parquet), or we use the conventional scan node with num_scanner_threads=1. TODO: Still need to add end-to-end tests. Change-Id: I91a60ea7b6e3ae4ee44be856615ddd3cd0af476d --- M be/src/exec/exec-node.cc M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/planner/PlannerContext.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java A testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test 10 files changed, 308 insertions(+), 33 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/4677/4 -- To view, visit http://gerrit.cloudera.org:8080/4677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I91a60ea7b6e3ae4ee44be856615ddd3cd0af476d Gerrit-PatchSet: 4 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Alex Behm Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Marcel Kornacker
[Impala-ASF-CR] IMPALA-4270: Gracefully fail unsupported queries with mt dop > 0.
Marcel Kornacker has posted comments on this change. Change subject: IMPALA-4270: Gracefully fail unsupported queries with mt_dop > 0. .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/4677/1/testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test File testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test: Line 5: PLAN > We need the current mode for executing joins in subplans where the build wi the build registration, etc., isn't expected to get in before code freeze, so we won't be able to run this. i don't see an alternative to returning an error here. -- To view, visit http://gerrit.cloudera.org:8080/4677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I91a60ea7b6e3ae4ee44be856615ddd3cd0af476d Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Alex Behm Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Marcel Kornacker Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4270: Gracefully fail unsupported queries with mt dop > 0.
Alex Behm has posted comments on this change. Change subject: IMPALA-4270: Gracefully fail unsupported queries with mt_dop > 0. .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/4677/1/testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test File testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test: Line 5: PLAN > but it's not executable, because the join sink isn't executable. We need the current mode for executing joins in subplans where the build will hopefully never be in a separate fragment. -- To view, visit http://gerrit.cloudera.org:8080/4677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I91a60ea7b6e3ae4ee44be856615ddd3cd0af476d Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Alex Behm Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Marcel Kornacker Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4270: Gracefully fail unsupported queries with mt dop > 0.
Alex Behm has uploaded a new patch set (#3). Change subject: IMPALA-4270: Gracefully fail unsupported queries with mt_dop > 0. .. IMPALA-4270: Gracefully fail unsupported queries with mt_dop > 0. MT_DOP > 0 is only supported for plans without distributed joins or table sinks. Adds validation to fail unsupported queries gracefully in planning. For scans in queries that are executable with MT_DOP > 0 we either use the optimized MT scan node BE implementation (only Parquet), or we use the conventional scan node with num_scanner_threads=1. TODO: Still need to add end-to-end tests. Change-Id: I91a60ea7b6e3ae4ee44be856615ddd3cd0af476d --- M be/src/exec/exec-node.cc M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/planner/PlannerContext.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java A testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test 10 files changed, 367 insertions(+), 33 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/4677/3 -- To view, visit http://gerrit.cloudera.org:8080/4677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I91a60ea7b6e3ae4ee44be856615ddd3cd0af476d Gerrit-PatchSet: 3 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Alex Behm Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Marcel Kornacker
[Impala-ASF-CR] IMPALA-4270: Gracefully fail unsupported queries with mt dop > 0.
Alex Behm has posted comments on this change. Change subject: IMPALA-4270: Gracefully fail unsupported queries with mt_dop > 0. .. Patch Set 2: (9 comments) http://gerrit.cloudera.org:8080/#/c/4677/2/be/src/exec/exec-node.cc File be/src/exec/exec-node.cc: Line 272: DCHECK_EQ(state->query_options().num_scanner_threads, 1); > roll into a single dcheck with a compound predicate Done http://gerrit.cloudera.org:8080/#/c/4677/2/common/thrift/PlanNodes.thrift File common/thrift/PlanNodes.thrift: Line 205: > trailing Done Line 207: // If this is true then the MT_DOP query option must be > 0. > leave todo that it should be removed later Done http://gerrit.cloudera.org:8080/#/c/4677/2/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java File fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java: Line 314:* Retrurns a set of file formats being scanned. > spelling Done Line 562: msg.hdfs_scan_node.setUse_mt_scan_nodeIsSet(useMtScanNode_); > ..IsSet? it doesn't look like you're calling the actual setter Weird. That seemed to work also. Fixed. http://gerrit.cloudera.org:8080/#/c/4677/2/fe/src/main/java/org/apache/impala/planner/Planner.java File fe/src/main/java/org/apache/impala/planner/Planner.java: Line 206: if (ctx_.getQueryOptions().mt_dop > 0) { > the if is redundant, but you can make it a checkstate at the start of the f Added Preconditions check. http://gerrit.cloudera.org:8080/#/c/4677/2/fe/src/main/java/org/apache/impala/planner/PlannerContext.java File fe/src/main/java/org/apache/impala/planner/PlannerContext.java: Line 83: public TQueryOptions getQueryOptions() { return getRootAnalyzer().getQueryOptions(); } > why the change? I added a convenience function Analyzer.getQueryOptions() and thought it would make sense to use it here. http://gerrit.cloudera.org:8080/#/c/4677/2/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java File fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java: Line 155:* - MT_DOP > 0 is not supported for plans with distributed joins or table sinks. > it's only supported for QueryStmts. take a look at Frontend.createExecReque My thinking was that from a user's perspective running a statement with mt_dop > 0 should have one of two outcomes: 1. The statement runs fine in multi-threaded mode. 2. The statement returns an error indicating that mt_dop is not supported for that statement. If that's the behavior we want, then it's irrelevant whether we can actually generate parallel plans for inserts/ctas. http://gerrit.cloudera.org:8080/#/c/4677/2/testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test File testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test: Line 51: | tuple-ids=0,1 row-size=8B cardinality=unavailable > what happened here? functional_parquet has no stats. I preferred to write the tests on Parquet because that's the preferred file format, happy to change it back or add more tests? -- To view, visit http://gerrit.cloudera.org:8080/4677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I91a60ea7b6e3ae4ee44be856615ddd3cd0af476d Gerrit-PatchSet: 2 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Alex Behm Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Marcel Kornacker Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4270: Gracefully fail unsupported queries with mt dop > 0.
Marcel Kornacker has posted comments on this change. Change subject: IMPALA-4270: Gracefully fail unsupported queries with mt_dop > 0. .. Patch Set 2: (10 comments) http://gerrit.cloudera.org:8080/#/c/4677/2/be/src/exec/exec-node.cc File be/src/exec/exec-node.cc: Line 272: DCHECK_EQ(state->query_options().num_scanner_threads, 1); roll into a single dcheck with a compound predicate http://gerrit.cloudera.org:8080/#/c/4677/2/common/thrift/PlanNodes.thrift File common/thrift/PlanNodes.thrift: Line 205: trailing Line 207: // If this is true then the MT_DOP query option must be > 0. leave todo that it should be removed later http://gerrit.cloudera.org:8080/#/c/4677/2/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java File fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java: Line 314:* Retrurns a set of file formats being scanned. spelling the set Line 562: msg.hdfs_scan_node.setUse_mt_scan_nodeIsSet(useMtScanNode_); ..IsSet? it doesn't look like you're calling the actual setter http://gerrit.cloudera.org:8080/#/c/4677/2/fe/src/main/java/org/apache/impala/planner/Planner.java File fe/src/main/java/org/apache/impala/planner/Planner.java: Line 206: if (ctx_.getQueryOptions().mt_dop > 0) { the if is redundant, but you can make it a checkstate at the start of the function, if you want. also, the planner only creates parallel plans for query stmts. http://gerrit.cloudera.org:8080/#/c/4677/2/fe/src/main/java/org/apache/impala/planner/PlannerContext.java File fe/src/main/java/org/apache/impala/planner/PlannerContext.java: Line 83: public TQueryOptions getQueryOptions() { return getRootAnalyzer().getQueryOptions(); } why the change? http://gerrit.cloudera.org:8080/#/c/4677/2/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java File fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java: Line 155:* - MT_DOP > 0 is not supported for plans with distributed joins or table sinks. it's only supported for QueryStmts. take a look at Frontend.createExecRequest() http://gerrit.cloudera.org:8080/#/c/4677/1/testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test File testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test: Line 5: PLAN > The purpose is to validate that this plan is executable with mt_dop>0 and n but it's not executable, because the join sink isn't executable. sorry, i overlooked that earlier: for the upcoming release we need to turn this off even for non-distributed joins. http://gerrit.cloudera.org:8080/#/c/4677/2/testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test File testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test: Line 51: | tuple-ids=0,1 row-size=8B cardinality=unavailable what happened here? -- To view, visit http://gerrit.cloudera.org:8080/4677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I91a60ea7b6e3ae4ee44be856615ddd3cd0af476d Gerrit-PatchSet: 2 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Alex Behm Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Marcel Kornacker Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4270: Gracefully fail unsupported queries with mt dop > 0.
Alex Behm has posted comments on this change. Change subject: IMPALA-4270: Gracefully fail unsupported queries with mt_dop > 0. .. Patch Set 1: (4 comments) http://gerrit.cloudera.org:8080/#/c/4677/1/testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test File testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test: Line 5: PLAN > do we need this section? or isn't this already validated somewhere else? The purpose is to validate that this plan is executable with mt_dop>0 and num_nodes=1. I don't think we test that elsewhere. Line 32: DISTRIBUTEDPLAN > we don't need this section Removed. Line 121: select row_number() over(partition by int_col order by id) > have you tried to run this? I tried several values of mt_dop and this query worked fine. Line 156: from tpch_nested_parquet.customer c, c.c_orders o, o.o_lineitems > have you tried this? I tried this and other similar queries. Works fine with mt_dop=2 but crashes with higher values. Looks like a bug. Should't this query be runnable with mt_dop>0? Btw, compute stats fails with mt_dop=2, so it looks like there are still some issues. -- To view, visit http://gerrit.cloudera.org:8080/4677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I91a60ea7b6e3ae4ee44be856615ddd3cd0af476d Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Alex Behm Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Marcel Kornacker Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4270: Gracefully fail unsupported queries with mt dop > 0.
Alex Behm has uploaded a new patch set (#2). Change subject: IMPALA-4270: Gracefully fail unsupported queries with mt_dop > 0. .. IMPALA-4270: Gracefully fail unsupported queries with mt_dop > 0. MT_DOP > 0 is only supported for plans without distributed joins or table sinks. Adds validation to fail unsupported queries gracefully in planning. For scans in queries that are executable with MT_DOP > 0 we either use the optimized MT scan node BE implementation (only Parquet), or we use the conventional scan node with num_scanner_threads=1. TODO: Still need to add end-to-end tests. Change-Id: I91a60ea7b6e3ae4ee44be856615ddd3cd0af476d --- M be/src/exec/exec-node.cc M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/planner/PlannerContext.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java A testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test 10 files changed, 368 insertions(+), 33 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/4677/2 -- To view, visit http://gerrit.cloudera.org:8080/4677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I91a60ea7b6e3ae4ee44be856615ddd3cd0af476d Gerrit-PatchSet: 2 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Alex Behm Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Marcel Kornacker
[Impala-ASF-CR] IMPALA-4270: Gracefully fail unsupported queries with mt dop > 0.
Marcel Kornacker has posted comments on this change. Change subject: IMPALA-4270: Gracefully fail unsupported queries with mt_dop > 0. .. Patch Set 1: (4 comments) http://gerrit.cloudera.org:8080/#/c/4677/1/testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test File testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test: Line 5: PLAN do we need this section? or isn't this already validated somewhere else? Line 32: DISTRIBUTEDPLAN we don't need this section Line 121: select row_number() over(partition by int_col order by id) have you tried to run this? Line 156: from tpch_nested_parquet.customer c, c.c_orders o, o.o_lineitems have you tried this? -- To view, visit http://gerrit.cloudera.org:8080/4677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I91a60ea7b6e3ae4ee44be856615ddd3cd0af476d Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Alex Behm Gerrit-Reviewer: Marcel Kornacker Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4270: Gracefully fail unsupported queries with mt dop > 0.
Alex Behm has uploaded a new change for review. http://gerrit.cloudera.org:8080/4677 Change subject: IMPALA-4270: Gracefully fail unsupported queries with mt_dop > 0. .. IMPALA-4270: Gracefully fail unsupported queries with mt_dop > 0. Change-Id: I91a60ea7b6e3ae4ee44be856615ddd3cd0af476d --- M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java A testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test 5 files changed, 305 insertions(+), 24 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/4677/1 -- To view, visit http://gerrit.cloudera.org:8080/4677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I91a60ea7b6e3ae4ee44be856615ddd3cd0af476d Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Alex Behm