[Impala-ASF-CR] IMPALA-11604 Planner changes for CPU usage
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19033 ) Change subject: IMPALA-11604 Planner changes for CPU usage .. Patch Set 45: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12349/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19033 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If32dc770dfffcdd0be2ba789a7720952c68a Gerrit-Change-Number: 19033 Gerrit-PatchSet: 45 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 10 Feb 2023 05:37:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11604 Planner changes for CPU usage
Riza Suminto has posted comments on this change. (
http://gerrit.cloudera.org:8080/19033 )
Change subject: IMPALA-11604 Planner changes for CPU usage
..
Patch Set 45:
(11 comments)
http://gerrit.cloudera.org:8080/#/c/19033/44//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19033/44//COMMIT_MSG@16
PS44, Line 16: then finally returns a
> May need to explain why blocking operators are considered in the context of
Removed this phase here and put more explanation in steps II and III.
http://gerrit.cloudera.org:8080/#/c/19033/44//COMMIT_MSG@19
PS44, Line 19:
> nit found
Done
http://gerrit.cloudera.org:8080/#/c/19033/37/be/src/util/backend-gflag-util.cc
File be/src/util/backend-gflag-util.cc:
http://gerrit.cloudera.org:8080/#/c/19033/37/be/src/util/backend-gflag-util.cc@201
PS37, Line 201: 2 w
> The scaling factor as defined is less intuitive, since one has to inverse i
Renamed to query_cpu_requirement_divisor.
http://gerrit.cloudera.org:8080/#/c/19033/37/fe/src/main/java/org/apache/impala/planner/PlanFragment.java
File fe/src/main/java/org/apache/impala/planner/PlanFragment.java:
http://gerrit.cloudera.org:8080/#/c/19033/37/fe/src/main/java/org/apache/impala/planner/PlanFragment.java@147
PS37, Line 147: Costs_ = null;
: private List
> nit. A positive value implies the instance count has been adjusted.
Done
http://gerrit.cloudera.org:8080/#/c/19033/37/fe/src/main/java/org/apache/impala/planner/PlanFragment.java@864
PS37, Line 864: r(Predicates.i
> Better renamed as getMaxParallelismByTotalWorkSize().
Done
http://gerrit.cloudera.org:8080/#/c/19033/37/fe/src/main/java/org/apache/impala/planner/PlanFragment.java@883
PS37, Line 883:Preconditions.checkState(p.getChildCount() > 1);
: long buildRowCount = p.getChild(1).getCardinality();
: if (((JoinNode) p).getDistributionMode() ==
DistributionMode.BROADCAST) {
: // For Broadcast join, all join receive the same work
size from the build.
: buildRowCount = buildRowCount * p.getNumInstances();
: }
:
> Repeated use from line 869. Can be refactored.
Removed.
http://gerrit.cloudera.org:8080/#/c/19033/37/fe/src/main/java/org/apache/impala/planner/PlanFragment.java@957
PS37, Line 957: protected int getAdjustedInstanceCount() { return
adjustedInstanceCount_; }
> Should add a comment.
Done
http://gerrit.cloudera.org:8080/#/c/19033/37/fe/src/main/java/org/apache/impala/planner/PlanFragment.java@972
PS37, Line 972: ngBuilder builder = new StringBuilder();
: for (int i = processingCosts_.size() - 1; i >= 0; i--)
> I wonder if the computation can be improved here e.g. by the size of the wo
In newer patch set, this is later lowered based on producer-consumer rate ratio.
http://gerrit.cloudera.org:8080/#/c/19033/37/fe/src/main/java/org/apache/impala/planner/PlanFragment.java@1098
PS37, Line 1098: Math.max(nodeStepCount,
getMaxParallelismByTotalWorkSize());
> add a comment should be helpful.
Done
http://gerrit.cloudera.org:8080/#/c/19033/37/fe/src/main/java/org/apache/impala/planner/PlanFragment.java@1113
PS37, Line 1113:
> does not sound right.
Removed.
http://gerrit.cloudera.org:8080/#/c/19033/44/testdata/workloads/functional-planner/queries/PlannerTest/tpcds-processing-cost.test
File
testdata/workloads/functional-planner/queries/PlannerTest/tpcds-processing-cost.test:
http://gerrit.cloudera.org:8080/#/c/19033/44/testdata/workloads/functional-planner/queries/PlannerTest/tpcds-processing-cost.test@75
PS44, Line 75: ss_sold_date_sk =
> Is it possible to show the new processing cost here too? It will be wonderf
Done. Add max-parallelism and fragment-costs as well.
--
To view, visit http://gerrit.cloudera.org:8080/19033
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If32dc770dfffcdd0be2ba789a7720952c68a
Gerrit-Change-Number: 19033
Gerrit-PatchSet: 45
Gerrit-Owner: Qifan Chen
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Qifan Chen
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Fri, 10 Feb 2023 05:28:40 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11604 Planner changes for CPU usage
Riza Suminto has uploaded a new patch set (#45) to the change originally created by Qifan Chen. ( http://gerrit.cloudera.org:8080/19033 ) Change subject: IMPALA-11604 Planner changes for CPU usage .. IMPALA-11604 Planner changes for CPU usage This patch augments IMPALA-10992 by establishing an infrastructure to allow the weighted total amount of data to process to be used as a new factor in the definition and selection of an executor group. At the basis of the CPU costing model, we define ProcessingCost as a cost for a distinct PlanNode / DataSink / PlanFragment to process its input rows globally across all of its instances. The costing algorithm then tries to adjust the number of instances for each fragment by considering their production-consumption ratio, and then finally returns a number representing an ideal CPU core count required for a query to run efficiently. A more detailed explanation of the CPU costing algorithm can be found in the four steps below. I. Compute ProcessingCost for each plan node and data sink. ProcessingCost of a PlanNode/DataSink is a weighted amount of data processed by that node/sink. The basic ProcessingCost is computed with a general formula as follows. ProcessingCost is a pair: PC(D, N), where D = I * (C + M) where D is the weighted amount of data processed I is the input cardinality C is the expression evaluation cost per row. Set to total weight of expression evaluation in node/sink. M is a materialization cost per row. Only used by scan and exchange node. Otherwise, 0. N is the number of instances. Default to D / MIN_COST_PER_THREAD (1 million), but is fixed for a certain node/sink and adjustable in step III. In this patch, the weight of each expression evaluation is set to a constant of 1. A description of the computation for each kind of PlanNode/DataSink is given below. 01. AggregationNode: Each AggregateInfo has its C as a sum of grouping expression and aggregate expression and then assigned a single ProcessingCost individually. These ProcessingCosts then summed to be the Aggregation node's ProcessingCost; 02. AnalyticEvalNode: C is the sum of the evaluation costs for analytic functions; 03. CardinalityCheckNode: Use the general formula, I = 1; 04. DataSourceScanNode: Follow the formula from the superclass ScanNode; 05. EmptySetNode: I = 0; 06. ExchangeNode: M = 1 / row batch size. A modification of the general formula when in broadcast mode: D = D * number of receivers; 07. HashJoinNode: probe cost = PC(I0 * C(equiJoin predicate), N) + PC(output cardinality * C(otherJoin predicate), N) build cost = PC(I1 * C(equi-join predicate), N) With I0 and I1 as input cardinality of the probe and build side accordingly. If the plan node does not have a separate build, ProcessingCost is the sum of probe cost and build cost. Otherwise, ProcessingCost is equal to probeCost. 08. HbaseScanNode, HdfsScanNode, and KuduScanNode: Follow the formula from the superclass ScanNode; 09. Nested loop join node: When the right child is not a SingularRowSrcNode: probe cost = PC(I0 * C(equiJoin predicate), N) + PC(output cardinality * C(otherJoin predicate), N) build cost = PC(I1 * C(equiJoin predicate), N) When the right child is a SingularRowSrcNode: probe cost = PC(I0, N) build cost = PC(I0 * I1, N) With I0 and I1 as input cardinality of the probe and build side accordingly. If the plan node does not have a separate build, ProcessingCost is the sum of probe cost and build cost. Otherwise, ProcessingCost is equal to probeCost. 10. ScanNode: M = 1 / row batch size; 11. SelectNode: Use the general formula; 12. SingularRowSrcNode: Since the node is involved once per input in nested loop join, the contribution of this node is computed in nested loop join; 13. SortNode: C is the evaluation cost for the sort expression; 14. SubplanNode: C is 1. I is the multiplication of the cardinality of the left and the right child; 15. Union node: C is the cost of result expression evaluation from all non-pass-through children; 16. Unnest node: I is the cardinality of the containing SubplanNode and C is 1. 17. DataStreamSink: M = 1 / num rows per batch. 18. JoinBuildSink: ProcessingCost is the build cost of its associated JoinNode. 29. PlanRootSink: If result spooling is enabled, C is the cost of output expression evaluation. Otherwise. ProcessingCost is zero. 20. TableSink: C is the cost of output expression evaluation. TableSink subclasses (including HBaseTableSink, HdfsTableSink, and KuduTableSink) follow the same formula; II. Compute the total ProcessingCost of a fragment. A query fragment
[Impala-ASF-CR] IMPALA-11850 Adds HTTP tracing headers when using the hs2-http protocol.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19428 ) Change subject: IMPALA-11850 Adds HTTP tracing headers when using the hs2-http protocol. .. IMPALA-11850 Adds HTTP tracing headers when using the hs2-http protocol. When using the hs2 protocol with the http transport, include several tracing http headers by default. These headers are: * X-Request-Id-- client defined string that identifies the http request, this string is meaningful only to the client * X-Impala-Session-Id -- session id generated by the Impala backend, will be omitted on http calls that occur before this id has been generated * X-Impala-Query-Id -- query id generated by the Impala backend, will be omitted on http calls that occur before this id has been generated The Impala shell includes these headers by default. The command line argument --no_http_tracing has been added to remove these headers. The Impala backend logs out these headers if they are on the http request. The log messages are written out at log level 2 (RPC). Testing: - manual testing (verified using debugging proxy and impala logs) - new python test Change-Id: I7857eb5ec03eba32e06ec8d4133480f2e958ad2f Reviewed-on: http://gerrit.cloudera.org:8080/19428 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/transport/THttpServer.cpp M be/src/transport/THttpServer.h M shell/ImpalaHttpClient.py M shell/impala_client.py M shell/impala_shell.py M shell/impala_shell_config_defaults.py M shell/option_parser.py M tests/common/test_dimensions.py A tests/custom_cluster/test_shell_commandline.py 9 files changed, 295 insertions(+), 8 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/19428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I7857eb5ec03eba32e06ec8d4133480f2e958ad2f Gerrit-Change-Number: 19428 Gerrit-PatchSet: 18 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-11850 Adds HTTP tracing headers when using the hs2-http protocol.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19428 ) Change subject: IMPALA-11850 Adds HTTP tracing headers when using the hs2-http protocol. .. Patch Set 17: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7857eb5ec03eba32e06ec8d4133480f2e958ad2f Gerrit-Change-Number: 19428 Gerrit-PatchSet: 17 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Fri, 10 Feb 2023 02:09:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11417: Support outer join elimination optimization
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18705 ) Change subject: IMPALA-11417: Support outer join elimination optimization .. Patch Set 12: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18705 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If2e68263a029ac84a4f35b0846b22aa42d7ceece Gerrit-Change-Number: 18705 Gerrit-PatchSet: 12 Gerrit-Owner: Jian Zhang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiang Yang Gerrit-Comment-Date: Thu, 09 Feb 2023 21:01:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11850 Adds HTTP tracing headers when using the hs2-http protocol.
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/19428 ) Change subject: IMPALA-11850 Adds HTTP tracing headers when using the hs2-http protocol. .. Patch Set 16: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7857eb5ec03eba32e06ec8d4133480f2e958ad2f Gerrit-Change-Number: 19428 Gerrit-PatchSet: 16 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Thu, 09 Feb 2023 20:55:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11850 Adds HTTP tracing headers when using the hs2-http protocol.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19428 ) Change subject: IMPALA-11850 Adds HTTP tracing headers when using the hs2-http protocol. .. Patch Set 17: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9042/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/19428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7857eb5ec03eba32e06ec8d4133480f2e958ad2f Gerrit-Change-Number: 19428 Gerrit-PatchSet: 17 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Thu, 09 Feb 2023 20:56:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11850 Adds HTTP tracing headers when using the hs2-http protocol.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19428 ) Change subject: IMPALA-11850 Adds HTTP tracing headers when using the hs2-http protocol. .. Patch Set 17: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7857eb5ec03eba32e06ec8d4133480f2e958ad2f Gerrit-Change-Number: 19428 Gerrit-PatchSet: 17 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Thu, 09 Feb 2023 20:56:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11850 Adds HTTP tracing headers when using the hs2-http protocol.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/19428 ) Change subject: IMPALA-11850 Adds HTTP tracing headers when using the hs2-http protocol. .. Patch Set 16: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/19428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7857eb5ec03eba32e06ec8d4133480f2e958ad2f Gerrit-Change-Number: 19428 Gerrit-PatchSet: 16 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Thu, 09 Feb 2023 20:35:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11850 Adds HTTP tracing headers when using the hs2-http protocol.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19428 ) Change subject: IMPALA-11850 Adds HTTP tracing headers when using the hs2-http protocol. .. Patch Set 16: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12348/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7857eb5ec03eba32e06ec8d4133480f2e958ad2f Gerrit-Change-Number: 19428 Gerrit-PatchSet: 16 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Thu, 09 Feb 2023 20:28:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11850 Adds HTTP tracing headers when using the hs2-http protocol.
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/19428 ) Change subject: IMPALA-11850 Adds HTTP tracing headers when using the hs2-http protocol. .. Patch Set 14: (1 comment) http://gerrit.cloudera.org:8080/#/c/19428/14/tests/custom_cluster/test_shell_commandline.py File tests/custom_cluster/test_shell_commandline.py: http://gerrit.cloudera.org:8080/#/c/19428/14/tests/custom_cluster/test_shell_commandline.py@29 PS14, Line 29: class TestImpalaShellCustomCluster(CustomClusterTestSuite): > I meant the class name. The description is nice too. Fixing that too. I agree the class name needs to be changed. -- To view, visit http://gerrit.cloudera.org:8080/19428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7857eb5ec03eba32e06ec8d4133480f2e958ad2f Gerrit-Change-Number: 19428 Gerrit-PatchSet: 14 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Thu, 09 Feb 2023 20:08:07 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11850 Adds HTTP tracing headers when using the hs2-http protocol.
Jason Fehr has uploaded a new patch set (#16). ( http://gerrit.cloudera.org:8080/19428 ) Change subject: IMPALA-11850 Adds HTTP tracing headers when using the hs2-http protocol. .. IMPALA-11850 Adds HTTP tracing headers when using the hs2-http protocol. When using the hs2 protocol with the http transport, include several tracing http headers by default. These headers are: * X-Request-Id-- client defined string that identifies the http request, this string is meaningful only to the client * X-Impala-Session-Id -- session id generated by the Impala backend, will be omitted on http calls that occur before this id has been generated * X-Impala-Query-Id -- query id generated by the Impala backend, will be omitted on http calls that occur before this id has been generated The Impala shell includes these headers by default. The command line argument --no_http_tracing has been added to remove these headers. The Impala backend logs out these headers if they are on the http request. The log messages are written out at log level 2 (RPC). Testing: - manual testing (verified using debugging proxy and impala logs) - new python test Change-Id: I7857eb5ec03eba32e06ec8d4133480f2e958ad2f --- M be/src/transport/THttpServer.cpp M be/src/transport/THttpServer.h M shell/ImpalaHttpClient.py M shell/impala_client.py M shell/impala_shell.py M shell/impala_shell_config_defaults.py M shell/option_parser.py M tests/common/test_dimensions.py A tests/custom_cluster/test_shell_commandline.py 9 files changed, 295 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/28/19428/16 -- To view, visit http://gerrit.cloudera.org:8080/19428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7857eb5ec03eba32e06ec8d4133480f2e958ad2f Gerrit-Change-Number: 19428 Gerrit-PatchSet: 16 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-11850 Adds HTTP tracing headers when using the hs2-http protocol.
Jason Fehr has posted comments on this change. (
http://gerrit.cloudera.org:8080/19428 )
Change subject: IMPALA-11850 Adds HTTP tracing headers when using the hs2-http
protocol.
..
Patch Set 10:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/19428/10/tests/shell/test_shell_commandline.py
File tests/shell/test_shell_commandline.py:
http://gerrit.cloudera.org:8080/#/c/19428/10/tests/shell/test_shell_commandline.py@1586
PS10, Line 1586: @CustomClusterTestSuite.with_args("-log_dir={0} -v
2".format(LOG_DIR_HTTP_TRACING))
> Oh, these are CustomCluster tests. They should be in one of tests/custom_cl
I moved all these tests to a new file
tests/custom_cluster/test_shell_commandline.py
--
To view, visit http://gerrit.cloudera.org:8080/19428
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7857eb5ec03eba32e06ec8d4133480f2e958ad2f
Gerrit-Change-Number: 19428
Gerrit-PatchSet: 10
Gerrit-Owner: Jason Fehr
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jason Fehr
Gerrit-Reviewer: Michael Smith
Gerrit-Comment-Date: Thu, 09 Feb 2023 19:47:16 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] [tools] Guard data load for Ozone
Hello Kurt Deschler, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19459 to look at the new patch set (#3). Change subject: [tools] Guard data load for Ozone .. [tools] Guard data load for Ozone Guards data load for Ozone as a usable snapshot is required. Also fixes a typo in fixed issues. Change-Id: Idc37d03780fca35427b977524b2b97a6892c88f7 --- M docs/topics/impala_fixed_issues.xml M testdata/bin/create-load-data.sh 2 files changed, 4 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/19459/3 -- To view, visit http://gerrit.cloudera.org:8080/19459 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idc37d03780fca35427b977524b2b97a6892c88f7 Gerrit-Change-Number: 19459 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] [tools] Guard data load for Ozone
Hello Kurt Deschler, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19459 to look at the new patch set (#2). Change subject: [tools] Guard data load for Ozone .. [tools] Guard data load for Ozone Guards data load for Ozone as a usable snapshot is required. Also fixes a typo in fixed issues. Change-Id: Idc37d03780fca35427b977524b2b97a6892c88f7 --- M docs/topics/impala_fixed_issues.xml M testdata/bin/create-load-data.sh 2 files changed, 4 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/19459/2 -- To view, visit http://gerrit.cloudera.org:8080/19459 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idc37d03780fca35427b977524b2b97a6892c88f7 Gerrit-Change-Number: 19459 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-11869: Use to string() in PrintValue for printing Thrift types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19487 ) Change subject: IMPALA-11869: Use to_string() in PrintValue for printing Thrift types .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19487 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibc5b847dea2bdea7ba0ab8e093a8bab5a8145019 Gerrit-Change-Number: 19487 Gerrit-PatchSet: 1 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Comment-Date: Thu, 09 Feb 2023 18:35:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11850 Adds HTTP tracing headers when using the hs2-http protocol.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/19428 ) Change subject: IMPALA-11850 Adds HTTP tracing headers when using the hs2-http protocol. .. Patch Set 14: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/19428/14/tests/custom_cluster/test_shell_commandline.py File tests/custom_cluster/test_shell_commandline.py: http://gerrit.cloudera.org:8080/#/c/19428/14/tests/custom_cluster/test_shell_commandline.py@29 PS14, Line 29: class TestImpalaShellCustomCluster(CustomClusterTestSuite): nit: it'd be nice if this was more descriptive of what specifically you're testing, and if the file name matched. We already have other custom_cluster tests dealing with the shell, such as test_shell_interactive.py. -- To view, visit http://gerrit.cloudera.org:8080/19428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7857eb5ec03eba32e06ec8d4133480f2e958ad2f Gerrit-Change-Number: 19428 Gerrit-PatchSet: 14 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Thu, 09 Feb 2023 18:26:38 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11850 Adds HTTP tracing headers when using the hs2-http protocol.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19428 ) Change subject: IMPALA-11850 Adds HTTP tracing headers when using the hs2-http protocol. .. Patch Set 14: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12347/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7857eb5ec03eba32e06ec8d4133480f2e958ad2f Gerrit-Change-Number: 19428 Gerrit-PatchSet: 14 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Thu, 09 Feb 2023 18:00:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11850 Adds HTTP tracing headers when using the hs2-http protocol.
Jason Fehr has uploaded a new patch set (#14). ( http://gerrit.cloudera.org:8080/19428 ) Change subject: IMPALA-11850 Adds HTTP tracing headers when using the hs2-http protocol. .. IMPALA-11850 Adds HTTP tracing headers when using the hs2-http protocol. When using the hs2 protocol with the http transport, include several tracing http headers by default. These headers are: * X-Request-Id-- client defined string that identifies the http request, this string is meaningful only to the client * X-Impala-Session-Id -- session id generated by the Impala backend, will be omitted on http calls that occur before this id has been generated * X-Impala-Query-Id -- query id generated by the Impala backend, will be omitted on http calls that occur before this id has been generated The Impala shell includes these headers by default. The command line argument --no_http_tracing has been added to remove these headers. The Impala backend logs out these headers if they are on the http request. The log messages are written out at log level 2 (RPC). Testing: - manual testing (verified using debugging proxy and impala logs) - new python test Change-Id: I7857eb5ec03eba32e06ec8d4133480f2e958ad2f --- M be/src/transport/THttpServer.cpp M be/src/transport/THttpServer.h M shell/ImpalaHttpClient.py M shell/impala_client.py M shell/impala_shell.py M shell/impala_shell_config_defaults.py M shell/option_parser.py M tests/common/test_dimensions.py A tests/custom_cluster/test_shell_commandline.py 9 files changed, 290 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/28/19428/14 -- To view, visit http://gerrit.cloudera.org:8080/19428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7857eb5ec03eba32e06ec8d4133480f2e958ad2f Gerrit-Change-Number: 19428 Gerrit-PatchSet: 14 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR](asf-site) IMPALA-11899: Add mandatory links to the Impala Apache home page
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/19477 ) Change subject: IMPALA-11899: Add mandatory links to the Impala Apache home page .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19477 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: Ibbb491fe2ec3c15305c3c66f1e8857a98fced10f Gerrit-Change-Number: 19477 Gerrit-PatchSet: 2 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 09 Feb 2023 17:06:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11417: Support outer join elimination optimization
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18705 ) Change subject: IMPALA-11417: Support outer join elimination optimization .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12346/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18705 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If2e68263a029ac84a4f35b0846b22aa42d7ceece Gerrit-Change-Number: 18705 Gerrit-PatchSet: 12 Gerrit-Owner: Jian Zhang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiang Yang Gerrit-Comment-Date: Thu, 09 Feb 2023 16:13:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11908: Parser change for Iceberg metadata querying
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19483 ) Change subject: IMPALA-11908: Parser change for Iceberg metadata querying .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19483 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0b5db884b5f3fecbd132fcb2c2cbd6c622ff965b Gerrit-Change-Number: 19483 Gerrit-PatchSet: 4 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 09 Feb 2023 16:12:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11417: Support outer join elimination optimization
Xiang Yang has uploaded a new patch set (#12) to the change originally created by Jian Zhang. ( http://gerrit.cloudera.org:8080/18705 ) Change subject: IMPALA-11417: Support outer join elimination optimization .. IMPALA-11417: Support outer join elimination optimization When two tables are outer joined but only fields from the outer side table are used and the join key of the inner side table is guaranteed to be unique, the query can be simplified to only scan the outer table: drop table if exists t; drop table if exists s; create table t(sid bigint, value bigint); create table s(id bigint, value bigint, primary key(id)); -- the test SQL: select t.* from t left join s on t.sid = s.id; The above query can be simplified to: select t.* from t; This optimization utilizes the primary key constraint when creating join nodes, eliminates the inner side when the join key on inner side is the primary key and only the slots from the outer side are used by the parent. Change-Id: If2e68263a029ac84a4f35b0846b22aa42d7ceece Signed-off-by: Jian Zhang --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java A testdata/workloads/functional-planner/queries/PlannerTest/outer-join-elimination.test A testdata/workloads/tpcds/queries/outer-join-elimination.test M tests/query_test/test_join_queries.py 11 files changed, 536 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/18705/12 -- To view, visit http://gerrit.cloudera.org:8080/18705 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If2e68263a029ac84a4f35b0846b22aa42d7ceece Gerrit-Change-Number: 18705 Gerrit-PatchSet: 12 Gerrit-Owner: Jian Zhang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiang Yang
[Impala-ASF-CR] IMPALA-11417: Support outer join elimination optimization
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18705 ) Change subject: IMPALA-11417: Support outer join elimination optimization .. Patch Set 12: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9041/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18705 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If2e68263a029ac84a4f35b0846b22aa42d7ceece Gerrit-Change-Number: 18705 Gerrit-PatchSet: 12 Gerrit-Owner: Jian Zhang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiang Yang Gerrit-Comment-Date: Thu, 09 Feb 2023 15:54:34 + Gerrit-HasComments: No
[Impala-ASF-CR](asf-site) IMPALA-11899: Add mandatory links to the Impala Apache home page
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/19477 ) Change subject: IMPALA-11899: Add mandatory links to the Impala Apache home page .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/19477/1/index.html File index.html: http://gerrit.cloudera.org:8080/#/c/19477/1/index.html@169 PS1, Line 169: > nit: Indentation looks off now that everything's in the tag. Done -- To view, visit http://gerrit.cloudera.org:8080/19477 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: Ibbb491fe2ec3c15305c3c66f1e8857a98fced10f Gerrit-Change-Number: 19477 Gerrit-PatchSet: 2 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 09 Feb 2023 15:47:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR](asf-site) IMPALA-11899: Add mandatory links to the Impala Apache home page
Hello Daniel Becker, Zoltan Borok-Nagy, Joe McDonnell, Csaba Ringhofer, Michael Smith, Impala Public Jenkins, Jim Apple, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19477 to look at the new patch set (#2). Change subject: IMPALA-11899: Add mandatory links to the Impala Apache home page .. IMPALA-11899: Add mandatory links to the Impala Apache home page The Apache Foundation has a number of requirements for TLP websites, which are checked regularly, with reports being published whimsy.apache.org. Impala failed in two categories: missing a copyright notice, and missing a link to the Apache Privacy Policy. This change to the Impala home page adds both. It also fixes a misindented source line, and wraps the whole footer into the tag originally reserved for the events box. Tested by loading the local copy of the page into Chrome and Firefox on my system. Change-Id: Ibbb491fe2ec3c15305c3c66f1e8857a98fced10f --- M index.html 1 file changed, 10 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/19477/2 -- To view, visit http://gerrit.cloudera.org:8080/19477 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibbb491fe2ec3c15305c3c66f1e8857a98fced10f Gerrit-Change-Number: 19477 Gerrit-PatchSet: 2 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-11869: Use to string() in PrintValue for printing Thrift types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19487 ) Change subject: IMPALA-11869: Use to_string() in PrintValue for printing Thrift types .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12345/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19487 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibc5b847dea2bdea7ba0ab8e093a8bab5a8145019 Gerrit-Change-Number: 19487 Gerrit-PatchSet: 1 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Comment-Date: Thu, 09 Feb 2023 13:53:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11869: Use to string() in PrintValue for printing Thrift types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19487 ) Change subject: IMPALA-11869: Use to_string() in PrintValue for printing Thrift types .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9040/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/19487 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibc5b847dea2bdea7ba0ab8e093a8bab5a8145019 Gerrit-Change-Number: 19487 Gerrit-PatchSet: 1 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Comment-Date: Thu, 09 Feb 2023 13:31:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11869: Use to string() in PrintValue for printing Thrift types
Daniel Becker has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19487 Change subject: IMPALA-11869: Use to_string() in PrintValue for printing Thrift types .. IMPALA-11869: Use to_string() in PrintValue for printing Thrift types IMPALA-11645 introduced the function PrintValue() which we use to convert Thrift types to strings. This function now uses operator<<, which is provided for the generated Thrift types. However, Thrift also generates a to_string() function that is overloaded for Thrift types. It would be more efficient to use this instead of involving streams with operator<<. This change reimplements PrintValue() using to_string() instead of operator<<. Change-Id: Ibc5b847dea2bdea7ba0ab8e093a8bab5a8145019 --- M be/src/util/debug-util.h 1 file changed, 3 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/19487/1 -- To view, visit http://gerrit.cloudera.org:8080/19487 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ibc5b847dea2bdea7ba0ab8e093a8bab5a8145019 Gerrit-Change-Number: 19487 Gerrit-PatchSet: 1 Gerrit-Owner: Daniel Becker
[Impala-ASF-CR] IMPALA-11908: Parser change for Iceberg metadata querying
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/19483 ) Change subject: IMPALA-11908: Parser change for Iceberg metadata querying .. Patch Set 4: (11 comments) http://gerrit.cloudera.org:8080/#/c/19483/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19483/4//COMMIT_MSG@13 PS4, Line 13: . Nit: comma? http://gerrit.cloudera.org:8080/#/c/19483/4//COMMIT_MSG@14 PS4, Line 14: metadat Nit: metadata. http://gerrit.cloudera.org:8080/#/c/19483/4/fe/src/main/java/org/apache/impala/analysis/Analyzer.java File fe/src/main/java/org/apache/impala/analysis/Analyzer.java: http://gerrit.cloudera.org:8080/#/c/19483/4/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3289 PS4, Line 3289: database 'dbName' is no longer a parameter of this function. http://gerrit.cloudera.org:8080/#/c/19483/4/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3337 PS4, Line 3337:* Adds a new metadata table to the stmt table cache. At this point it is unknown if Could add that this is for Iceberg only (at least now). http://gerrit.cloudera.org:8080/#/c/19483/4/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3342 PS4, Line 3342: MetaVirtual MetadataVirtualTable? http://gerrit.cloudera.org:8080/#/c/19483/4/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3349 PS4, Line 3349: virtualTableName It's a bit confusing that we get 'originalTable' from 'virtualTableName' and not for example 'originalTableName'. http://gerrit.cloudera.org:8080/#/c/19483/4/fe/src/main/java/org/apache/impala/analysis/TableName.java File fe/src/main/java/org/apache/impala/analysis/TableName.java: http://gerrit.cloudera.org:8080/#/c/19483/4/fe/src/main/java/org/apache/impala/analysis/TableName.java@42 PS4, Line 42: vTbl_ Could add a comment describing what 'vTbl_' is used for. http://gerrit.cloudera.org:8080/#/c/19483/4/fe/src/main/java/org/apache/impala/analysis/TableName.java@55 PS4, Line 55: Preconditions.checkNotNull(db); In the other constructor on L44, we check that db == null || !db.isEmpty() I guess we don't allow 'db' to be null because of what is written on L82, but do we allow 'db.isEmpty()'? http://gerrit.cloudera.org:8080/#/c/19483/4/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java File fe/src/main/java/org/apache/impala/catalog/IcebergTable.java: http://gerrit.cloudera.org:8080/#/c/19483/4/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java@36 PS4, Line 36: import org.apache.impala.catalog.HdfsPartition.FileDescriptor; : import org.apache.impala.common.Pair; Unused imports. http://gerrit.cloudera.org:8080/#/c/19483/4/fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergMetadataTable.java File fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergMetadataTable.java: http://gerrit.cloudera.org:8080/#/c/19483/4/fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergMetadataTable.java@95 PS4, Line 95: Boolean Can't we return a primitive 'boolean'? http://gerrit.cloudera.org:8080/#/c/19483/4/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java File fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java: http://gerrit.cloudera.org:8080/#/c/19483/4/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java@2216 PS4, Line 2216: that are Nit: "which is currently not supported" would be better. -- To view, visit http://gerrit.cloudera.org:8080/19483 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0b5db884b5f3fecbd132fcb2c2cbd6c622ff965b Gerrit-Change-Number: 19483 Gerrit-PatchSet: 4 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 09 Feb 2023 12:29:38 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11908: Parser change for Iceberg metadata querying
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19483 ) Change subject: IMPALA-11908: Parser change for Iceberg metadata querying .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9039/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/19483 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0b5db884b5f3fecbd132fcb2c2cbd6c622ff965b Gerrit-Change-Number: 19483 Gerrit-PatchSet: 4 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 09 Feb 2023 11:00:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11908: Parser change for Iceberg metadata querying
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19483 ) Change subject: IMPALA-11908: Parser change for Iceberg metadata querying .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12344/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19483 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0b5db884b5f3fecbd132fcb2c2cbd6c622ff965b Gerrit-Change-Number: 19483 Gerrit-PatchSet: 4 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 09 Feb 2023 10:57:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11908: Parser change for Iceberg metadata querying
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19483 ) Change subject: IMPALA-11908: Parser change for Iceberg metadata querying .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12343/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19483 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0b5db884b5f3fecbd132fcb2c2cbd6c622ff965b Gerrit-Change-Number: 19483 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 09 Feb 2023 10:53:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11908: Parser change for Iceberg metadata querying
Tamas Mate has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/19483 ) Change subject: IMPALA-11908: Parser change for Iceberg metadata querying .. IMPALA-11908: Parser change for Iceberg metadata querying This change extends parsing table references with Iceberg metadata tables. The TableName class has been extended with an extra vTbl field which is filled when a virtual table reference is suspected. This additional field helps to keep the real table in the statment table cache next to the virtual table. Which should be loaded so Iceberg metadat tables can be created. Iceberg provides a rich API to query metadata, these Iceberg API tables are accessible through the MetadataTableUtils class. Using these table schemas it is possible to create an Impala table that can be queried later on. Querying a metadata table at this point is expected to throw a NotImplementedExpception. Testing: - Added E2E test to test it for some tables. Change-Id: I0b5db884b5f3fecbd132fcb2c2cbd6c622ff965b --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/FromClause.java A fe/src/main/java/org/apache/impala/analysis/IcebergMetadataTableRef.java M fe/src/main/java/org/apache/impala/analysis/Path.java M fe/src/main/java/org/apache/impala/analysis/TableName.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTimeTravelTable.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java A fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergMetadataTable.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/IcebergSchemaConverter.java A testdata/workloads/functional-query/queries/QueryTest/iceberg-metadata-tables.test M tests/query_test/test_iceberg.py 14 files changed, 312 insertions(+), 28 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/19483/4 -- To view, visit http://gerrit.cloudera.org:8080/19483 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0b5db884b5f3fecbd132fcb2c2cbd6c622ff965b Gerrit-Change-Number: 19483 Gerrit-PatchSet: 4 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-11908: Parser change for Iceberg metadata querying
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/19483 ) Change subject: IMPALA-11908: Parser change for Iceberg metadata querying .. Patch Set 3: Finished cleaning up, I have already tested it with a verify job, but will start another one just to be sure. LMK your thoughts. -- To view, visit http://gerrit.cloudera.org:8080/19483 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0b5db884b5f3fecbd132fcb2c2cbd6c622ff965b Gerrit-Change-Number: 19483 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 09 Feb 2023 10:34:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11908: Parser change for Iceberg metadata querying
Tamas Mate has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/19483 ) Change subject: IMPALA-11908: Parser change for Iceberg metadata querying .. IMPALA-11908: Parser change for Iceberg metadata querying This change extends parsing table references with Iceberg metadata tables. The TableName class has been extended with an extra vTbl field which is filled when a virtual table reference is suspected. This additional field helps to keep the real table in the statment table cache next to the virtual table. Which should be loaded so Iceberg metadat tables can be created. Iceberg provides a rich API to query metadata, these Iceberg API tables are accessible through the MetadataTableUtils class. Using these table schemas it is possible to create an Impala table that can be queried later on. Querying a metadata table at this point is expected to throw a NotImplementedExpception. Testing: - Added E2E test to test it for some tables. Change-Id: I0b5db884b5f3fecbd132fcb2c2cbd6c622ff965b --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/FromClause.java A fe/src/main/java/org/apache/impala/analysis/IcebergMetadataTableRef.java M fe/src/main/java/org/apache/impala/analysis/Path.java M fe/src/main/java/org/apache/impala/analysis/TableName.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTimeTravelTable.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java A fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergMetadataTable.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/IcebergSchemaConverter.java A testdata/workloads/functional-query/queries/QueryTest/iceberg-metadata-tables.test M tests/query_test/test_iceberg.py 14 files changed, 311 insertions(+), 28 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/19483/3 -- To view, visit http://gerrit.cloudera.org:8080/19483 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0b5db884b5f3fecbd132fcb2c2cbd6c622ff965b Gerrit-Change-Number: 19483 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-11629: Support for huawei OBS FileSystem
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 ) Change subject: IMPALA-11629: Support for huawei OBS FileSystem .. Patch Set 12: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1 Gerrit-Change-Number: 19110 Gerrit-PatchSet: 12 Gerrit-Owner: Xiang Yang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiang Yang Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Thu, 09 Feb 2023 08:10:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11629: Support for huawei OBS FileSystem
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19110 ) Change subject: IMPALA-11629: Support for huawei OBS FileSystem .. IMPALA-11629: Support for huawei OBS FileSystem This patch adds support for huawei OBS (Object Storage Service) FileSystem. The implementation is similar to other remote FileSystems. New flags for OBS: - num_obs_io_threads: Number of OBS I/O threads. Defaults to be 16. Testing: - Upload hdfs test data to an OBS bucket. Modify all locations in HMS DB to point to the OBS bucket. Remove some hdfs caching params. Run CORE tests. Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1 Reviewed-on: http://gerrit.cloudera.org:8080/19110 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/runtime/io/disk-io-mgr-test.cc M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/disk-io-mgr.h M be/src/util/hdfs-util.cc M be/src/util/hdfs-util.h M bin/impala-config.sh M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M java/executor-deps/pom.xml M java/pom.xml M testdata/bin/create-load-data.sh M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py M tests/common/impala_test_suite.py M tests/common/skip.py M tests/custom_cluster/test_metastore_service.py M tests/util/filesystem_utils.py 15 files changed, 117 insertions(+), 18 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/19110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1 Gerrit-Change-Number: 19110 Gerrit-PatchSet: 13 Gerrit-Owner: Xiang Yang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiang Yang Gerrit-Reviewer: Yida Wu
