[Impala-ASF-CR] IMPALA-9859: Full ACID Milestone 4: Part 1 Reading modified tables (primitive types)
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/16082 )
Change subject: IMPALA-9859: Full ACID Milestone 4: Part 1 Reading modified
tables (primitive types)
..
Patch Set 9:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/16082/9/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
File fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java:
http://gerrit.cloudera.org:8080/#/c/16082/9/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java@45
PS9, Line 45: import org.apache.impala.analysis.Path.PathType;
nit: keep the import list sorted in groups (usually the IDE will do this for
you).
http://gerrit.cloudera.org:8080/#/c/16082/9/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java@1418
PS9, Line 1418: if (addAcidSlotsIfNeeded(analyzer, hdfsTblRef,
partitions)) {
nit: what about merging this if-statement with its outer scope so they are
if (isPartitionKeyScan && queryOpts.optimize_partition_key_scans) {
...
} else if (addAcidSlotsIfNeeded(analyzer, hdfsTblRef, partitions)) {
...
} else {
...
}
http://gerrit.cloudera.org:8080/#/c/16082/9/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java@1518
PS9, Line 1518: // Let's separate insert delta File Descriptors from delete
delta FDs.
I think we should separate the file descriptors in catalogd after loading them
from HDFS instead of doing it here for each query. We can introduce two fileds
in HdfsPartition: encodedInsertDeltaFileDescriptors_ and
encodedDeleteDeltaFileDescriptors_ (and related fields in THdfsPartition and
TPartialPartitionInfo). If a partition contains delete deltas, we separate them
by setting these two fields and leaving encodedFileDescriptors_ null. We can
also introduce two methods for FeFsPartition: genInsertDeltaPartition() and
genDeleteDeltaPartition() using HdfsPartition.Builder in this way:
public HdfsPartition genInsertDeltaPartition() {
return new HdfsPartition.Builder(this)
.setFileDescriptors(InsertDeltaFileDescriptors_)
.build();
}
With this we don't need to remove the "final" marker of encodedFileDescriptors_
and add back the setFileDescriptor() method, which violates our goal to make
HdfsPartition immutable. The setFileDescriptor() method may encourage future
developers to modify HdfsPartitions in-place in catalogd, which will break
IMPALA-7533.
--
To view, visit http://gerrit.cloudera.org:8080/16082
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15c8feabf40be1658f3dd46883f5a1b2aa5d0659
Gerrit-Change-Number: 16082
Gerrit-PatchSet: 9
Gerrit-Owner: Zoltan Borok-Nagy
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Wed, 01 Jul 2020 06:57:52 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP - IMPALA-9898: generate grouping set plans
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16128 ) Change subject: WIP - IMPALA-9898: generate grouping set plans .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6473/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16128 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie454c5bf7aee266321dee615548d7f2b71380197 Gerrit-Change-Number: 16128 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 01 Jul 2020 04:25:54 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP - IMPALA-9898: generate grouping set plans
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16128 ) Change subject: WIP - IMPALA-9898: generate grouping set plans .. Patch Set 6: (2 comments) http://gerrit.cloudera.org:8080/#/c/16128/5/fe/src/main/java/org/apache/impala/analysis/GroupByClause.java File fe/src/main/java/org/apache/impala/analysis/GroupByClause.java: http://gerrit.cloudera.org:8080/#/c/16128/5/fe/src/main/java/org/apache/impala/analysis/GroupByClause.java@145 PS5, Line 145: } There's a problem here that affects plan generation - we need to include grouping expressions that were added during rewrites and aren't part of the original grouping sets. http://gerrit.cloudera.org:8080/#/c/16128/5/fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java File fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java: http://gerrit.cloudera.org:8080/#/c/16128/5/fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java@227 PS5, Line 227: return; > Set isAnalyzed_ = true ? Seems analyzeGroupingSets() is not setting this f I'm relying on analyzeCustomClasses to do that, but it's kinda buried - added a comment in analyzeGroupingSets(). -- To view, visit http://gerrit.cloudera.org:8080/16128 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie454c5bf7aee266321dee615548d7f2b71380197 Gerrit-Change-Number: 16128 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 01 Jul 2020 03:59:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP - IMPALA-9898: generate grouping set plans
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16128 Change subject: WIP - IMPALA-9898: generate grouping set plans .. WIP - IMPALA-9898: generate grouping set plans Testing: * Add analysis test to make sure we reject unsupported queries. * Added targeted planner tests to ensure we generate the correct aggregation classes for a variety of cases. TODO: * Add detailed functional tests. Added five TPC-DS queries that use ROLLUP, building on some work done by Fang-Yu Rao. Some tweaks were required for these tests. * Add an extra ORDER BY clause to q77 to make fully deterministic. * Add backticks around `returns` to avoid reserved word. * Add INTERVAL keyword to date/timestamp arithmetic. We should also be able to run q27 and q80, but I didn't have those readily available. Change-Id: Ie454c5bf7aee266321dee615548d7f2b71380197 --- M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/main/java/org/apache/impala/analysis/GroupByClause.java M fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/common/RuntimeEnv.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java A testdata/workloads/functional-planner/queries/PlannerTest/grouping-sets.test M testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q18.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q22.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q5.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q67.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q77.test M tests/query_test/test_tpcds_queries.py M tests/util/parse_util.py 19 files changed, 3,623 insertions(+), 60 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/28/16128/6 -- To view, visit http://gerrit.cloudera.org:8080/16128 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ie454c5bf7aee266321dee615548d7f2b71380197 Gerrit-Change-Number: 16128 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] WIP - IMPALA-9898: generate grouping set plans
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/16128 )
Change subject: WIP - IMPALA-9898: generate grouping set plans
..
Patch Set 6:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/16128/6/fe/src/main/java/org/apache/impala/analysis/GroupByClause.java
File fe/src/main/java/org/apache/impala/analysis/GroupByClause.java:
http://gerrit.cloudera.org:8080/#/c/16128/6/fe/src/main/java/org/apache/impala/analysis/GroupByClause.java@137
PS6, Line 137: private void addGroupingID(long id, List groupingExprs,
List addtlGroupingExprs)
line too long (94 > 90)
http://gerrit.cloudera.org:8080/#/c/16128/6/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java:
http://gerrit.cloudera.org:8080/#/c/16128/6/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@2472
PS6, Line 2472: AnalysisError("select distinct int_col, bool_col, count(*)
from functional.alltypes " +
line too long (91 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/16128
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie454c5bf7aee266321dee615548d7f2b71380197
Gerrit-Change-Number: 16128
Gerrit-PatchSet: 6
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: David Rorke
Gerrit-Reviewer: Fang-Yu Rao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 01 Jul 2020 03:57:58 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9913: Use table id to match the table in drop table event
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16129 ) Change subject: IMPALA-9913: Use table id to match the table in drop table event .. IMPALA-9913: Use table id to match the table in drop table event This patch modifies the matching logic of the table while processing a DROP_TABLE event. The EventsProcessor currently relies on createTime of the table to determine if the table in the catalog matches with the table in the event. It is possible that if there is a time-skew between the different HMS instances and a same table is dropped and recreated, the creation time of the table which is present in the catalog is less than or equals to the creationTime of a table which has been dropped. In such case, there is a false negative during the matching and the DROP_TABLE event is skipped. HIVE-20556 exposes a id field of the table which can be used to uniquely identify a table. A table which is dropped and recreated with the same name has a different id. This patch makes use of this id to match the table in catalog with the table from the HMS event. Note that the DROP_DATABASE event continues to use createTime for matching. We should use a database id for it when it is available. I have created HIVE-23785 for tracking that on the HMS side. Testing: 1. Ran Events processor tests. Change-Id: Id6a80bbf5757e46318af1b57911fc127d7dd1f01 Reviewed-on: http://gerrit.cloudera.org:8080/16129 Reviewed-by: Anurag Mantripragada Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 2 files changed, 10 insertions(+), 10 deletions(-) Approvals: Anurag Mantripragada: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/16129 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Id6a80bbf5757e46318af1b57911fc127d7dd1f01 Gerrit-Change-Number: 16129 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9913: Use table id to match the table in drop table event
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16129 ) Change subject: IMPALA-9913: Use table id to match the table in drop table event .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16129 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id6a80bbf5757e46318af1b57911fc127d7dd1f01 Gerrit-Change-Number: 16129 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 01 Jul 2020 00:16:52 + Gerrit-HasComments: No
[native-toolchain-CR] Bump Kudu version to 23f67ae0d
Bankim Bhavsar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16130 ) Change subject: Bump Kudu version to 23f67ae0d .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/16130 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I036889f91a0901ff09fce31069bdb0bf96b7bb1e Gerrit-Change-Number: 16130 Gerrit-PatchSet: 1 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Bankim Bhavsar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 30 Jun 2020 23:01:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9692 (part 3): Model QuerySchedule as a protobuf
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15961 ) Change subject: IMPALA-9692 (part 3): Model QuerySchedule as a protobuf .. Patch Set 10: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15961 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1db64e72f84604b1d8ac24e0bdd4ad6bedd6bcd9 Gerrit-Change-Number: 15961 Gerrit-PatchSet: 10 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 30 Jun 2020 22:06:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9692 (part 3): Model QuerySchedule as a protobuf
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15961 ) Change subject: IMPALA-9692 (part 3): Model QuerySchedule as a protobuf .. IMPALA-9692 (part 3): Model QuerySchedule as a protobuf In order to support the new admission control service, we need to be able to return the results of an admission attempt, i.e. the query schedule, to the coordinator. To enable this, this patch moves all parts of the QuerySchedule class and related classes that are required by the coordinator into a new message QuerySchedulePB. The main admission control interface, SubmitForAdmission(), now returns a QuerySchedulePB. Some notable changes: - Previously, QuerySchedule was used by Coordinator as a way to pass around references to parts of the TExecRequest to places like Coordinator::ExecSummary and Coordinator::BackendState. This has been replaced with the ExecParams class, which is a container for references to the TExecRequest and QuerySchedulePB along with convenience functions for accessing them. - Similarly, FragmentExecParams, which is part of QuerySchedule, contains references to the associated TPlanFragment, owned by the TExecRequest, which were used by the Coordinator when iterating over the schedule to initiate the query. Since FragmentExecParamsPB can't contain these references, they were replaced by a map between fragment idx and TPlanFragment in ExecParams. - In order to keep payloads reasonable for the eventual RPC interface, AdmissionController::ReleaseQuery() and ReleaseQueryBackend() now take a query id as a parameter instead of a QuerySchedule. To facilitate this, AdmissionController now maintains a map from query ids of running queries to the resources that were allocated for them so that it can look the resources up when releasing them. This map will be necessary when implementing the admission control service to facilitate proper accounting of resouces in cases like coordinator failures. - As scheduling is currently organized, we first construct the FragmentExecParams with the FInstanceExecParams as their children, then we construct the BackendExecParams which get references to their FInstanceExecParams. Since we can't send references like these through an rpc, we now instead Swap() the FInstanceExecParamsPB into the BackendExecParamsPB. Testing: - Updated related tests. - Passed a full run of existing tests. Change-Id: I1db64e72f84604b1d8ac24e0bdd4ad6bedd6bcd9 Reviewed-on: http://gerrit.cloudera.org:8080/15961 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/runtime/CMakeLists.txt M be/src/runtime/coordinator-backend-resource-state.cc M be/src/runtime/coordinator-backend-state-test.cc M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/runtime/fragment-instance-state.cc M be/src/runtime/query-driver.cc A be/src/runtime/query-exec-params.cc A be/src/runtime/query-exec-params.h M be/src/scheduling/CMakeLists.txt M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/cluster-membership-mgr-test.cc M be/src/scheduling/cluster-membership-mgr.cc M be/src/scheduling/cluster-membership-mgr.h M be/src/scheduling/cluster-membership-test-util.cc M be/src/scheduling/query-schedule.cc M be/src/scheduling/query-schedule.h M be/src/scheduling/scheduler-test-util.cc M be/src/scheduling/scheduler-test-util.h M be/src/scheduling/scheduler.cc M be/src/scheduling/scheduler.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-http-handler.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/util/debug-util.h M be/src/util/uid-util.h M common/protobuf/CMakeLists.txt A common/protobuf/admission_control_service.proto 34 files changed, 1,082 insertions(+), 776 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/15961 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I1db64e72f84604b1d8ac24e0bdd4ad6bedd6bcd9 Gerrit-Change-Number: 15961 Gerrit-PatchSet: 11 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-9540 Test that Impala Shell no longer sends duplicate "Host" headers in http mode.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15752 ) Change subject: IMPALA-9540 Test that Impala Shell no longer sends duplicate "Host" headers in http mode. .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15752 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I82996015d0205923e854dac8bb88604778684c46 Gerrit-Change-Number: 15752 Gerrit-PatchSet: 6 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 30 Jun 2020 21:46:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9540 Test that Impala Shell no longer sends duplicate "Host" headers in http mode.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15752 ) Change subject: IMPALA-9540 Test that Impala Shell no longer sends duplicate "Host" headers in http mode. .. IMPALA-9540 Test that Impala Shell no longer sends duplicate "Host" headers in http mode. Many http servers will not accept an http request that has multiple copies of the "Host" header. A recent toolchain change patches Thrift so that will not send the extraneous header (in THttpClient). This change tests that the duplicate headers are not sent, TESTING: Ran all end-to-end tests. Rewrote an existing Shell test to check that only one "Host" header is sent. Change-Id: I82996015d0205923e854dac8bb88604778684c46 Reviewed-on: http://gerrit.cloudera.org:8080/15752 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M tests/shell/test_shell_interactive.py M tests/shell/util.py 2 files changed, 54 insertions(+), 35 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/15752 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I82996015d0205923e854dac8bb88604778684c46 Gerrit-Change-Number: 15752 Gerrit-PatchSet: 7 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9897: GROUPING SETS/CUBE/ROLLUP parsing and analysis
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16112 ) Change subject: IMPALA-9897: GROUPING SETS/CUBE/ROLLUP parsing and analysis .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6472/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16112 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I178e45de94d736630c97ae1ec4a92423cd74621f Gerrit-Change-Number: 16112 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Jun 2020 21:30:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9294: Support DATE for min-max runtime filter
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16103 ) Change subject: IMPALA-9294: Support DATE for min-max runtime filter .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6471/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic2f6e2dc6949735d5f0fcf317361cc2969a5e82c Gerrit-Change-Number: 16103 Gerrit-PatchSet: 6 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 30 Jun 2020 21:17:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9897: GROUPING SETS/CUBE/ROLLUP parsing and analysis
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16112 ) Change subject: IMPALA-9897: GROUPING SETS/CUBE/ROLLUP parsing and analysis .. Patch Set 11: > LGTM, nice approach of keeping the groupingExprs_ in the select stmt. I copied that from you ;) -- To view, visit http://gerrit.cloudera.org:8080/16112 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I178e45de94d736630c97ae1ec4a92423cd74621f Gerrit-Change-Number: 16112 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Jun 2020 21:08:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9897: GROUPING SETS/CUBE/ROLLUP parsing and analysis
Hello Aman Sinha, Shant Hovsepian, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16112 to look at the new patch set (#11). Change subject: IMPALA-9897: GROUPING SETS/CUBE/ROLLUP parsing and analysis .. IMPALA-9897: GROUPING SETS/CUBE/ROLLUP parsing and analysis Supports a single ROLLUP, CUBE or GROUPING SETS clause in GROUP BY. Also adds non-standard "WITH ROLLUP" and "WITH CUBE" syntax that is supported by some other SQL dialects. This implements basic parsing and validation of the query, then raises an AnalysisException to report that it is not supported so that incorrect plans will not be generated. This patch adds a GroupByClause to each SelectStmt that contains info about the grouping sets and the original GROUP BY list. The grouping exprs are still represented as a List in SelectStmt. Most of the logic, including statement and expr rewrites, can operate on this list of expressions without requiring special handling for grouping sets. Testing: * Add Parser test. * Add toSql() test. * Added analysis tests to check that analysis accepts or rejects queries correctly. Change-Id: I178e45de94d736630c97ae1ec4a92423cd74621f --- M fe/src/main/cup/sql-parser.cup A fe/src/main/java/org/apache/impala/analysis/GroupByClause.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java M fe/src/main/java/org/apache/impala/common/RuntimeEnv.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java 10 files changed, 741 insertions(+), 23 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/16112/11 -- To view, visit http://gerrit.cloudera.org:8080/16112 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I178e45de94d736630c97ae1ec4a92423cd74621f Gerrit-Change-Number: 16112 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9897: GROUPING SETS/CUBE/ROLLUP parsing and analysis
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/16112 )
Change subject: IMPALA-9897: GROUPING SETS/CUBE/ROLLUP parsing and analysis
..
Patch Set 10:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/16112/10/fe/src/main/cup/sql-parser.cup
File fe/src/main/cup/sql-parser.cup:
http://gerrit.cloudera.org:8080/#/c/16112/10/fe/src/main/cup/sql-parser.cup@3076
PS10, Line 3076: // * GROUP BY a, b, c WITH CUBE - non-standard CUBE syntax
supported by some systems
> Add example of GROUP BY GROUPING SETS ?
Done
http://gerrit.cloudera.org:8080/#/c/16112/10/fe/src/main/cup/sql-parser.cup@3076
PS10, Line 3076: // * GROUP BY a, b, c WITH CUBE - non-standard CUBE syntax
supported by some systems
> Add example of GROUP BY GROUPING SETS ?
Done
http://gerrit.cloudera.org:8080/#/c/16112/10/fe/src/main/java/org/apache/impala/analysis/GroupByClause.java
File fe/src/main/java/org/apache/impala/analysis/GroupByClause.java:
http://gerrit.cloudera.org:8080/#/c/16112/10/fe/src/main/java/org/apache/impala/analysis/GroupByClause.java@121
PS10, Line 121: private void addGroupingID(long id, List groupingExprs)
throws AnalysisException {
> Nit but might be cleaner to use the JDK BitSet class especially down the li
It's probably easier to leave as-is, if only because enumerating classes for
CUBE is way more concise using additional on an integer.
I added some comments to explain that bit manipulation logic to make it less
cryptic.
--
To view, visit http://gerrit.cloudera.org:8080/16112
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I178e45de94d736630c97ae1ec4a92423cd74621f
Gerrit-Change-Number: 16112
Gerrit-PatchSet: 10
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Shant Hovsepian
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Tue, 30 Jun 2020 21:03:21 +
Gerrit-HasComments: Yes
[native-toolchain-CR] Bump Kudu version to 23f67ae0d
Wenzhe Zhou has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16130 Change subject: Bump Kudu version to 23f67ae0d .. Bump Kudu version to 23f67ae0d This pulls in the latest version of Kudu which includes a performance enhancement in Kudu server for Bloom filter, which uses heuristics to disable predicate evaluation for Bloom filter. Testing: - Built native-toolchain on local machine with Kudu version of 23f67ae0d, and built Impala with new version of Kudu from native- toolchain built on local machine, then started mini-cluster with the new version of Kudu server. Ran single_node_perf_run.py with TPCH for Kudu and verified that the regression issue on TPCH-Q9 was fixed. - Passed test_kudu.py and test_runtime_filters.py against new version of Kudu server. - Successfully built native-toolchain on Jenkins without publishing dependencies to S3. Change-Id: I036889f91a0901ff09fce31069bdb0bf96b7bb1e --- M buildall.sh 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/30/16130/1 -- To view, visit http://gerrit.cloudera.org:8080/16130 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I036889f91a0901ff09fce31069bdb0bf96b7bb1e Gerrit-Change-Number: 16130 Gerrit-PatchSet: 1 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-9897: GROUPING SETS/CUBE/ROLLUP parsing and analysis
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16112 ) Change subject: IMPALA-9897: GROUPING SETS/CUBE/ROLLUP parsing and analysis .. Patch Set 10: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16112 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I178e45de94d736630c97ae1ec4a92423cd74621f Gerrit-Change-Number: 16112 Gerrit-PatchSet: 10 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Jun 2020 20:59:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9294: Support DATE for min-max runtime filter
Wenzhe Zhou has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/16103 ) Change subject: IMPALA-9294: Support DATE for min-max runtime filter .. IMPALA-9294: Support DATE for min-max runtime filter Implemented Date min-max filter and applied it to Kudu as other min-max runtime filters. Added new test cases for Date min-max filters. Testing: Passed all core tests. Change-Id: Ic2f6e2dc6949735d5f0fcf317361cc2969a5e82c --- M be/src/codegen/gen_ir_descriptions.py M be/src/runtime/date-value.h M be/src/util/min-max-filter-ir.cc M be/src/util/min-max-filter-test.cc M be/src/util/min-max-filter.cc M be/src/util/min-max-filter.h M common/protobuf/common.proto M common/thrift/Data.thrift M fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java M testdata/workloads/functional-planner/queries/PlannerTest/min-max-runtime-filters.test M testdata/workloads/functional-query/queries/QueryTest/all_runtime_filters.test M testdata/workloads/functional-query/queries/QueryTest/min_max_filters.test 12 files changed, 376 insertions(+), 167 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/16103/6 -- To view, visit http://gerrit.cloudera.org:8080/16103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic2f6e2dc6949735d5f0fcf317361cc2969a5e82c Gerrit-Change-Number: 16103 Gerrit-PatchSet: 6 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-9913: Use table id to match the table in drop table event
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16129 ) Change subject: IMPALA-9913: Use table id to match the table in drop table event .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6470/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16129 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id6a80bbf5757e46318af1b57911fc127d7dd1f01 Gerrit-Change-Number: 16129 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 30 Jun 2020 19:35:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9907: Fix NullPointerException in ParallelFileMetadataLoader's load() method
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16121 ) Change subject: IMPALA-9907: Fix NullPointerException in ParallelFileMetadataLoader's load() method .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16121 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4322a18a1f07e4106789591b80d25105c565a2b6 Gerrit-Change-Number: 16121 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 30 Jun 2020 19:33:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9907: Fix NullPointerException in ParallelFileMetadataLoader's load() method
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16121 ) Change subject: IMPALA-9907: Fix NullPointerException in ParallelFileMetadataLoader's load() method .. IMPALA-9907: Fix NullPointerException in ParallelFileMetadataLoader's load() method ParallelFileMetadataLoader.loaders_ became a Map, but at one place we still treat it as a List: loaders_.get(i).getPartDir() 'i' is an integer, while the loaders_'s key type is Path. Unfortunately the code compiled which means we got a NullPointerException for this statement. I changed the logic to avoid the NPE. Testing: * Tested manually on an ACID table with deleted rows * I didn't add an automated test because I'm about to extend the ACID functionalities, hence getting rid of these exceptions * The other case when we can hit this NPE is when we get an IOException in the file metadata loader's load() method Change-Id: I4322a18a1f07e4106789591b80d25105c565a2b6 Reviewed-on: http://gerrit.cloudera.org:8080/16121 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java 1 file changed, 7 insertions(+), 4 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16121 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I4322a18a1f07e4106789591b80d25105c565a2b6 Gerrit-Change-Number: 16121 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-9913: Use table id to match the table in drop table event
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16129 ) Change subject: IMPALA-9913: Use table id to match the table in drop table event .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6469/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16129 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id6a80bbf5757e46318af1b57911fc127d7dd1f01 Gerrit-Change-Number: 16129 Gerrit-PatchSet: 1 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 30 Jun 2020 19:32:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9913: Use table id to match the table in drop table event
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16129 ) Change subject: IMPALA-9913: Use table id to match the table in drop table event .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6083/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16129 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id6a80bbf5757e46318af1b57911fc127d7dd1f01 Gerrit-Change-Number: 16129 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 30 Jun 2020 19:23:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9913: Use table id to match the table in drop table event
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/16129 ) Change subject: IMPALA-9913: Use table id to match the table in drop table event .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16129 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id6a80bbf5757e46318af1b57911fc127d7dd1f01 Gerrit-Change-Number: 16129 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 30 Jun 2020 19:22:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9913: Use table id to match the table in drop table event
Vihang Karajgaonkar has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/16129 ) Change subject: IMPALA-9913: Use table id to match the table in drop table event .. IMPALA-9913: Use table id to match the table in drop table event This patch modifies the matching logic of the table while processing a DROP_TABLE event. The EventsProcessor currently relies on createTime of the table to determine if the table in the catalog matches with the table in the event. It is possible that if there is a time-skew between the different HMS instances and a same table is dropped and recreated, the creation time of the table which is present in the catalog is less than or equals to the creationTime of a table which has been dropped. In such case, there is a false negative during the matching and the DROP_TABLE event is skipped. HIVE-20556 exposes a id field of the table which can be used to uniquely identify a table. A table which is dropped and recreated with the same name has a different id. This patch makes use of this id to match the table in catalog with the table from the HMS event. Note that the DROP_DATABASE event continues to use createTime for matching. We should use a database id for it when it is available. I have created HIVE-23785 for tracking that on the HMS side. Testing: 1. Ran Events processor tests. Change-Id: Id6a80bbf5757e46318af1b57911fc127d7dd1f01 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 2 files changed, 10 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/16129/2 -- To view, visit http://gerrit.cloudera.org:8080/16129 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id6a80bbf5757e46318af1b57911fc127d7dd1f01 Gerrit-Change-Number: 16129 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9913: Use table id to match the table in drop table event
Vihang Karajgaonkar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16129 Change subject: IMPALA-9913: Use table id to match the table in drop table event .. IMPALA-9913: Use table id to match the table in drop table event This patch modifies the matching logic of the table while processing a DROP_TABLE event. The EventsProcessor currently relies on createTime of the table to determine if the table in the catalog matching with the table in the event. It is possible that if there is a time-skew between the different HMS instances and a same table is dropped and recreated, the the creation time of the table which is present in the catalog is less than or equals to the creationTime of a table which is dropped. In such case, there is a false negative during the matching and the DROP_TABLE event is skipped. HIVE-20556 exposes a id field of the table in which can be used to uniquely identify a table. A table which is dropped and recreated with the same name has a different id. This patch makes use of this id to match the table in catalog with the table from the HMS event. Note that the DROP_DATABASE event continues to use createTime for matching. We should use a database id for it when it is available. I have created HIVE-23785 for tracking that on the HMS side. Testing: 1. Ran Events processor tests. Change-Id: Id6a80bbf5757e46318af1b57911fc127d7dd1f01 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 2 files changed, 10 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/16129/1 -- To view, visit http://gerrit.cloudera.org:8080/16129 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id6a80bbf5757e46318af1b57911fc127d7dd1f01 Gerrit-Change-Number: 16129 Gerrit-PatchSet: 1 Gerrit-Owner: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-9692 (part 4): Rename QuerySchedule to ScheduleState
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16122 ) Change subject: IMPALA-9692 (part 4): Rename QuerySchedule to ScheduleState .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6468/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16122 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I57069c4a426f3e697df7e2a07754d063bdea26f7 Gerrit-Change-Number: 16122 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 30 Jun 2020 18:22:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9692 (part 4): Rename QuerySchedule to ScheduleState
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16122 to look at the new patch set (#3). Change subject: IMPALA-9692 (part 4): Rename QuerySchedule to ScheduleState .. IMPALA-9692 (part 4): Rename QuerySchedule to ScheduleState This is the final patch in the refactor of QuerySchedule for the single admission controller work. It was kept separate for ease of reviewing. This patch renames QuerySchedule and its related classes (FInstanceExecParams, BackendExecParams, and FragmentExecParams) to use the name 'ScheduleState', reflecting the fact that they are used as containers for intermediate info about a query during scheduling. The messages that are included in the QuerySchedulePB struct retain the 'ExecParams' name, reflecting the fact that they are used by the coordinator to start execution. Change-Id: I57069c4a426f3e697df7e2a07754d063bdea26f7 --- M be/src/scheduling/CMakeLists.txt M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h R be/src/scheduling/schedule-state.cc R be/src/scheduling/schedule-state.h M be/src/scheduling/scheduler-test-util.h M be/src/scheduling/scheduler.cc M be/src/scheduling/scheduler.h 9 files changed, 631 insertions(+), 625 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/16122/3 -- To view, visit http://gerrit.cloudera.org:8080/16122 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I57069c4a426f3e697df7e2a07754d063bdea26f7 Gerrit-Change-Number: 16122 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-9898: Plan generation and execution for grouping sets
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16115 ) Change subject: IMPALA-9898: Plan generation and execution for grouping sets .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6467/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16115 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id474c5373860b0d8014ee9c844a3fb90092be968 Gerrit-Change-Number: 16115 Gerrit-PatchSet: 3 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Jun 2020 17:34:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9692 (part 4): Rename QuerySchedule to ScheduleState
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16122 ) Change subject: IMPALA-9692 (part 4): Rename QuerySchedule to ScheduleState .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6466/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16122 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I57069c4a426f3e697df7e2a07754d063bdea26f7 Gerrit-Change-Number: 16122 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 30 Jun 2020 17:31:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9898: Plan generation and execution for grouping sets
Aman Sinha has posted comments on this change. (
http://gerrit.cloudera.org:8080/16115 )
Change subject: IMPALA-9898: Plan generation and execution for grouping sets
..
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/16115/2/fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java
File fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java:
http://gerrit.cloudera.org:8080/#/c/16115/2/fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java@544
PS2, Line 544: if (aggInfo.getGroupingExprs().size() == 0) {
> I think it would be more efficient to support not providing the NULL groupi
Yes, the NULL exprs are included in the intermediate tuples. I felt that
getting the positional index right (for grouping_id/grouping function support)
could be prone to bugs if all grouping sets did not have the fixed number of
exprs. I have left this as-is for now in the latest patchset but if needed can
do some refactoring as Part-2 based on the integration with your patch
--
To view, visit http://gerrit.cloudera.org:8080/16115
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id474c5373860b0d8014ee9c844a3fb90092be968
Gerrit-Change-Number: 16115
Gerrit-PatchSet: 3
Gerrit-Owner: Aman Sinha
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Tue, 30 Jun 2020 17:25:32 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9898: Plan generation and execution for grouping sets
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16115 to look at the new patch set (#3). Change subject: IMPALA-9898: Plan generation and execution for grouping sets .. IMPALA-9898: Plan generation and execution for grouping sets This patch enhances the MultiAggregateInfo to handle grouping sets and rollup (which is converted to grouping sets). This patch does not itself do parsing/validation of grouping sets syntax but rather provides the following supporting functionality: - A separate analyze method that accepts aggregation classes and aggregation info that have been created separately. - A modified Transpose phase that uses combination of aggif(), valid_tid() functions and CASE exprs to choose exactly which slots from the underlying aggregate classes need to be output based on the tuple id. - Modified materialization step where all aggregate slots and grouping slots are materialized in case of grouping sets. - Creates grouping_id value for grouping sets. The grouping_id function in SQL describes which expression is grouped-by in a particular row of a query with grouping sets. Testing: This patch is not individually testable but will be tested as part of the overall grouping set support. Change-Id: Id474c5373860b0d8014ee9c844a3fb90092be968 --- M fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java 1 file changed, 266 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/16115/3 -- To view, visit http://gerrit.cloudera.org:8080/16115 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id474c5373860b0d8014ee9c844a3fb90092be968 Gerrit-Change-Number: 16115 Gerrit-PatchSet: 3 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9692 (part 3): Model QuerySchedule as a protobuf
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15961 ) Change subject: IMPALA-9692 (part 3): Model QuerySchedule as a protobuf .. Patch Set 10: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15961 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1db64e72f84604b1d8ac24e0bdd4ad6bedd6bcd9 Gerrit-Change-Number: 15961 Gerrit-PatchSet: 10 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 30 Jun 2020 17:03:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9692 (part 3): Model QuerySchedule as a protobuf
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15961 ) Change subject: IMPALA-9692 (part 3): Model QuerySchedule as a protobuf .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6082/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15961 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1db64e72f84604b1d8ac24e0bdd4ad6bedd6bcd9 Gerrit-Change-Number: 15961 Gerrit-PatchSet: 10 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 30 Jun 2020 17:03:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9692 (part 3): Model QuerySchedule as a protobuf
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/15961 ) Change subject: IMPALA-9692 (part 3): Model QuerySchedule as a protobuf .. Patch Set 9: Code-Review+2 (1 comment) rebased, carrying forward http://gerrit.cloudera.org:8080/#/c/15961/4/be/src/scheduling/query-schedule.h File be/src/scheduling/query-schedule.h: http://gerrit.cloudera.org:8080/#/c/15961/4/be/src/scheduling/query-schedule.h@58 PS4, Line 58: er > JIRA? https://gerrit.cloudera.org/#/c/16122/ -- To view, visit http://gerrit.cloudera.org:8080/15961 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1db64e72f84604b1d8ac24e0bdd4ad6bedd6bcd9 Gerrit-Change-Number: 15961 Gerrit-PatchSet: 9 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 30 Jun 2020 17:03:08 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9692 (part 4): Rename QuerySchedule to ScheduleState
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/16122 ) Change subject: IMPALA-9692 (part 4): Rename QuerySchedule to ScheduleState .. Patch Set 2: (18 comments) http://gerrit.cloudera.org:8080/#/c/16122/1/be/src/scheduling/admission-controller-test.cc File be/src/scheduling/admission-controller-test.cc: http://gerrit.cloudera.org:8080/#/c/16122/1/be/src/scheduling/admission-controller-test.cc@131 PS1, Line 131: backend_schedule_state.exec_params->set_min_mem_reservation_bytes( > line too long (99 > 90) Done http://gerrit.cloudera.org:8080/#/c/16122/1/be/src/scheduling/admission-controller.h File be/src/scheduling/admission-controller.h: http://gerrit.cloudera.org:8080/#/c/16122/1/be/src/scheduling/admission-controller.h@246 PS1, Line 246: /// afterwards call CanAdmitRequest() for each of the schedules. Executor groups are > line too long (91 > 90) Done http://gerrit.cloudera.org:8080/#/c/16122/1/be/src/scheduling/admission-controller.h@993 PS1, Line 993: /// Backend is an Executor or a Coordinator. > line too long (92 > 90) Done http://gerrit.cloudera.org:8080/#/c/16122/1/be/src/scheduling/admission-controller.cc File be/src/scheduling/admission-controller.cc: http://gerrit.cloudera.org:8080/#/c/16122/1/be/src/scheduling/admission-controller.cc@675 PS1, Line 675: largest_min_mem_reservation = > line too long (91 > 90) Done http://gerrit.cloudera.org:8080/#/c/16122/1/be/src/scheduling/schedule-state.h File be/src/scheduling/schedule-state.h: http://gerrit.cloudera.org:8080/#/c/16122/1/be/src/scheduling/schedule-state.h@61 PS1, Line 61: explicit BackendScheduleState(BackendExecParamsPB* exec_params) > line too long (95 > 90) Done http://gerrit.cloudera.org:8080/#/c/16122/1/be/src/scheduling/schedule-state.h@65 PS1, Line 65: /// Map from an impalad backend address to the exec params for that backend. > line too long (91 > 90) Done http://gerrit.cloudera.org:8080/#/c/16122/1/be/src/scheduling/schedule-state.h@83 PS1, Line 83: /// Contains any info that needs to be sent back to the coordinator. Computed during > line too long (94 > 90) Done http://gerrit.cloudera.org:8080/#/c/16122/1/be/src/scheduling/schedule-state.h@134 PS1, Line 134: /// pointers to corresponding FragmentExecParamsPBs created in the ExecParamsPB. > line too long (91 > 90) Done http://gerrit.cloudera.org:8080/#/c/16122/1/be/src/scheduling/schedule-state.h@210 PS1, Line 210: return per_backend_schedule_states_; > line too long (97 > 90) Done http://gerrit.cloudera.org:8080/#/c/16122/1/be/src/scheduling/schedule-state.cc File be/src/scheduling/schedule-state.cc: http://gerrit.cloudera.org:8080/#/c/16122/1/be/src/scheduling/schedule-state.cc@163 PS1, Line 163: count_map.insert( > line too long (98 > 90) Done http://gerrit.cloudera.org:8080/#/c/16122/1/be/src/scheduling/schedule-state.cc@204 PS1, Line 204: const BackendScheduleState& bp = elem.second; > line too long (93 > 90) Done http://gerrit.cloudera.org:8080/#/c/16122/1/be/src/scheduling/scheduler.h File be/src/scheduling/scheduler.h: http://gerrit.cloudera.org:8080/#/c/16122/1/be/src/scheduling/scheduler.h@355 PS1, Line 355: /// Computes execution parameters for all backends assigned in the query and always one > line too long (93 > 90) Done http://gerrit.cloudera.org:8080/#/c/16122/1/be/src/scheduling/scheduler.h@413 PS1, Line 413: const std::vector& scan_ids, > line too long (93 > 90) Done http://gerrit.cloudera.org:8080/#/c/16122/1/be/src/scheduling/scheduler.cc File be/src/scheduling/scheduler.cc: http://gerrit.cloudera.org:8080/#/c/16122/1/be/src/scheduling/scheduler.cc@224 PS1, Line 224: FragmentScheduleState* src_params = > line too long (92 > 90) Done http://gerrit.cloudera.org:8080/#/c/16122/1/be/src/scheduling/scheduler.cc@828 PS1, Line 828: coord_min_reservation = coord_be_state.exec_params->min_mem_reservation_bytes(); > line too long (92 > 90) Done http://gerrit.cloudera.org:8080/#/c/16122/1/be/src/scheduling/scheduler.cc@842 PS1, Line 842: } > line too long (96 > 90) Done http://gerrit.cloudera.org:8080/#/c/16122/1/be/src/scheduling/scheduler.cc@853 PS1, Line 853:<< ") "; > line too long (98 > 90) Done http://gerrit.cloudera.org:8080/#/c/16122/1/be/src/scheduling/scheduler.cc@857 PS1, Line 857: TUnit::UNIT) > line too long (94 > 90) Done -- To view, visit http://gerrit.cloudera.org:8080/16122 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I57069c4a426f3e697df7e2a07754d063bdea26f7 Gerrit-Change-Number: 16122 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 30 Jun 2020 17
[Impala-ASF-CR] IMPALA-9692 (part 4): Rename QuerySchedule to ScheduleState
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16122 to look at the new patch set (#2). Change subject: IMPALA-9692 (part 4): Rename QuerySchedule to ScheduleState .. IMPALA-9692 (part 4): Rename QuerySchedule to ScheduleState This is the final patch in the refactor of QuerySchedule for the single admission controller work. It was kept separate for ease of reviewing. This patch renames QuerySchedule and its related classes (FInstanceExecParams, BackendExecParams, and FragmentExecParams) to use the name 'ScheduleState', reflecting the fact that they are used as containers for intermediate info about a query during scheduling. The messages that are included in the QuerySchedulePB struct retain the 'ExecParams' name, reflecting the fact that they are used by the coordinator to start execution. Change-Id: I57069c4a426f3e697df7e2a07754d063bdea26f7 --- M .clang-format M be/src/scheduling/CMakeLists.txt M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h R be/src/scheduling/schedule-state.cc R be/src/scheduling/schedule-state.h M be/src/scheduling/scheduler-test-util.h M be/src/scheduling/scheduler.cc M be/src/scheduling/scheduler.h 10 files changed, 616 insertions(+), 611 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/16122/2 -- To view, visit http://gerrit.cloudera.org:8080/16122 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I57069c4a426f3e697df7e2a07754d063bdea26f7 Gerrit-Change-Number: 16122 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9897: GROUPING SETS/CUBE/ROLLUP parsing and analysis
Shant Hovsepian has posted comments on this change. (
http://gerrit.cloudera.org:8080/16112 )
Change subject: IMPALA-9897: GROUPING SETS/CUBE/ROLLUP parsing and analysis
..
Patch Set 10: Code-Review+1
(2 comments)
LGTM, nice approach of keeping the groupingExprs_ in the select stmt.
http://gerrit.cloudera.org:8080/#/c/16112/10/fe/src/main/cup/sql-parser.cup
File fe/src/main/cup/sql-parser.cup:
http://gerrit.cloudera.org:8080/#/c/16112/10/fe/src/main/cup/sql-parser.cup@3076
PS10, Line 3076: // * GROUP BY a, b, c WITH CUBE - non-standard CUBE syntax
supported by some systems
Add example of GROUP BY GROUPING SETS ?
http://gerrit.cloudera.org:8080/#/c/16112/10/fe/src/main/java/org/apache/impala/analysis/GroupByClause.java
File fe/src/main/java/org/apache/impala/analysis/GroupByClause.java:
http://gerrit.cloudera.org:8080/#/c/16112/10/fe/src/main/java/org/apache/impala/analysis/GroupByClause.java@121
PS10, Line 121: private void addGroupingID(long id, List groupingExprs)
throws AnalysisException {
Nit but might be cleaner to use the JDK BitSet class especially down the line
when filling out the multiagg info.
--
To view, visit http://gerrit.cloudera.org:8080/16112
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I178e45de94d736630c97ae1ec4a92423cd74621f
Gerrit-Change-Number: 16112
Gerrit-PatchSet: 10
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Shant Hovsepian
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Tue, 30 Jun 2020 16:52:03 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9540 Test that Impala Shell no longer sends duplicate "Host" headers in http mode.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15752 ) Change subject: IMPALA-9540 Test that Impala Shell no longer sends duplicate "Host" headers in http mode. .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15752 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I82996015d0205923e854dac8bb88604778684c46 Gerrit-Change-Number: 15752 Gerrit-PatchSet: 6 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 30 Jun 2020 16:45:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9540 Test that Impala Shell no longer sends duplicate "Host" headers in http mode.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15752 ) Change subject: IMPALA-9540 Test that Impala Shell no longer sends duplicate "Host" headers in http mode. .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6081/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15752 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I82996015d0205923e854dac8bb88604778684c46 Gerrit-Change-Number: 15752 Gerrit-PatchSet: 6 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 30 Jun 2020 16:45:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9540 Test that Impala Shell no longer sends duplicate "Host" headers in http mode.
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/15752 ) Change subject: IMPALA-9540 Test that Impala Shell no longer sends duplicate "Host" headers in http mode. .. Patch Set 5: Code-Review+2 Bring forward +2 I rebased and ran all tests -- To view, visit http://gerrit.cloudera.org:8080/15752 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I82996015d0205923e854dac8bb88604778684c46 Gerrit-Change-Number: 15752 Gerrit-PatchSet: 5 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 30 Jun 2020 16:42:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9897: GROUPING SETS/CUBE/ROLLUP parsing and analysis
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16112 ) Change subject: IMPALA-9897: GROUPING SETS/CUBE/ROLLUP parsing and analysis .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6080/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/16112 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I178e45de94d736630c97ae1ec4a92423cd74621f Gerrit-Change-Number: 16112 Gerrit-PatchSet: 10 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Jun 2020 15:59:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5444: Asynchronous code generation
Csaba Ringhofer has posted comments on this change. (
http://gerrit.cloudera.org:8080/15105 )
Change subject: IMPALA-5444: Asynchronous code generation
..
Patch Set 43: Code-Review+2
(5 comments)
I have a few optional comments about the tests.
Great work!
http://gerrit.cloudera.org:8080/#/c/15105/43/tests/query_test/test_async_codegen.py
File tests/query_test/test_async_codegen.py:
http://gerrit.cloudera.org:8080/#/c/15105/43/tests/query_test/test_async_codegen.py@83
PS43, Line 83:
cls.ImpalaTestMatrix.add_dimension(create_exec_option_dimension(
: cluster_sizes=[1],
: disable_codegen_options=[False],
: batch_sizes=[0],
: disable_codegen_rows_threshold_options=[0],
:
debug_action_options=[cls.DEBUG_ACTION_CODEGEN_FINISH_BEFORE_EXEC_START,
: cls.DEBUG_ACTION_CODEGEN_FINISH_DURING_EXEC,
: cls.DEBUG_ACTION_EXEC_FINISH_BEFORE_CODEGEN]))
maybe it could be simpler to run the query with the 3 different debug actions
in test_async_codegen instead of creating an extra dimension
http://gerrit.cloudera.org:8080/#/c/15105/43/tests/query_test/test_async_codegen.py@136
PS43, Line 136: def __find_event_sequence(self, profile):
: # The lines corresponding to the events in the event
sequence.
: events = []
:
: # The number of leading whitespace in the lines containing
the events, used to
: # detect the line after the last event.
: indent_len = None
:
: # Set to true when encountering the header before the first
event. This means the
: # following lines contain the events.
: found_events_start = False
:
: for line in profile.split('\n'):
: if found_events_start:
: leading_whitespace = len(line) - len(line.lstrip())
: if indent_len is None:
: # This was the first event. We store the indentation
of the events.
: indent_len = leading_whitespace
: elif leading_whitespace < indent_len:
: # We've reached the line after the events, stop the
iteration.
: break
:
: # If we reach here we are processing a line containing
an event.
: events.append(self.__extract_event_name(line))
:
: elif 'Fragment Instance Lifecycle Event Timeline' in line:
: found_events_start = True
:
: return events
:
: def __extract_event_name(self, line):
: start = line.index('-') + 2 # There is a space after the
dash.
: end = line.index(':')
: return line[start:end]
This could be moved to common code, e.g.
https://github.com/apache/impala/blob/master/tests/common/test_result_verifier.py
We do similar things at other places, e.g.
https://github.com/apache/impala/blob/2e07d0c07febf1d1ee9324708d543b792fb45b00/tests/query_test/test_observability.py#L404,
but I prefer your version.
http://gerrit.cloudera.org:8080/#/c/15105/43/tests/query_test/test_queries.py
File tests/query_test/test_queries.py:
http://gerrit.cloudera.org:8080/#/c/15105/43/tests/query_test/test_queries.py@75
PS43, Line 75: parquet
nit: Parquet
http://gerrit.cloudera.org:8080/#/c/15105/43/tests/query_test/test_queries.py@80
PS43, Line 80: beeswax
I would prefer hs2, as beeswax may be deprecated in the not too far future.
http://gerrit.cloudera.org:8080/#/c/15105/43/tests/query_test/test_query_mem_limit.py
File tests/query_test/test_query_mem_limit.py:
http://gerrit.cloudera.org:8080/#/c/15105/43/tests/query_test/test_query_mem_limit.py@30
PS43, Line 30: add_exec_option_dimension,
Did something change in this file?
--
To view, visit http://gerrit.cloudera.org:8080/15105
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia7cbfa7c6734dcf03641629429057d6a4194aa6b
Gerrit-Change-Number: 15105
Gerrit-PatchSet: 43
Gerrit-Owner: Daniel Becker
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Daniel Becker
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Tue, 30 Jun 2020 15:13:37 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9744: Treat corrupt table stats as missing to avoid bad plans
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/16098 ) Change subject: IMPALA-9744: Treat corrupt table stats as missing to avoid bad plans .. Patch Set 11: (4 comments) http://gerrit.cloudera.org:8080/#/c/16098/10/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java File fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java: http://gerrit.cloudera.org:8080/#/c/16098/10/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@a1164 PS10, Line 1164: > I thought that getNumClusteringCols.size() == number of partition columns f Yes by a single partitioned table I mean a non-partitioned table. We use the term a lot in my previous job. http://gerrit.cloudera.org:8080/#/c/16098/10/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@1179 PS10, Line 1179: // If all partitions have good stats, return the total row count, contributed : // by all of them, as the row count for the table. > > So to summarize, the goal here (or at least original intention of this JI Yes, I agree that treating the missing and corrupt stats the same is a good idea, in the context of providing a good/useful row count (RC). My first impression of the original logic, at line 1179, if (numPartitionsWithNumRows_ > 0) return partitionNumRows_; is that it can seriously under-estimate the RC when only one partition has the good stats. This is addressed by the fix. http://gerrit.cloudera.org:8080/#/c/16098/10/tests/metadata/test_compute_stats.py File tests/metadata/test_compute_stats.py: http://gerrit.cloudera.org:8080/#/c/16098/10/tests/metadata/test_compute_stats.py@199 PS10, Line 199: int_col int, > Isn't it the load data local inpath query that is setting the stats to a co Yes, the bad stats was created by the loading part. Sorry I did not state it clearly in my comment above and missed a point as follows. If we create the table with 'create table like' in Impala, the testing table itself becomes an Impala one, regardless of being internal or external. See https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/impala_tables.html#tables. Normally, one needs to create a table "natively" in an engine to create a "native" table. Thus, if the objective is to test against a Hive table, the current version does the job. We could add a test case as you suggested to test the impala table. http://gerrit.cloudera.org:8080/#/c/16098/10/tests/metadata/test_compute_stats.py@215 PS10, Line 215: # Make the table visible in Impala. > It's set to true by default in Hive already: https://github.com/apache/hive Maybe we add a comment here? The rational is to call out the condition to reproduce the bad stats in the test, regardless of the default setting in Hive. -- To view, visit http://gerrit.cloudera.org:8080/16098 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9f4c64616ff7c0b6d5a48f2b5331325feeff3576 Gerrit-Change-Number: 16098 Gerrit-PatchSet: 11 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Jun 2020 15:04:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6692: Trigger sort node run before hitting memory limit.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15963 ) Change subject: IMPALA-6692: Trigger sort node run before hitting memory limit. .. Patch Set 15: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6465/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15963 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2a0ba7c4bae4f1d300d4d9d7f594f63ced06a240 Gerrit-Change-Number: 15963 Gerrit-PatchSet: 15 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Jun 2020 14:45:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9907: Fix NullPointerException in ParallelFileMetadataLoader's load() method
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16121 ) Change subject: IMPALA-9907: Fix NullPointerException in ParallelFileMetadataLoader's load() method .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6079/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16121 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4322a18a1f07e4106789591b80d25105c565a2b6 Gerrit-Change-Number: 16121 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 30 Jun 2020 14:33:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9907: Fix NullPointerException in ParallelFileMetadataLoader's load() method
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16121 ) Change subject: IMPALA-9907: Fix NullPointerException in ParallelFileMetadataLoader's load() method .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16121 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4322a18a1f07e4106789591b80d25105c565a2b6 Gerrit-Change-Number: 16121 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 30 Jun 2020 14:33:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6692: Trigger sort node run before hitting memory limit.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/15963 ) Change subject: IMPALA-6692: Trigger sort node run before hitting memory limit. .. Patch Set 15: (8 comments) Thank you, Csaba! http://gerrit.cloudera.org:8080/#/c/15963/14//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15963/14//COMMIT_MSG@10 PS14, Line 10: batches are added to current unsorted run or memory limit is hit, > missing 'are'? Done http://gerrit.cloudera.org:8080/#/c/15963/14//COMMIT_MSG@11 PS14, Line 11: sorter will immediately start the run. If the latter case happens, > nit: happens Done http://gerrit.cloudera.org:8080/#/c/15963/14//COMMIT_MSG@13 PS14, Line 13: new unsorted run object, and continue to add the next row batches, and > missing 'to' Done http://gerrit.cloudera.org:8080/#/c/15963/14//COMMIT_MSG@15 PS14, Line 15: > nit: tries Done http://gerrit.cloudera.org:8080/#/c/15963/14//COMMIT_MSG@23 PS14, Line 23: > nit: speeds up Done http://gerrit.cloudera.org:8080/#/c/15963/14//COMMIT_MSG@26 PS14, Line 26: 'sort_ > nit: exceeded Done http://gerrit.cloudera.org:8080/#/c/15963/14//COMMIT_MSG@32 PS14, Line 32: ssion > nit: fit Done http://gerrit.cloudera.org:8080/#/c/15963/14/be/src/runtime/sorter.cc File be/src/runtime/sorter.cc: http://gerrit.cloudera.org:8080/#/c/15963/14/be/src/runtime/sorter.cc@952 PS14, Line 952: 3 > I didn't catch this one - I think that it should be also VLOG(3) Done -- To view, visit http://gerrit.cloudera.org:8080/15963 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2a0ba7c4bae4f1d300d4d9d7f594f63ced06a240 Gerrit-Change-Number: 15963 Gerrit-PatchSet: 15 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Jun 2020 14:18:47 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6692: Trigger sort node run before hitting memory limit.
Hello David Rorke, Tim Armstrong, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15963 to look at the new patch set (#15). Change subject: IMPALA-6692: Trigger sort node run before hitting memory limit. .. IMPALA-6692: Trigger sort node run before hitting memory limit. Sorter node works by adding row batches to a sort run. After all batches are added to current unsorted run or memory limit is hit, sorter will immediately start the run. If the latter case happens, sorter will spill the sorted run to disk after sort complete, create new unsorted run object, and continue to add the next row batches, and so on. This algorithm tries to fit as much rows into memory before start sorting. However, in the case of partitioned sort with large number of row batches, fitting too much rows into memory will cause the sort to be slow and block the sorter node for a long time before it can release some memory and continue accepting the next row batch from exchange node. One slow sorter node can block exchange node from sending row batches to other sorter node that is free. This patch speeds up the decision to start the sort without waiting it to hit memory limit first by capping the intermediary quicksort run to lower memory limit, determined by query option 'sort_run_bytes_limit'. If the total used reservation of quicksort has exceeded sort_run_bytes_limit, current unsorted_run_ will be wrapped up, sorted, and then spilled. Thus, overlapping the next sort run with spill from previous sort run. To reduce regression for cases where total input size of sort node might be fully fit into available memory, sort_run_bytes_limit will not be enforced for the first sort run. However, it will stay limited by sort_run_bytes_limit if planner estimates hint that spill is inevitably will happen. We also add new summary counter 'AddBatchTime' to get summary of how much time spent in Sorter::AddBatch. Max of 'AddBatchTime' indicate the longest time spent in Sorter::AddBatch, presumably busy doing intermediary sort. Testing: - Add new e2e test TestQueryFullSort::test_multiple_sort_run_bytes_limits - Run core tests - Run data loading of 3 largest TPC-DS facts table of 300GB scale into real cluster using 5 backends, and 4GB mem_limit. sort_run_bytes_limit is varied between unspecified (not limited) vs 512 MB. The performance result is summarized in the following table. +---+-+--+---+-+ | Insert table | #Rows | Avg | no limit| 512 MB limit | | | | SortDataSize ++--+-+---+ | | | per Node | Query | Max | Query | Max | | | | | Time | AddBatchTime | Time | AddBatchTime | +---+-+--++--+-+---+ | store_sales | 864.00M | 15.29 GB | 30m18s | 53s311ms | 20m | 5s634ms | +---+-+--++--+-+---+ | catalog_sales | 431.97M | 11.34 GB | 23m24s | 31s212ms | 15m27s | 3s603ms | +---+-+--++--+-+---+ | web_sales | 216.01M | 5.67 GB | 8m16s | 29s250ms | 6m41s | 3s856ms | +---+-+--++--+-+---+ Change-Id: I2a0ba7c4bae4f1d300d4d9d7f594f63ced06a240 --- M be/src/exec/sort-node.cc M be/src/exec/sort-node.h M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/query-state.cc M be/src/runtime/query-state.h M be/src/runtime/sorter.cc M be/src/runtime/sorter.h M be/src/service/query-options-test.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/planner/SortNode.java M tests/query_test/test_sort.py 15 files changed, 225 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/15963/15 -- To view, visit http://gerrit.cloudera.org:8080/15963 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2a0ba7c4bae4f1d300d4d9d7f594f63ced06a240 Gerrit-Change-Number: 15963 Gerrit-PatchSet: 15 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9859: Full ACID Milestone 4: Part 1 Reading modified tables (primitive types)
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16082 ) Change subject: IMPALA-9859: Full ACID Milestone 4: Part 1 Reading modified tables (primitive types) .. Patch Set 9: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16082 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I15c8feabf40be1658f3dd46883f5a1b2aa5d0659 Gerrit-Change-Number: 16082 Gerrit-PatchSet: 9 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 30 Jun 2020 14:02:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9907: Fix NullPointerException in ParallelFileMetadataLoader's load() method
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16121 ) Change subject: IMPALA-9907: Fix NullPointerException in ParallelFileMetadataLoader's load() method .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6077/ -- To view, visit http://gerrit.cloudera.org:8080/16121 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4322a18a1f07e4106789591b80d25105c565a2b6 Gerrit-Change-Number: 16121 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 30 Jun 2020 13:32:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9890 (Part 1): Add more TPCDS queries to Impala's test suite
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16119 ) Change subject: IMPALA-9890 (Part 1): Add more TPCDS queries to Impala's test suite .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16119 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib5f260e75a3803aabe9ccef271ba94036f96e5cf Gerrit-Change-Number: 16119 Gerrit-PatchSet: 5 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Jun 2020 13:06:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9890 (Part 1): Add more TPCDS queries to Impala's test suite
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16119 ) Change subject: IMPALA-9890 (Part 1): Add more TPCDS queries to Impala's test suite .. IMPALA-9890 (Part 1): Add more TPCDS queries to Impala's test suite This patch adds the following 12 TPCDS queries to the class of TestTpcdsDecimalV2Query: Q26, Q30, Q31, Q47, Q48, Q57, Q58, Q59, Q63, Q83, Q85, and Q89. All the queries except for Q31 are added to the class of TestTpcdsQuery as well because Impala returns one fewer row than expected for TestTpcdsQuery::test_tpcds_q31(), which requires further investigation. To verify whether or not the returned result set from Impala for a given query is correct, we compare the result set with that produced by the HiveServer2 (HS2) in Impala's mini-cluster. We could execute SQL statements in HS2 via Beeline, HS2's command line shell, which could be launched by the following command. beeline -u "jdbc:hive2://localhost:11050/default" We note that among these 12 queries, the execution of Q31, Q58, and Q83 result in the error of "Counters limit exceeded" by TEZ. To work around this problem, for these 3 queries we have to execute the following statement before running them to increase the default number of counters, which is set to 120. set tez.counters.max=1200 On the other hand, the table of 'reason' is referenced by Q85. This table was not referenced by any TPCDS query before this patch and thus was not created. In this regard, in this patch we also modify tpcds_schema_template.sql to create this additional table along with its data. Testing: - Verified that this patch passes the exhaustive tests in the DEBUG build. Change-Id: Ib5f260e75a3803aabe9ccef271ba94036f96e5cf Reviewed-on: http://gerrit.cloudera.org:8080/16119 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M testdata/datasets/tpcds/tpcds_schema_template.sql A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q26.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q30.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q31.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q47.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q48.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q57.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q58.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q59.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q63.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q83.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q85.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q89.test A testdata/workloads/tpcds/queries/tpcds-q26.test A testdata/workloads/tpcds/queries/tpcds-q30.test A testdata/workloads/tpcds/queries/tpcds-q47.test A testdata/workloads/tpcds/queries/tpcds-q48.test A testdata/workloads/tpcds/queries/tpcds-q57.test A testdata/workloads/tpcds/queries/tpcds-q58.test A testdata/workloads/tpcds/queries/tpcds-q59.test A testdata/workloads/tpcds/queries/tpcds-q63.test A testdata/workloads/tpcds/queries/tpcds-q83.test A testdata/workloads/tpcds/queries/tpcds-q85.test A testdata/workloads/tpcds/queries/tpcds-q89.test M tests/query_test/test_tpcds_queries.py M tests/util/parse_util.py 26 files changed, 2,622 insertions(+), 1 deletion(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16119 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ib5f260e75a3803aabe9ccef271ba94036f96e5cf Gerrit-Change-Number: 16119 Gerrit-PatchSet: 6 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9907: Fix NullPointerException in ParallelFileMetadataLoader's load() method
Zoltan Borok-Nagy has posted comments on this change. (
http://gerrit.cloudera.org:8080/16121 )
Change subject: IMPALA-9907: Fix NullPointerException in
ParallelFileMetadataLoader's load() method
..
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/16121/1/fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java
File fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java:
http://gerrit.cloudera.org:8080/#/c/16121/1/fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java@166
PS1, Line 166: loader,
: pool.submit(() -> { loader.load(); return null; })));
> nit, can be moved to earlier line.
Done
--
To view, visit http://gerrit.cloudera.org:8080/16121
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4322a18a1f07e4106789591b80d25105c565a2b6
Gerrit-Change-Number: 16121
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Tue, 30 Jun 2020 12:57:44 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9569: Fix progress bar and live summary to show info of the retried query
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16096 ) Change subject: IMPALA-9569: Fix progress bar and live_summary to show info of the retried query .. IMPALA-9569: Fix progress bar and live_summary to show info of the retried query Impala-shell periodically calls GetExecSummary() when the query is queuing or running. If the query is being retried, GetExecSummary() should return the TExecSummary of the retried query. So the progress bar and live_summary can reflect the most recent state. This patch also modifies get_summary() to return retry information in error_logs of TExecSummary. Impala-shell and other clients can print the info right after the query starts being retried. Modified impala-shell to print the retried query link when the retried query is running. Example output when the retried query is running: Query: select count(*) from functional.alltypes where bool_col = sleep(60) Query submitted at: 2020-06-18 22:08:49 (Coordinator: http://quanlong-OptiPlex-BJ:25000) Query progress can be monitored at: http://quanlong-OptiPlex-BJ:25000/query_plan?query_id=9444fe7f0df0da28:29134b08 Failed due to unreachable impalad(s): quanlong-OptiPlex-BJ:22001 Retrying query using query id: 5748d9a3ccc28ba8:a75e2fab Retried query link: http://quanlong-OptiPlex-BJ:25000/query_plan?query_id=5748d9a3ccc28ba8:a75e2fab [### ] 50% Tests: - Manually verify the progress bar and live_summary work when the query is being retried. - Add tests in test_query_retries.py to validate the get_summary() results. Change-Id: I8f96919f00e0b64d589efd15b6b5ec82fb725d56 Reviewed-on: http://gerrit.cloudera.org:8080/16096 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/service/impala-server.cc M be/src/service/impala-server.h M shell/impala_shell.py M tests/common/impala_connection.py M tests/custom_cluster/test_query_retries.py 5 files changed, 120 insertions(+), 28 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16096 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I8f96919f00e0b64d589efd15b6b5ec82fb725d56 Gerrit-Change-Number: 16096 Gerrit-PatchSet: 8 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-9569: Fix progress bar and live summary to show info of the retried query
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16096 ) Change subject: IMPALA-9569: Fix progress bar and live_summary to show info of the retried query .. Patch Set 7: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16096 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8f96919f00e0b64d589efd15b6b5ec82fb725d56 Gerrit-Change-Number: 16096 Gerrit-PatchSet: 7 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 30 Jun 2020 12:11:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9859: Full ACID Milestone 4: Part 1 Reading modified tables (primitive types)
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16082 ) Change subject: IMPALA-9859: Full ACID Milestone 4: Part 1 Reading modified tables (primitive types) .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6464/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16082 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I15c8feabf40be1658f3dd46883f5a1b2aa5d0659 Gerrit-Change-Number: 16082 Gerrit-PatchSet: 9 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 30 Jun 2020 09:28:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9859: Full ACID Milestone 4: Part 1 Reading modified tables (primitive types)
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16082 ) Change subject: IMPALA-9859: Full ACID Milestone 4: Part 1 Reading modified tables (primitive types) .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6078/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/16082 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I15c8feabf40be1658f3dd46883f5a1b2aa5d0659 Gerrit-Change-Number: 16082 Gerrit-PatchSet: 9 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 30 Jun 2020 09:02:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9859: Full ACID Milestone 4: Part 1 Reading modified tables (primitive types)
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16082 ) Change subject: IMPALA-9859: Full ACID Milestone 4: Part 1 Reading modified tables (primitive types) .. Patch Set 9: Code-Review+1 PS9 is a rebase. Carrying +1. -- To view, visit http://gerrit.cloudera.org:8080/16082 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I15c8feabf40be1658f3dd46883f5a1b2aa5d0659 Gerrit-Change-Number: 16082 Gerrit-PatchSet: 9 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 30 Jun 2020 09:02:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9859: Full ACID Milestone 4: Part 1 Reading modified tables (primitive types)
Hello Aman Sinha, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16082 to look at the new patch set (#9). Change subject: IMPALA-9859: Full ACID Milestone 4: Part 1 Reading modified tables (primitive types) .. IMPALA-9859: Full ACID Milestone 4: Part 1 Reading modified tables (primitive types) Hive ACID supports row-level DELETE and UPDATE operations on a table. It achieves it via assigning a unique row-id for each row, and maintaining two sets of files in a table. The first set is in the base/delta directories, they contain the INSERTed rows. The second set of files are in the delete-delta directories, they contain the DELETEd rows. (UPDATE operations are implemented via DELETE+INSERT.) In the filesystem it looks like e.g.: * full_acid/delta_001_001_/_0 * full_acid/delta_002_002_/_0 * full_acid/delete_delta_003_003_/_0 During scanning we need to return INSERTed rows minus DELETEd rows. This patch implements it by creating an ANTI JOIN between the INSERT and DELETE sets. It is a planner-only modification. Every HDFS SCAN that scans full ACID tables (that also have deleted rows) are converted to two HDFS SCANs, one for the INSERT deltas, and one for the DELETE deltas. Then a LEFT ANTI HASH JOIN with BROADCAST distribution mode is created above them. Later we can add support for other distribution modes if the performance requires it. E.g. if we have too many deleted rows then probably we are better off with PARTITIONED distribution mode. We could estimate the number of deleted rows by sampling the delete delta files. The current patch only works for primitive types. I.e. we cannot select nested data if the table has deleted rows. Testing: * added planner test * added e2e tests Change-Id: I15c8feabf40be1658f3dd46883f5a1b2aa5d0659 --- M fe/src/main/java/org/apache/impala/catalog/FeFsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java M fe/src/main/java/org/apache/impala/planner/JoinNode.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/util/AcidUtils.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M fe/src/test/java/org/apache/impala/util/AcidUtilsTest.java M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv A testdata/workloads/functional-planner/queries/PlannerTest/acid-scans.test M testdata/workloads/functional-query/queries/QueryTest/acid-negative.test A testdata/workloads/functional-query/queries/QueryTest/full-acid-scans.test M tests/query_test/test_acid.py 15 files changed, 1,148 insertions(+), 89 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/16082/9 -- To view, visit http://gerrit.cloudera.org:8080/16082 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I15c8feabf40be1658f3dd46883f5a1b2aa5d0659 Gerrit-Change-Number: 16082 Gerrit-PatchSet: 9 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-9907: Fix NullPointerException in ParallelFileMetadataLoader's load() method
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16121 ) Change subject: IMPALA-9907: Fix NullPointerException in ParallelFileMetadataLoader's load() method .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6463/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16121 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4322a18a1f07e4106789591b80d25105c565a2b6 Gerrit-Change-Number: 16121 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 30 Jun 2020 08:44:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9907: Fix NullPointerException in ParallelFileMetadataLoader's load() method
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16121 ) Change subject: IMPALA-9907: Fix NullPointerException in ParallelFileMetadataLoader's load() method .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6077/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16121 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4322a18a1f07e4106789591b80d25105c565a2b6 Gerrit-Change-Number: 16121 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 30 Jun 2020 08:28:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9907: Fix NullPointerException in ParallelFileMetadataLoader's load() method
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16121 ) Change subject: IMPALA-9907: Fix NullPointerException in ParallelFileMetadataLoader's load() method .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16121 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4322a18a1f07e4106789591b80d25105c565a2b6 Gerrit-Change-Number: 16121 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 30 Jun 2020 08:28:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9907: Fix NullPointerException in ParallelFileMetadataLoader's load() method
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16121 ) Change subject: IMPALA-9907: Fix NullPointerException in ParallelFileMetadataLoader's load() method .. Patch Set 2: Code-Review+2 Carrying +2 -- To view, visit http://gerrit.cloudera.org:8080/16121 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4322a18a1f07e4106789591b80d25105c565a2b6 Gerrit-Change-Number: 16121 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 30 Jun 2020 08:28:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9907: Fix NullPointerException in ParallelFileMetadataLoader's load() method
Hello Vihang Karajgaonkar, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16121 to look at the new patch set (#2). Change subject: IMPALA-9907: Fix NullPointerException in ParallelFileMetadataLoader's load() method .. IMPALA-9907: Fix NullPointerException in ParallelFileMetadataLoader's load() method ParallelFileMetadataLoader.loaders_ became a Map, but at one place we still treat it as a List: loaders_.get(i).getPartDir() 'i' is an integer, while the loaders_'s key type is Path. Unfortunately the code compiled which means we got a NullPointerException for this statement. I changed the logic to avoid the NPE. Testing: * Tested manually on an ACID table with deleted rows * I didn't add an automated test because I'm about to extend the ACID functionalities, hence getting rid of these exceptions * The other case when we can hit this NPE is when we get an IOException in the file metadata loader's load() method Change-Id: I4322a18a1f07e4106789591b80d25105c565a2b6 --- M fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java 1 file changed, 7 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/16121/2 -- To view, visit http://gerrit.cloudera.org:8080/16121 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4322a18a1f07e4106789591b80d25105c565a2b6 Gerrit-Change-Number: 16121 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-6692: Trigger sort node run before hitting memory limit.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/15963 ) Change subject: IMPALA-6692: Trigger sort node run before hitting memory limit. .. Patch Set 14: Code-Review+1 (8 comments) http://gerrit.cloudera.org:8080/#/c/15963/14//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15963/14//COMMIT_MSG@10 PS14, Line 10: batches added to current unsorted run or memory limit is hit, sorter missing 'are'? http://gerrit.cloudera.org:8080/#/c/15963/14//COMMIT_MSG@11 PS14, Line 11: will immediately start the run. If the latter case happen, sorter will nit: happens http://gerrit.cloudera.org:8080/#/c/15963/14//COMMIT_MSG@13 PS14, Line 13: run object, and continue add the next row batches, and so on. missing 'to' http://gerrit.cloudera.org:8080/#/c/15963/14//COMMIT_MSG@15 PS14, Line 15: try nit: tries http://gerrit.cloudera.org:8080/#/c/15963/14//COMMIT_MSG@23 PS14, Line 23: speedup nit: speeds up http://gerrit.cloudera.org:8080/#/c/15963/14//COMMIT_MSG@26 PS14, Line 26: exceed nit: exceeded http://gerrit.cloudera.org:8080/#/c/15963/14//COMMIT_MSG@32 PS14, Line 32: fitted nit: fit http://gerrit.cloudera.org:8080/#/c/15963/14/be/src/runtime/sorter.cc File be/src/runtime/sorter.cc: http://gerrit.cloudera.org:8080/#/c/15963/14/be/src/runtime/sorter.cc@952 PS14, Line 952: 2 I didn't catch this one - I think that it should be also VLOG(3) -- To view, visit http://gerrit.cloudera.org:8080/15963 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2a0ba7c4bae4f1d300d4d9d7f594f63ced06a240 Gerrit-Change-Number: 15963 Gerrit-PatchSet: 14 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Jun 2020 08:19:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9890 (Part 1): Add more TPCDS queries to Impala's test suite
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/16119 ) Change subject: IMPALA-9890 (Part 1): Add more TPCDS queries to Impala's test suite .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16119 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib5f260e75a3803aabe9ccef271ba94036f96e5cf Gerrit-Change-Number: 16119 Gerrit-PatchSet: 4 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Jun 2020 07:56:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9890 (Part 1): Add more TPCDS queries to Impala's test suite
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16119 ) Change subject: IMPALA-9890 (Part 1): Add more TPCDS queries to Impala's test suite .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6076/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16119 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib5f260e75a3803aabe9ccef271ba94036f96e5cf Gerrit-Change-Number: 16119 Gerrit-PatchSet: 5 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Jun 2020 07:57:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9890 (Part 1): Add more TPCDS queries to Impala's test suite
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16119 ) Change subject: IMPALA-9890 (Part 1): Add more TPCDS queries to Impala's test suite .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16119 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib5f260e75a3803aabe9ccef271ba94036f96e5cf Gerrit-Change-Number: 16119 Gerrit-PatchSet: 5 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Jun 2020 07:57:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9531: Dropped support for dateless timestamps
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15866 ) Change subject: IMPALA-9531: Dropped support for dateless timestamps .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6462/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15866 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I48c49bf027cc4b917849b3d58518facba372b322 Gerrit-Change-Number: 15866 Gerrit-PatchSet: 8 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Adam Tamas Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 30 Jun 2020 07:29:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9569: Fix progress bar and live summary to show info of the retried query
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16096 ) Change subject: IMPALA-9569: Fix progress bar and live_summary to show info of the retried query .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6075/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16096 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8f96919f00e0b64d589efd15b6b5ec82fb725d56 Gerrit-Change-Number: 16096 Gerrit-PatchSet: 7 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 30 Jun 2020 07:08:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9569: Fix progress bar and live summary to show info of the retried query
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16096 ) Change subject: IMPALA-9569: Fix progress bar and live_summary to show info of the retried query .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16096 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8f96919f00e0b64d589efd15b6b5ec82fb725d56 Gerrit-Change-Number: 16096 Gerrit-PatchSet: 7 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 30 Jun 2020 07:08:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9531: Dropped support for dateless timestamps
Adam Tamas has uploaded a new patch set (#8). (
http://gerrit.cloudera.org:8080/15866 )
Change subject: IMPALA-9531: Dropped support for dateless timestamps
..
IMPALA-9531: Dropped support for dateless timestamps
Removed the support for dateless timestamps.
During dateless timestamp casts if the format doesn't contain
date part we get an error during tokenization of the format.
If the input str doesn't contain a date part then we get null result.
Examples:
select cast('01:02:59' as timestamp);
This will come back as NULL value.
select to_timestamp('01:01:01', 'HH:mm:ss');
select cast('01:02:59' as timestamp format 'HH12:MI:SS');
select cast('12 AM' as timestamp FORMAT 'AM.HH12');
These will come back with a parsing errors.
Casting from a table will generate similar results.
Testing:
Modified the previous tests related to dateless timestamps.
Added test to read fromtables which are still containing dateless
timestamps and covered timestamp to string path when no date tokens
are requested in the output string.
Change-Id: I48c49bf027cc4b917849b3d58518facba372b322
---
M be/src/benchmarks/convert-timestamp-benchmark.cc
M be/src/benchmarks/parse-timestamp-benchmark.cc
M be/src/exec/text-converter.inline.h
M be/src/exprs/cast-functions-ir.cc
M be/src/exprs/expr-test.cc
M be/src/exprs/scalar-expr-evaluator.cc
M be/src/exprs/timestamp-functions-ir.cc
M be/src/exprs/timestamp-functions.cc
M be/src/exprs/timestamp-functions.h
M be/src/runtime/date-parse-util.cc
M be/src/runtime/date-test.cc
M be/src/runtime/datetime-iso-sql-format-tokenizer.cc
M be/src/runtime/datetime-parser-common.cc
M be/src/runtime/datetime-parser-common.h
M be/src/runtime/datetime-simple-date-format-parser.cc
M be/src/runtime/datetime-simple-date-format-parser.h
M be/src/runtime/timestamp-parse-util.cc
M be/src/runtime/timestamp-test.cc
M be/src/runtime/timestamp-value.h
M bin/rat_exclude_files.txt
M common/function-registry/impala_functions.py
M fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java
M testdata/data/README
A testdata/data/dateless_timestamps.parq
A testdata/data/dateless_timestamps.txt
M testdata/data/lazy_timestamp.csv
M testdata/workloads/functional-query/queries/QueryTest/date.test
A
testdata/workloads/functional-query/queries/QueryTest/dateless_timestamp_parquet.test
A
testdata/workloads/functional-query/queries/QueryTest/dateless_timestamp_text.test
M testdata/workloads/functional-query/queries/QueryTest/exprs.test
M
testdata/workloads/functional-query/queries/QueryTest/select-lazy-timestamp.test
M tests/data_errors/test_data_errors.py
M tests/query_test/test_cast_with_format.py
M tests/query_test/test_scanners.py
34 files changed, 278 insertions(+), 231 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/15866/8
--
To view, visit http://gerrit.cloudera.org:8080/15866
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I48c49bf027cc4b917849b3d58518facba372b322
Gerrit-Change-Number: 15866
Gerrit-PatchSet: 8
Gerrit-Owner: Adam Tamas
Gerrit-Reviewer: Adam Tamas
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
