[Impala-ASF-CR] IMPALA-7135. Skeleton implementation of LocalCatalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10627 ) Change subject: IMPALA-7135. Skeleton implementation of LocalCatalog .. Patch Set 7: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10627 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iab653371188b21c72f50ee1ec4e94950aa6fb9ee Gerrit-Change-Number: 10627 Gerrit-PatchSet: 7 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 15 Jun 2018 04:10:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7135. Skeleton implementation of LocalCatalog
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10627 ) Change subject: IMPALA-7135. Skeleton implementation of LocalCatalog .. IMPALA-7135. Skeleton implementation of LocalCatalog This adds some of the high level classes for implementing the local catalog: - LocalCatalog is the top level implementation. The plan is to instantiate this once per query, so that no thread safety is required. - It loads metadata from a MetaProvider interface. The current implementation fetches directly from HMS and provides no caching. A future subtask will add a CachingMetaProvider implementation. Separating out caching will make it easier to experiment with different policies or storage mechanisms. - It instantiates LocalDb and LocalTable objects to implement FeDb and FeTable. These are mostly stubbed out except for the most basic functionality. Functionality will be filled in incrementally in further patches. Since it's not yet possible to hook this up to most of the existing tests, a very simple new unit test is included to cover the bits of functionality that are not stubbed out. I didn't concentrate on too much test coverage here, since once we've implemented more functionality we can switch over all of the existing tests to get coverage of the new implementation. Change-Id: Iab653371188b21c72f50ee1ec4e94950aa6fb9ee Reviewed-on: http://gerrit.cloudera.org:8080/10627 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogObject.java M fe/src/main/java/org/apache/impala/catalog/FeDb.java A fe/src/main/java/org/apache/impala/catalog/HasName.java A fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java A fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java A fe/src/main/java/org/apache/impala/catalog/local/LocalCatalogException.java A fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java A fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java A fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java A fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java 11 files changed, 823 insertions(+), 6 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/10627 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iab653371188b21c72f50ee1ec4e94950aa6fb9ee Gerrit-Change-Number: 10627 Gerrit-PatchSet: 8 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7174: fix test cancellation for RELEASE builds
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10722 ) Change subject: IMPALA-7174: fix test_cancellation for RELEASE builds .. IMPALA-7174: fix test_cancellation for RELEASE builds The test was DOA when run against a release build because the debug actions that it depends on were disabled. The fix is to enable the debug actions for release builds, similar to other debug actions. I assume the original motivation of the NDEBUG checks was to avoid adding overhead to release builds. The cost is minimised by quickly checking whether the string is empty before proceeding with any further work. Also remove wonky exception handling - the test was swallowing exceptions but we don't expect that code to throw exceptions. Testing: Looped the test on a release build. Change-Id: I41da7b5ac58a468a8ed11969906f63df6d4b Reviewed-on: http://gerrit.cloudera.org:8080/10722 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/scheduling/admission-controller.cc M be/src/service/client-request-state.cc M be/src/util/debug-util.cc M be/src/util/debug-util.h M tests/custom_cluster/test_admission_controller.py 5 files changed, 14 insertions(+), 21 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/10722 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I41da7b5ac58a468a8ed11969906f63df6d4b Gerrit-Change-Number: 10722 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7174: fix test cancellation for RELEASE builds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10722 ) Change subject: IMPALA-7174: fix test_cancellation for RELEASE builds .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10722 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I41da7b5ac58a468a8ed11969906f63df6d4b Gerrit-Change-Number: 10722 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 15 Jun 2018 03:21:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10709 ) Change subject: IMPALA-7111: avoid use of boost::split in CheckPluginEnabled .. IMPALA-7111: avoid use of boost::split in CheckPluginEnabled This is an attempt to either avoid the bug or make it easier to diagnose if it reoccurs. My suspicion is that somehow boost::split() is accessing the input string in a non-thread-safe manner, but the implementation is opaque enough that it's not obvious how. Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8 Reviewed-on: http://gerrit.cloudera.org:8080/10709 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/exec/hdfs-plugin-text-scanner.cc M be/src/util/string-util-test.cc M be/src/util/string-util.cc M be/src/util/string-util.h 4 files changed, 48 insertions(+), 6 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/10709 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8 Gerrit-Change-Number: 10709 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-5604: document DISABLE CODEGEN ROWS THRESHOLD
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10727 ) Change subject: IMPALA-5604: document DISABLE_CODEGEN_ROWS_THRESHOLD .. IMPALA-5604: document DISABLE_CODEGEN_ROWS_THRESHOLD Also fix a couple of nits in EXEC_SINGLE_NODE_ROWS_THRESHOLD. Change-Id: I709cd55e3869888feb645f85e61a99901d41d479 Reviewed-on: http://gerrit.cloudera.org:8080/10727 Reviewed-by: Alex Rodoni Tested-by: Impala Public Jenkins --- M docs/impala.ditamap M docs/impala_keydefs.ditamap A docs/topics/impala_disable_codegen_rows_threshold.xml M docs/topics/impala_exec_single_node_rows_threshold.xml 4 files changed, 105 insertions(+), 5 deletions(-) Approvals: Alex Rodoni: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/10727 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I709cd55e3869888feb645f85e61a99901d41d479 Gerrit-Change-Number: 10727 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-5604: document DISABLE CODEGEN ROWS THRESHOLD
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10727 ) Change subject: IMPALA-5604: document DISABLE_CODEGEN_ROWS_THRESHOLD .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10727 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I709cd55e3869888feb645f85e61a99901d41d479 Gerrit-Change-Number: 10727 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 15 Jun 2018 02:00:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5604: document DISABLE CODEGEN ROWS THRESHOLD
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/10727 ) Change subject: IMPALA-5604: document DISABLE_CODEGEN_ROWS_THRESHOLD .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10727 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I709cd55e3869888feb645f85e61a99901d41d479 Gerrit-Change-Number: 10727 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Alex Rodoni Gerrit-Comment-Date: Fri, 15 Jun 2018 01:50:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5604: document DISABLE CODEGEN ROWS THRESHOLD
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10727 ) Change subject: IMPALA-5604: document DISABLE_CODEGEN_ROWS_THRESHOLD .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-docs-submit/319/ -- To view, visit http://gerrit.cloudera.org:8080/10727 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I709cd55e3869888feb645f85e61a99901d41d479 Gerrit-Change-Number: 10727 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 15 Jun 2018 01:51:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7140 (part 3): load partitions for FS tables
Vuk Ercegovac has posted comments on this change. (
http://gerrit.cloudera.org:8080/10713 )
Change subject: IMPALA-7140 (part 3): load partitions for FS tables
..
Patch Set 2:
(16 comments)
http://gerrit.cloudera.org:8080/#/c/10713/2/fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java
File fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java:
http://gerrit.cloudera.org:8080/#/c/10713/2/fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java@171
PS2, Line 171: for (String partitionKey: hmsPartitionValues)
this'll allow for a subset of partitioning columns to be returned for table. I
assume that a subset would be incorrect so lets guard against it. It was not
needed from the place this was factored since that list of strings came
directly from the hms structure, so we can assume its correct. However, as a
lib, who knows what those strings are.
http://gerrit.cloudera.org:8080/#/c/10713/2/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
File fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java:
http://gerrit.cloudera.org:8080/#/c/10713/2/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java@546
PS2, Line 546: // only to end up throwing away the result.
yikes... yes, just ran across this today as well.
http://gerrit.cloudera.org:8080/#/c/10713/2/fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java:
http://gerrit.cloudera.org:8080/#/c/10713/2/fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java@132
PS2, Line 132: Warehouse.makeSpecFromName
something seems off here-- we're going back to hms for partitioning
information. we should have this schema info already-- do we ever load
partitions before the schema?
if SchemaInfo were pulled out perhaps? I see that kinda breaks the layering
here where objects (db, table) depend on metaprovider (and metaprovider knows
nothing from above). other ideas?
http://gerrit.cloudera.org:8080/#/c/10713/2/fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java@143
PS2, Line 143:
nit: consistent spacing
http://gerrit.cloudera.org:8080/#/c/10713/2/fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java@148
PS2, Line 148: name
how does name correspond to the partition name that's passed in? I assume it
must be a subset of the input names.
http://gerrit.cloudera.org:8080/#/c/10713/2/fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java:
http://gerrit.cloudera.org:8080/#/c/10713/2/fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java@122
PS2, Line 122: // TODO Auto-generated catch block
remove
http://gerrit.cloudera.org:8080/#/c/10713/2/fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java@144
PS2, Line 144: // TODO(todd): copy-paste from HdfsPartition
mark it as expensive (like in the other place)
http://gerrit.cloudera.org:8080/#/c/10713/2/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java:
http://gerrit.cloudera.org:8080/#/c/10713/2/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java@63
PS2, Line 63: /**
nit: newline
http://gerrit.cloudera.org:8080/#/c/10713/2/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java@183
PS2, Line 183: List names = Lists.newArrayList();
is loadPartitionSpecs() missing?
http://gerrit.cloudera.org:8080/#/c/10713/2/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java@198
PS2, Line 198:
nit: remove empty line
http://gerrit.cloudera.org:8080/#/c/10713/2/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java@214
PS2, Line 214: if (spec.getName().isEmpty()) continue;
when does this happen?
http://gerrit.cloudera.org:8080/#/c/10713/2/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java@279
PS2, Line 279: if (getNumClusteringCols() == 0) {
perhaps this is modeled better as two classes, unpartitioned and partitioned.
the place where LocalFsTable is created (LocalTable) gets the schema first, so
has enough info to construct the right one.
http://gerrit.cloudera.org:8080/#/c/10713/2/fe/src/main/java/org/apache/impala/catalog/local/LocalPartitionSpec.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalPartitionSpec.java:
http://gerrit.cloudera.org:8080/#/c/10713/2/fe/src/main/java/org/apache/impala/catalog/local/LocalPartitionSpec.java@43
PS2, Line 43: private final ImmutableList partitionValues;
nit: _ suffix
http://gerrit.cloudera.org:8080/#/c/10713/2/fe/src/main/java/org/apache/impala/catalog/local/LocalPartitionSpec.java@51
PS2, Line 51: getPartValuesFromPartName
I see a similar method in CatalogOpExecutor (getPartValsFromName). Perhaps it
[Impala-ASF-CR] IMPALA-7135. Skeleton implementation of LocalCatalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10627 ) Change subject: IMPALA-7135. Skeleton implementation of LocalCatalog .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2682/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/10627 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iab653371188b21c72f50ee1ec4e94950aa6fb9ee Gerrit-Change-Number: 10627 Gerrit-PatchSet: 7 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 15 Jun 2018 00:50:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7135. Skeleton implementation of LocalCatalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10627 ) Change subject: IMPALA-7135. Skeleton implementation of LocalCatalog .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10627 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iab653371188b21c72f50ee1ec4e94950aa6fb9ee Gerrit-Change-Number: 10627 Gerrit-PatchSet: 7 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 15 Jun 2018 00:50:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4848: Add WIDTH BUCKET() function
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/6023 )
Change subject: IMPALA-4848: Add WIDTH_BUCKET() function
..
Patch Set 15:
(3 comments)
This looks ok to me. I trust Taras has a deeper understanding of the decimal
logic than I do.
Is there some way we can add this to the decimal fuzz test?
http://gerrit.cloudera.org:8080/#/c/6023/15/be/src/exprs/expr-test.cc
File be/src/exprs/expr-test.cc:
http://gerrit.cloudera.org:8080/#/c/6023/15/be/src/exprs/expr-test.cc@582
PS15, Line 582: void TestErrorString(const string& expr, const string&
error_string) {
Can you document briefly what the function does?
E.g. "Execute 'expr' and check that the returned error ends with 'error_string'"
http://gerrit.cloudera.org:8080/#/c/6023/15/be/src/exprs/expr-test.cc@587
PS15, Line 587: status = executor_->FetchResult(_row);
ASSERT_FALSE(status.ok())
Otherwise we'll deference the null message below and crash.
http://gerrit.cloudera.org:8080/#/c/6023/15/be/src/exprs/expr-test.cc@588
PS15, Line 588:
ASSERT_TRUE(status.msg().msg().compare(status.msg().msg().size() -
This is basically EndsWith(), right? Can you factor out into a utility in
StringUtil, just to make it more obvious what the code is doing.
--
To view, visit http://gerrit.cloudera.org:8080/6023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I081bc916b1bef7b929ca161a9aade3b54c6b858f
Gerrit-Change-Number: 6023
Gerrit-PatchSet: 15
Gerrit-Owner: anujphadke
Gerrit-Reviewer: Alex Behm
Gerrit-Reviewer: Dan Hecht
Gerrit-Reviewer: Michael Brown
Gerrit-Reviewer: Taras Bobrovytsky
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: anujphadke
Gerrit-Comment-Date: Fri, 15 Jun 2018 00:48:48 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10611 ) Change subject: IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition .. Patch Set 7: This change did not cherrypick successfully into branch 2.x. To resolve this, please do the cherry-pick manually and submit it to Gerrit at refs/for/2.x or add an exception to the branch 2.x copy of bin/ignored_commits.json. Thanks, your friendly bot at https://jenkins.impala.io/job/cherrypick-2.x-and-test/629/ . -- To view, visit http://gerrit.cloudera.org:8080/10611 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id55f7d2e94d81e66ce720acb6315f15a89621b31 Gerrit-Change-Number: 10611 Gerrit-PatchSet: 7 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 15 Jun 2018 00:48:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6917: Implement COMMENT ON TABLE/VIEW
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10478 ) Change subject: IMPALA-6917: Implement COMMENT ON TABLE/VIEW .. IMPALA-6917: Implement COMMENT ON TABLE/VIEW This patch implements updating comment on a table or view. Syntax: COMMENT ON TABLE t IS 'comment' COMMENT ON VIEW v IS 'comment' Testing: - Added new front-end tests - Ran all front-end tests - Added new end-to-end tests - Ran end-to-end DDL tests Change-Id: I497c17342f79ff7c99931fd8a0ddec0c79303dbf Reviewed-on: http://gerrit.cloudera.org:8080/10478 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M common/thrift/JniCatalog.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/CommentOnDbStmt.java M fe/src/main/java/org/apache/impala/analysis/CommentOnStmt.java A fe/src/main/java/org/apache/impala/analysis/CommentOnTableOrViewStmt.java A fe/src/main/java/org/apache/impala/analysis/CommentOnTableStmt.java A fe/src/main/java/org/apache/impala/analysis/CommentOnViewStmt.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M tests/metadata/test_ddl.py M tests/metadata/test_ddl_base.py 13 files changed, 339 insertions(+), 5 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/10478 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I497c17342f79ff7c99931fd8a0ddec0c79303dbf Gerrit-Change-Number: 10478 Gerrit-PatchSet: 9 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-6917: Implement COMMENT ON TABLE/VIEW
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10478 ) Change subject: IMPALA-6917: Implement COMMENT ON TABLE/VIEW .. Patch Set 8: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10478 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I497c17342f79ff7c99931fd8a0ddec0c79303dbf Gerrit-Change-Number: 10478 Gerrit-PatchSet: 8 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 15 Jun 2018 00:42:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6812: Fix flaky Kudu scan tests
David Ribeiro Alves has posted comments on this change. ( http://gerrit.cloudera.org:8080/10503 ) Change subject: IMPALA-6812: Fix flaky Kudu scan tests .. Patch Set 3: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/10503 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70df84f2cbc663107f2ad029565d3c15bdfbd47c Gerrit-Change-Number: 10503 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 15 Jun 2018 00:41:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6812: Fix flaky Kudu scan tests
David Ribeiro Alves has posted comments on this change. ( http://gerrit.cloudera.org:8080/10503 ) Change subject: IMPALA-6812: Fix flaky Kudu scan tests .. Patch Set 3: Code-Review-1 (1 comment) http://gerrit.cloudera.org:8080/#/c/10503/3/be/src/exec/kudu-util.h File be/src/exec/kudu-util.h: http://gerrit.cloudera.org:8080/#/c/10503/3/be/src/exec/kudu-util.h@112 PS3, Line 112: nit extra space -- To view, visit http://gerrit.cloudera.org:8080/10503 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70df84f2cbc663107f2ad029565d3c15bdfbd47c Gerrit-Change-Number: 10503 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 15 Jun 2018 00:41:17 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7121: Clean up partitionIds from HdfsTable
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10654 ) Change subject: IMPALA-7121: Clean up partitionIds_ from HdfsTable .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/10654/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/10654/2//COMMIT_MSG@16 PS2, Line 16: One thing needs extra attention here is that modifying the result of > I'm not sure I get the point. I have replace all the matchingIds.addAll(tbl Ah got it, I was just not paying close enough attention. I think either of Todd's options makes sense though... -- To view, visit http://gerrit.cloudera.org:8080/10654 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8b5a480e570aeae565fafd4f3e2b279e7a98c7da Gerrit-Change-Number: 10654 Gerrit-PatchSet: 2 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 15 Jun 2018 00:33:14 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6625: Skip computing parquet conjuncts for non-Parquet scans
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10704 ) Change subject: IMPALA-6625: Skip computing parquet conjuncts for non-Parquet scans .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/10704/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/10704/2//COMMIT_MSG@23 PS2, Line 23: This was done because the eventual explain plan returned to Can we add a test for this case to parquet-filtering.test? Maybe we can just add a test that points at functional (which has text files) rather than functional_parquet. It would also be good to add a test for a mixed-format table with some parquet to make sure that the predicates are still created in that case. functional.complextypes_multifileformat is the one table we have, I think, but we should probably add more IMPALA-6874 -- To view, visit http://gerrit.cloudera.org:8080/10704 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9d6c26d42db090c8a15c602f6419ad6399c329e7 Gerrit-Change-Number: 10704 Gerrit-PatchSet: 2 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 15 Jun 2018 00:15:02 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6034: Add Cpu and scanned bytes limits per query
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10415 ) Change subject: IMPALA-6034: Add Cpu and scanned bytes limits per query .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10415 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c6015e21da684bb9f33e236d71309dd4c178a20 Gerrit-Change-Number: 10415 Gerrit-PatchSet: 6 Gerrit-Owner: Mostafa Mokhtar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 15 Jun 2018 00:11:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6942: Reword error message to say "Failed" rather than "Cancelled"
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10717 ) Change subject: IMPALA-6942: Reword error message to say "Failed" rather than "Cancelled" .. IMPALA-6942: Reword error message to say "Failed" rather than "Cancelled" In this case, the query is failing. It happens to use the cancellation path to cleanup, but from a user's perspective this is a query failure not a cancellation. Reword the message to reflect that. Change-Id: I4d8e755aef196e5c25205094af9c8486eb899344 Reviewed-on: http://gerrit.cloudera.org:8080/10717 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/service/impala-server.cc 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/10717 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I4d8e755aef196e5c25205094af9c8486eb899344 Gerrit-Change-Number: 10717 Gerrit-PatchSet: 4 Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6942: Reword error message to say "Failed" rather than "Cancelled"
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10717 ) Change subject: IMPALA-6942: Reword error message to say "Failed" rather than "Cancelled" .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10717 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4d8e755aef196e5c25205094af9c8486eb899344 Gerrit-Change-Number: 10717 Gerrit-PatchSet: 3 Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 15 Jun 2018 00:00:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7174: fix test cancellation for RELEASE builds
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10722 ) Change subject: IMPALA-7174: fix test_cancellation for RELEASE builds .. Patch Set 3: Hit HBase flake in data log -- To view, visit http://gerrit.cloudera.org:8080/10722 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I41da7b5ac58a468a8ed11969906f63df6d4b Gerrit-Change-Number: 10722 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 14 Jun 2018 23:57:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7174: fix test cancellation for RELEASE builds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10722 ) Change subject: IMPALA-7174: fix test_cancellation for RELEASE builds .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10722 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I41da7b5ac58a468a8ed11969906f63df6d4b Gerrit-Change-Number: 10722 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 14 Jun 2018 23:55:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7174: fix test cancellation for RELEASE builds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10722 ) Change subject: IMPALA-7174: fix test_cancellation for RELEASE builds .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2681/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/10722 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I41da7b5ac58a468a8ed11969906f63df6d4b Gerrit-Change-Number: 10722 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 14 Jun 2018 23:55:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7174: fix test cancellation for RELEASE builds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10722 ) Change subject: IMPALA-7174: fix test_cancellation for RELEASE builds .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2680/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/10722 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I41da7b5ac58a468a8ed11969906f63df6d4b Gerrit-Change-Number: 10722 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 14 Jun 2018 23:54:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5604: document DISABLE CODEGEN ROWS THRESHOLD
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10727 Change subject: IMPALA-5604: document DISABLE_CODEGEN_ROWS_THRESHOLD .. IMPALA-5604: document DISABLE_CODEGEN_ROWS_THRESHOLD Also fix a couple of nits in EXEC_SINGLE_NODE_ROWS_THRESHOLD. Change-Id: I709cd55e3869888feb645f85e61a99901d41d479 --- M docs/impala.ditamap M docs/impala_keydefs.ditamap A docs/topics/impala_disable_codegen_rows_threshold.xml M docs/topics/impala_exec_single_node_rows_threshold.xml 4 files changed, 105 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/10727/1 -- To view, visit http://gerrit.cloudera.org:8080/10727 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I709cd55e3869888feb645f85e61a99901d41d479 Gerrit-Change-Number: 10727 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-7174: fix test cancellation for RELEASE builds
Dan Hecht has posted comments on this change. ( http://gerrit.cloudera.org:8080/10722 ) Change subject: IMPALA-7174: fix test_cancellation for RELEASE builds .. Patch Set 2: Code-Review+2 Sounds fine. And I'll be sure not to revert it with https://gerrit.cloudera.org/#/c/10690/ -- To view, visit http://gerrit.cloudera.org:8080/10722 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I41da7b5ac58a468a8ed11969906f63df6d4b Gerrit-Change-Number: 10722 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 14 Jun 2018 23:45:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7174: fix test cancellation for RELEASE builds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10722 ) Change subject: IMPALA-7174: fix test_cancellation for RELEASE builds .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2678/ -- To view, visit http://gerrit.cloudera.org:8080/10722 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I41da7b5ac58a468a8ed11969906f63df6d4b Gerrit-Change-Number: 10722 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 14 Jun 2018 23:41:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7128 (part 2): add an interface for data sources
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10626 ) Change subject: IMPALA-7128 (part 2): add an interface for data sources .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10626 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe704197dc2ad7c09b8340865f17567096aa630e Gerrit-Change-Number: 10626 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 14 Jun 2018 23:38:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7128 (part 2): add an interface for data sources
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10626 ) Change subject: IMPALA-7128 (part 2): add an interface for data sources .. IMPALA-7128 (part 2): add an interface for data sources This changes most of the usage of DataSource and DataSourceTable to use interfaces instead of implementation classes. There are still various usages of the implementation for functionality like creating and dropping data sources. We'll address those as part of IMPALA-7131 at a later date. Change-Id: Ibe704197dc2ad7c09b8340865f17567096aa630e Reviewed-on: http://gerrit.cloudera.org:8080/10626 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/CreateTableDataSrcStmt.java M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java M fe/src/main/java/org/apache/impala/catalog/DataSource.java M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java M fe/src/main/java/org/apache/impala/catalog/FeCatalog.java A fe/src/main/java/org/apache/impala/catalog/FeDataSource.java A fe/src/main/java/org/apache/impala/catalog/FeDataSourceTable.java M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java 14 files changed, 110 insertions(+), 32 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/10626 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ibe704197dc2ad7c09b8340865f17567096aa630e Gerrit-Change-Number: 10626 Gerrit-PatchSet: 7 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-5168: Codegen HASH PARTITIONED KrpcDataStreamSender::Send()
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10421 ) Change subject: IMPALA-5168: Codegen HASH_PARTITIONED KrpcDataStreamSender::Send() .. IMPALA-5168: Codegen HASH_PARTITIONED KrpcDataStreamSender::Send() This change codegens the hash partitioning logic of KrpcDataStreamSender::Send() when the partitioning strategy is HASH_PARTITIONED. It does so by unrolling the loop which evaluates each row against the partitioning expressions and hashes the result. It also replaces the number of channels of that sender with a constant at runtime. With this change, we get reasonable speedup with some benchmarks: ++---+-++++ | Workload | File Format | Avg (s) | Delta(Avg) | GeoMean(s) | Delta(GeoMean) | ++---+-++++ | TPCH(_300) | parquet / none / none | 20.03 | -6.44% | 13.56 | -7.15% | ++---+-++++ +-+---+-++++ | Workload| File Format | Avg (s) | Delta(Avg) | GeoMean(s) | Delta(GeoMean) | +-+---+-++++ | TARGETED-PERF(_300) | parquet / none / none | 58.59 | -5.56% | 12.28 | -5.30% | +-+---+-++++ +-+---+-++++ | Workload| File Format | Avg (s) | Delta(Avg) | GeoMean(s) | Delta(GeoMean) | +-+---+-++++ | TPCDS-UNMODIFIED(_1000) | parquet / none / none | 15.60 | -3.10% | 7.16 | -4.33% | +-+---+-++++ +---+---+-++++ | Workload | File Format | Avg (s) | Delta(Avg) | GeoMean(s) | Delta(GeoMean) | +---+---+-++++ | TPCH_NESTED(_300) | parquet / none / none | 30.93 | -3.02% | 17.46 | -4.71% | +---+---+-++++ Change-Id: I1c44cc9312c062cc7a5a4ac9156ceaa31fb887ff Reviewed-on: http://gerrit.cloudera.org:8080/10421 Reviewed-by: Michael Ho Tested-by: Impala Public Jenkins --- M be/src/codegen/gen_ir_descriptions.py M be/src/codegen/impala-ir.cc M be/src/exec/data-sink.cc M be/src/exec/data-sink.h M be/src/exec/exchange-node.cc M be/src/exec/exec-node.cc M be/src/exec/exec-node.h M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/partial-sort-node.cc M be/src/exec/partitioned-aggregation-node.cc M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-builder.h M be/src/exec/partitioned-hash-join-node.cc M be/src/exec/sort-node.cc M be/src/exec/topn-node.cc M be/src/runtime/CMakeLists.txt M be/src/runtime/fragment-instance-state.cc A be/src/runtime/krpc-data-stream-sender-ir.cc M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/krpc-data-stream-sender.h M be/src/runtime/raw-value-ir.cc M be/src/runtime/raw-value.cc M be/src/runtime/runtime-state.h M be/src/util/runtime-profile.h A testdata/workloads/functional-query/queries/QueryTest/datastream-sender-codegen.test M tests/query_test/test_codegen.py 26 files changed, 425 insertions(+), 101 deletions(-) Approvals: Michael Ho: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/10421 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I1c44cc9312c062cc7a5a4ac9156ceaa31fb887ff Gerrit-Change-Number: 10421 Gerrit-PatchSet: 10 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-5168: Codegen HASH PARTITIONED KrpcDataStreamSender::Send()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10421 ) Change subject: IMPALA-5168: Codegen HASH_PARTITIONED KrpcDataStreamSender::Send() .. Patch Set 9: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10421 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1c44cc9312c062cc7a5a4ac9156ceaa31fb887ff Gerrit-Change-Number: 10421 Gerrit-PatchSet: 9 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 14 Jun 2018 23:36:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6305: Allow column definitions in ALTER VIEW
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/10720 )
Change subject: IMPALA-6305: Allow column definitions in ALTER VIEW
..
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/10720/2/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java:
http://gerrit.cloudera.org:8080/#/c/10720/2/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java@1101
PS2, Line 1101: AnalyzesOk("alter view functional.alltypes_view (aaa, bbb)
as "
Add a test for altering a view with the same column as the existing one.
http://gerrit.cloudera.org:8080/#/c/10720/2/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java@1105
PS2, Line 1105: AnalyzesOk("alter view functional.alltypes_view (cnt) as "
Can we alter a view from its own?
alter view functional.alltypes_view (new_int_col) as select int_col from
functional.alltypes_view
If yes, does that mean that this is one possible way to rename column names and
comments?
--
To view, visit http://gerrit.cloudera.org:8080/10720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6073444a814a24d97e80df15fcd39be2812f63fc
Gerrit-Change-Number: 10720
Gerrit-PatchSet: 2
Gerrit-Owner: Pooja Nilangekar
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Pooja Nilangekar
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Thu, 14 Jun 2018 23:25:02 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6802 (part 4): Clean up authorization tests
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10442 ) Change subject: IMPALA-6802 (part 4): Clean up authorization tests .. Patch Set 4: I'm fine with it.. one more small change needed to comment style -- To view, visit http://gerrit.cloudera.org:8080/10442 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic4cf3a4751b41908ef81ec35d89a2713d9fa0dc4 Gerrit-Change-Number: 10442 Gerrit-PatchSet: 4 Gerrit-Owner: Adam Holley Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 14 Jun 2018 23:14:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6305: Allow column definitions in ALTER VIEW
Vuk Ercegovac has posted comments on this change. (
http://gerrit.cloudera.org:8080/10720 )
Change subject: IMPALA-6305: Allow column definitions in ALTER VIEW
..
Patch Set 2:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/10720/2/fe/src/main/java/org/apache/impala/analysis/AlterViewStmt.java
File fe/src/main/java/org/apache/impala/analysis/AlterViewStmt.java:
http://gerrit.cloudera.org:8080/#/c/10720/2/fe/src/main/java/org/apache/impala/analysis/AlterViewStmt.java@36
PS2, Line 36: TableName tableName, ArrayList columnDefs,
QueryStmt viewDefStmt) {
> Use List instead of ArrayList
Likely extends to the super class. If so, would be good to fix that as well.
http://gerrit.cloudera.org:8080/#/c/10720/2/fe/src/main/java/org/apache/impala/analysis/AlterViewStmt.java@44
PS2, Line 44: viewDefStmt_.analyze(analyzer);
have a look at CreateViewStmt's analyze. would be useful to double-check the
delta-- might indicate a test gap or perhaps the create view does more than
needed? couple of diffs to note:
- complex typed columns
- accessEvent?
http://gerrit.cloudera.org:8080/#/c/10720/2/fe/src/main/java/org/apache/impala/analysis/AlterViewStmt.java@69
PS2, Line 69: }
add the column defs? see CreateViewStmt
is there a test for this?
http://gerrit.cloudera.org:8080/#/c/10720/2/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java:
http://gerrit.cloudera.org:8080/#/c/10720/2/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java@1094
PS2, Line 1094: + "select int_col x, string_col y from
functional.alltypes");
does "*" in the select also work?
you might increase coverage by piggy-backing the alter view with existing
create view tests (got the idea since the two have a common parent class)
http://gerrit.cloudera.org:8080/#/c/10720/2/testdata/workloads/functional-query/queries/QueryTest/views-ddl.test
File testdata/workloads/functional-query/queries/QueryTest/views-ddl.test:
http://gerrit.cloudera.org:8080/#/c/10720/2/testdata/workloads/functional-query/queries/QueryTest/views-ddl.test@234
PS2, Line 234: abc comment 'agg', xyz comment 'gby'
these verify that the new syntax is admitted. can you also check that columns
can be renamed (and verified via describe and a query)?
--
To view, visit http://gerrit.cloudera.org:8080/10720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6073444a814a24d97e80df15fcd39be2812f63fc
Gerrit-Change-Number: 10720
Gerrit-PatchSet: 2
Gerrit-Owner: Pooja Nilangekar
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Pooja Nilangekar
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Thu, 14 Jun 2018 23:12:49 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10709 ) Change subject: IMPALA-7111: avoid use of boost::split in CheckPluginEnabled .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10709 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8 Gerrit-Change-Number: 10709 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 14 Jun 2018 22:46:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10709 ) Change subject: IMPALA-7111: avoid use of boost::split in CheckPluginEnabled .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2679/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/10709 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8 Gerrit-Change-Number: 10709 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 14 Jun 2018 22:46:39 + Gerrit-HasComments: No
[Impala-ASF-CR] Fail cleanly when in process server can't bind
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10726 Change subject: Fail cleanly when in process server can't bind .. Fail cleanly when in process server can't bind This doesn't solve IMPALA-7151 but makes the failure cleaner because there is no crash. Change-Id: I376a2aa559f4b5cf3b96fa3465520e9983ecec4b --- M be/src/exprs/expr-test.cc M be/src/service/session-expiry-test.cc M be/src/statestore/statestore-test.cc M be/src/testutil/in-process-servers.cc M be/src/testutil/in-process-servers.h 5 files changed, 26 insertions(+), 27 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/10726/1 -- To view, visit http://gerrit.cloudera.org:8080/10726 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I376a2aa559f4b5cf3b96fa3465520e9983ecec4b Gerrit-Change-Number: 10726 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/10709 ) Change subject: IMPALA-7111: avoid use of boost::split in CheckPluginEnabled .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10709 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8 Gerrit-Change-Number: 10709 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 14 Jun 2018 22:38:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10709 ) Change subject: IMPALA-7111: avoid use of boost::split in CheckPluginEnabled .. Patch Set 3: Does someone want to +2? -- To view, visit http://gerrit.cloudera.org:8080/10709 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8 Gerrit-Change-Number: 10709 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 14 Jun 2018 22:36:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6802 (part 4): Clean up authorization tests
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10442 ) Change subject: IMPALA-6802 (part 4): Clean up authorization tests .. Patch Set 4: I think this should be ok to merge whenever Vuk is happy with it. -- To view, visit http://gerrit.cloudera.org:8080/10442 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic4cf3a4751b41908ef81ec35d89a2713d9fa0dc4 Gerrit-Change-Number: 10442 Gerrit-PatchSet: 4 Gerrit-Owner: Adam Holley Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 14 Jun 2018 22:29:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6305: Allow column definitions in ALTER VIEW
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/10720 )
Change subject: IMPALA-6305: Allow column definitions in ALTER VIEW
..
Patch Set 2:
(12 comments)
Can also update the AuthorizationTest with the new syntax?
http://gerrit.cloudera.org:8080/#/c/10720/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/10720/2//COMMIT_MSG@11
PS2, Line 11: and the AlterViewStmt constructor.
Add an example of the new SQL syntax.
http://gerrit.cloudera.org:8080/#/c/10720/2/fe/src/main/java/org/apache/impala/analysis/AlterViewStmt.java
File fe/src/main/java/org/apache/impala/analysis/AlterViewStmt.java:
http://gerrit.cloudera.org:8080/#/c/10720/2/fe/src/main/java/org/apache/impala/analysis/AlterViewStmt.java@36
PS2, Line 36: TableName tableName, ArrayList columnDefs,
QueryStmt viewDefStmt) {
Use List instead of ArrayList
http://gerrit.cloudera.org:8080/#/c/10720/2/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java:
http://gerrit.cloudera.org:8080/#/c/10720/2/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java@1091
PS2, Line 1091: +
nit: looking at the style of other code in this file, we use + and then new
line. Similarly with the rest of the code below.
http://gerrit.cloudera.org:8080/#/c/10720/2/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java@1146
PS2, Line 1146:
nit: remove indentation
http://gerrit.cloudera.org:8080/#/c/10720/2/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java@1148
PS2, Line 1148:
nit: remove indentation
http://gerrit.cloudera.org:8080/#/c/10720/2/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java@1150
PS2, Line 1150:
nit: remove indentation
http://gerrit.cloudera.org:8080/#/c/10720/2/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java@1152
PS2, Line 1152:
nit: remove indentation
http://gerrit.cloudera.org:8080/#/c/10720/2/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java@1155
PS2, Line 1155:
nit: remove indentation
http://gerrit.cloudera.org:8080/#/c/10720/2/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java@1156
PS2, Line 1156:
nit: remove indentation
http://gerrit.cloudera.org:8080/#/c/10720/2/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java@1160
PS2, Line 1160:
nit: remove indentation
http://gerrit.cloudera.org:8080/#/c/10720/2/testdata/workloads/functional-query/queries/QueryTest/views-ddl.test
File testdata/workloads/functional-query/queries/QueryTest/views-ddl.test:
http://gerrit.cloudera.org:8080/#/c/10720/2/testdata/workloads/functional-query/queries/QueryTest/views-ddl.test@241
PS2, Line 241: 'View has been altered.'
Can we also verify the altered view?
http://gerrit.cloudera.org:8080/#/c/10720/2/testdata/workloads/functional-query/queries/QueryTest/views-ddl.test@248
PS2, Line 248: 'View has been altered.'
Verify the altered view.
--
To view, visit http://gerrit.cloudera.org:8080/10720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6073444a814a24d97e80df15fcd39be2812f63fc
Gerrit-Change-Number: 10720
Gerrit-PatchSet: 2
Gerrit-Owner: Pooja Nilangekar
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Pooja Nilangekar
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Thu, 14 Jun 2018 22:28:59 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5552: Add support for authorized proxy groups
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10510 ) Change subject: IMPALA-5552: Add support for authorized proxy groups .. Patch Set 17: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10510 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6953f89c293b06b72f523b11802232133d9d6cbb Gerrit-Change-Number: 10510 Gerrit-PatchSet: 17 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 14 Jun 2018 22:26:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5552: Add support for authorized proxy groups
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10510 ) Change subject: IMPALA-5552: Add support for authorized proxy groups .. IMPALA-5552: Add support for authorized proxy groups The patch adds support for mapping of users to a list of proxy groups. The following flags are added in impalad: - authorized_proxy_group_config - authorized_proxy_group_config_delimiter Example: --authorized_proxy_group_config=hue=group1,group2;user1=* This feature is not supported on Shell-based Hadoop groups mapping providers. The authorized_proxy_user/group_config parser will now strip leading and trailing whitespaces from the user/group names. Testing: - Added FE unit test to check for groups mapping provider - Added BE unit test for the parsing logic - Added a new test in test_authorization.py - Ran all end-to-end test_authorization.py Change-Id: I6953f89c293b06b72f523b11802232133d9d6cbb Reviewed-on: http://gerrit.cloudera.org:8080/10510 Tested-by: Impala Public Jenkins Reviewed-by: Tim Armstrong --- M be/src/service/CMakeLists.txt M be/src/service/frontend.cc M be/src/service/frontend.h A be/src/service/impala-server-test.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/Frontend.thrift M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A fe/src/test/java/org/apache/impala/service/JniFrontendTest.java M tests/authorization/test_authorization.py 13 files changed, 384 insertions(+), 36 deletions(-) Approvals: Impala Public Jenkins: Verified Tim Armstrong: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/10510 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I6953f89c293b06b72f523b11802232133d9d6cbb Gerrit-Change-Number: 10510 Gerrit-PatchSet: 18 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7174: fix test cancellation for RELEASE builds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10722 ) Change subject: IMPALA-7174: fix test_cancellation for RELEASE builds .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2678/ -- To view, visit http://gerrit.cloudera.org:8080/10722 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I41da7b5ac58a468a8ed11969906f63df6d4b Gerrit-Change-Number: 10722 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 14 Jun 2018 22:20:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7174: xfail test admission controller.test cancellation
Tim Armstrong has abandoned this change. ( http://gerrit.cloudera.org:8080/10719 ) Change subject: IMPALA-7174: xfail test_admission_controller.test_cancellation .. Abandoned Replaced with the actual fix. -- To view, visit http://gerrit.cloudera.org:8080/10719 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I4567fe11ffd482156286f6c657b3cdcf09600ae0 Gerrit-Change-Number: 10719 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7174: fix test cancellation for RELEASE builds
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10722 Change subject: IMPALA-7174: fix test_cancellation for RELEASE builds .. IMPALA-7174: fix test_cancellation for RELEASE builds The test was DOA when run against a release build because the debug actions that it depends on were disabled. The fix is to enable the debug actions for release builds, similar to other debug actions. I assume the original motivation of the NDEBUG checks was to avoid adding overhead to release builds. The cost is minimised by quickly checking whether the string is empty before proceeding with any further work. Testing: Looped the test on a release build. Change-Id: I41da7b5ac58a468a8ed11969906f63df6d4b --- M be/src/scheduling/admission-controller.cc M be/src/service/client-request-state.cc M be/src/util/debug-util.cc M be/src/util/debug-util.h 4 files changed, 12 insertions(+), 17 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/10722/1 -- To view, visit http://gerrit.cloudera.org:8080/10722 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I41da7b5ac58a468a8ed11969906f63df6d4b Gerrit-Change-Number: 10722 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-6305: Allow column definitions in ALTER VIEW
Pooja Nilangekar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10720 Change subject: IMPALA-6305: Allow column definitions in ALTER VIEW .. IMPALA-6305: Allow column definitions in ALTER VIEW This change adds support to change column definitions in ALTER VIEW statements. This support only required minor changes in the parser and the AlterViewStmt constructor. The following tests were modified: 1. ParserTest - To check that the parser handles column definitions for alter view statements. 2. AnalyzerDDLTest - To ensure that the analyzer supports the change column definitions parsed. 3. TestDdlStatements - To verify the end-to-end functioning of ALTER VIEW statements with change column definitions. Change-Id: I6073444a814a24d97e80df15fcd39be2812f63fc --- M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AlterViewStmt.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M testdata/workloads/functional-query/queries/QueryTest/views-ddl.test 5 files changed, 74 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/10720/2 -- To view, visit http://gerrit.cloudera.org:8080/10720 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I6073444a814a24d97e80df15fcd39be2812f63fc Gerrit-Change-Number: 10720 Gerrit-PatchSet: 2 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6812: Fix flaky Kudu scan tests
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10503 ) Change subject: IMPALA-6812: Fix flaky Kudu scan tests .. Patch Set 3: Code-Review+1 Got it.. thanks for pointing me to that. -- To view, visit http://gerrit.cloudera.org:8080/10503 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70df84f2cbc663107f2ad029565d3c15bdfbd47c Gerrit-Change-Number: 10503 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 14 Jun 2018 22:07:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7174: xfail test admission controller.test cancellation
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10719 ) Change subject: IMPALA-7174: xfail test_admission_controller.test_cancellation .. Patch Set 2: I'm pretty sure I figured out the root cause of the test failure. I'll let this merge job continue but will push out the follow-on patch once I've confirmed that it works. -- To view, visit http://gerrit.cloudera.org:8080/10719 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4567fe11ffd482156286f6c657b3cdcf09600ae0 Gerrit-Change-Number: 10719 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 14 Jun 2018 22:05:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6812: Fix flaky Kudu scan tests
Thomas Marshall has posted comments on this change. (
http://gerrit.cloudera.org:8080/10503 )
Change subject: IMPALA-6812: Fix flaky Kudu scan tests
..
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/10503/3/tests/query_test/test_kudu.py
File tests/query_test/test_kudu.py:
http://gerrit.cloudera.org:8080/#/c/10503/3/tests/query_test/test_kudu.py@310
PS3, Line 310: cursor.execute("set kudu_read_mode=READ_AT_SNAPSHOT")
> I'm not familiar with the cursor fixture - does it create a new session for
Yes, there is a new cursor for each test. See the comment in tests/conftest.py
at 'def cursor()'
--
To view, visit http://gerrit.cloudera.org:8080/10503
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I70df84f2cbc663107f2ad029565d3c15bdfbd47c
Gerrit-Change-Number: 10503
Gerrit-PatchSet: 3
Gerrit-Owner: Thomas Marshall
Gerrit-Reviewer: Dan Hecht
Gerrit-Reviewer: David Ribeiro Alves
Gerrit-Reviewer: Thomas Marshall
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Todd Lipcon
Gerrit-Comment-Date: Thu, 14 Jun 2018 21:54:12 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3816, IMPALA-4065: Remove the indirection to TupleRowComparator::Compare()
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/10680 ) Change subject: IMPALA-3816, IMPALA-4065: Remove the indirection to TupleRowComparator::Compare() .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/10680/3/be/src/codegen/gen_ir_descriptions.py File be/src/codegen/gen_ir_descriptions.py: http://gerrit.cloudera.org:8080/#/c/10680/3/be/src/codegen/gen_ir_descriptions.py@203 PS3, Line 203: ["COMPARE_INTERPRETED", : "_ZNK6impala18TupleRowComparator18CompareInterpretedEPKNS_8TupleRowES3_"] It seems odd that we are including this interpreted code into cross-compilation only to use it for replacement. Is there a reason the typical approach of matching against the function name not working or is it somehow not robust enough ? http://gerrit.cloudera.org:8080/#/c/10680/3/be/src/codegen/llvm-codegen.cc File be/src/codegen/llvm-codegen.cc: http://gerrit.cloudera.org:8080/#/c/10680/3/be/src/codegen/llvm-codegen.cc@915 PS3, Line 915: llvm::Function* LlvmCodeGen::ReplaceCallSitesRecursively(llvm::Function* caller, As discussed offline, this is an interesting idea. On the other hand, this doesn't seem easy to maintain in its current form. For the TopNNode, it may be worth exploring whether using the constant argument approach is feasible. For the sorter and the rest, we should be able to achieve what we want to do with the normal clone and replacement albeit a bit tedious. If we decide to go down this route in the long run, I think it warrants more discussion (e.g. can it inadvertently clone a large set of functions which we didn't anticipate because we automate it with this algorithm here now ?) Also, the algorithm implemented probably needs to be validated with functions which exhibit different types of call graphs to validate its behavior. Most importantly, it needs more documentation (e.g. how does it work in general, and why DFS doesn't work etc) and it can be broken down into smaller functions. With all that said, I am not sure we need to implement this algorithm in this patch. -- To view, visit http://gerrit.cloudera.org:8080/10680 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4657ac09daf20408797856d94521d417d8cf171 Gerrit-Change-Number: 10680 Gerrit-PatchSet: 3 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Thu, 14 Jun 2018 21:46:52 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6812: Fix flaky Kudu scan tests
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/10503 )
Change subject: IMPALA-6812: Fix flaky Kudu scan tests
..
Patch Set 3:
(1 comment)
Thanks for doing this, seems like a really pragmatic solution. Just had one
question.
http://gerrit.cloudera.org:8080/#/c/10503/3/tests/query_test/test_kudu.py
File tests/query_test/test_kudu.py:
http://gerrit.cloudera.org:8080/#/c/10503/3/tests/query_test/test_kudu.py@310
PS3, Line 310: cursor.execute("set kudu_read_mode=READ_AT_SNAPSHOT")
I'm not familiar with the cursor fixture - does it create a new session for
every test? Otherwise if it's reused we might end up with the query options
leaking into the next test.
--
To view, visit http://gerrit.cloudera.org:8080/10503
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I70df84f2cbc663107f2ad029565d3c15bdfbd47c
Gerrit-Change-Number: 10503
Gerrit-PatchSet: 3
Gerrit-Owner: Thomas Marshall
Gerrit-Reviewer: Dan Hecht
Gerrit-Reviewer: David Ribeiro Alves
Gerrit-Reviewer: Thomas Marshall
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Todd Lipcon
Gerrit-Comment-Date: Thu, 14 Jun 2018 21:38:15 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7174: xfail test admission controller.test cancellation
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10719 Change subject: IMPALA-7174: xfail test_admission_controller.test_cancellation .. IMPALA-7174: xfail test_admission_controller.test_cancellation XFAIL for now if we hit an exception until we can figure out the cause of the flakiness. Testing: Added a bogus exception, made sure that the test got xfailed correctly. Ensured that the test passes normally when we're not in a flaky case. Change-Id: I4567fe11ffd482156286f6c657b3cdcf09600ae0 --- M tests/custom_cluster/test_admission_controller.py 1 file changed, 5 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/10719/1 -- To view, visit http://gerrit.cloudera.org:8080/10719 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I4567fe11ffd482156286f6c657b3cdcf09600ae0 Gerrit-Change-Number: 10719 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-7174: xfail test admission controller.test cancellation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10719 ) Change subject: IMPALA-7174: xfail test_admission_controller.test_cancellation .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2677/ -- To view, visit http://gerrit.cloudera.org:8080/10719 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4567fe11ffd482156286f6c657b3cdcf09600ae0 Gerrit-Change-Number: 10719 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 14 Jun 2018 21:25:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7140 (part 2). Create skeleton for LocalFsTable
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10712 ) Change subject: IMPALA-7140 (part 2). Create skeleton for LocalFsTable .. Patch Set 2: Code-Review+2 (3 comments) http://gerrit.cloudera.org:8080/#/c/10712/2/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java File fe/src/main/java/org/apache/impala/catalog/FeFsTable.java: http://gerrit.cloudera.org:8080/#/c/10712/2/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java@82 PS2, Line 82: Hdfs sorry about not catching this before, but pls add a todo (assign it to me) to change the "Hdfs" to "Fs" for consistent naming with this interface. looking at all the places where this is used, the misalignment stands out. http://gerrit.cloudera.org:8080/#/c/10712/2/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java File fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java: http://gerrit.cloudera.org:8080/#/c/10712/2/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java@164 PS2, Line 164: public Map> getFilesSample(Collection inputParts, nit: too long line http://gerrit.cloudera.org:8080/#/c/10712/2/fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java File fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java: http://gerrit.cloudera.org:8080/#/c/10712/2/fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java@95 PS2, Line 95: "__HIVE_DEFAULT_PARTITION__" make this a public member so the test can use it as well. -- To view, visit http://gerrit.cloudera.org:8080/10712 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id2b184104d92e128250df5a08ac7ffb3dde011a8 Gerrit-Change-Number: 10712 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 14 Jun 2018 21:21:10 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6917: Implement COMMENT ON TABLE/VIEW
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10478 ) Change subject: IMPALA-6917: Implement COMMENT ON TABLE/VIEW .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2676/ -- To view, visit http://gerrit.cloudera.org:8080/10478 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I497c17342f79ff7c99931fd8a0ddec0c79303dbf Gerrit-Change-Number: 10478 Gerrit-PatchSet: 8 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 14 Jun 2018 21:11:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6917: Implement COMMENT ON TABLE/VIEW
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10478 ) Change subject: IMPALA-6917: Implement COMMENT ON TABLE/VIEW .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10478 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I497c17342f79ff7c99931fd8a0ddec0c79303dbf Gerrit-Change-Number: 10478 Gerrit-PatchSet: 8 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 14 Jun 2018 21:11:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6034: Add Cpu and scanned bytes limits per query
Hello Tim Armstrong, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10415 to look at the new patch set (#6). Change subject: IMPALA-6034: Add Cpu and scanned bytes limits per query .. IMPALA-6034: Add Cpu and scanned bytes limits per query To protect Impala from runaway queries add per query limits for Cpu and scanned bytes, limits are enforced via query options and applied to the entire query, not per backend like mem_limit. If a query exceeds any of the limits it will get cancelled. Query profile is updated to include query wide and per backend metrics for Cpu and scanned bytes. Testing: Added tests for various permutations for MAX_CPU_TIME_S and MAX_SCAN_BYTES Change-Id: I4c6015e21da684bb9f33e236d71309dd4c178a20 --- M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/service/query-options-test.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift A testdata/workloads/functional-query/queries/QueryTest/query-resource-limits.test A tests/query_test/test_query_resource_limits.py 13 files changed, 375 insertions(+), 22 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/10415/6 -- To view, visit http://gerrit.cloudera.org:8080/10415 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4c6015e21da684bb9f33e236d71309dd4c178a20 Gerrit-Change-Number: 10415 Gerrit-PatchSet: 6 Gerrit-Owner: Mostafa Mokhtar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6034: Add Cpu and scanned bytes limits per query
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10415 ) Change subject: IMPALA-6034: Add Cpu and scanned bytes limits per query .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2675/ -- To view, visit http://gerrit.cloudera.org:8080/10415 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c6015e21da684bb9f33e236d71309dd4c178a20 Gerrit-Change-Number: 10415 Gerrit-PatchSet: 6 Gerrit-Owner: Mostafa Mokhtar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 14 Jun 2018 20:59:08 + Gerrit-HasComments: No
[Impala-ASF-CR] Optimize dependencies for Codegen
Dan Hecht has posted comments on this change. ( http://gerrit.cloudera.org:8080/10688 ) Change subject: Optimize dependencies for Codegen .. Patch Set 1: Looks like the current way was done as a result of IMPALA-1896. Could you check that JIRA to see if removing this will cause that problem? -- To view, visit http://gerrit.cloudera.org:8080/10688 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie3294cd27c2d35388a04934440a1d2b0ba3a0dd9 Gerrit-Change-Number: 10688 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 14 Jun 2018 20:58:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6812: Fix flaky Kudu scan tests
Hello David Ribeiro Alves, Todd Lipcon, Tim Armstrong, Dan Hecht, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10503 to look at the new patch set (#3). Change subject: IMPALA-6812: Fix flaky Kudu scan tests .. IMPALA-6812: Fix flaky Kudu scan tests Many of our Kudu related tests have been flaky with the symptom that scans appear to not return rows that were just inserted. This occurs because our default Kudu scan level of READ_LATEST doesn't make any consistency guarantees. This patch adds a query option 'kudu_read_mode', which overrides the startup flag of the same name, and then set that option to READ_AT_SNAPSHOT for all tests with Kudu inserts and scans, which should give us more consistent test results. Testing: - Passed a full exhaustive run. Does not appear to increase time to run by any significant amount. Change-Id: I70df84f2cbc663107f2ad029565d3c15bdfbd47c --- M be/src/exec/kudu-scanner.cc M be/src/exec/kudu-util.cc M be/src/exec/kudu-util.h M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M tests/common/test_dimensions.py M tests/custom_cluster/test_kudu.py M tests/metadata/test_ddl.py M tests/query_test/test_kudu.py 13 files changed, 119 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/10503/3 -- To view, visit http://gerrit.cloudera.org:8080/10503 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I70df84f2cbc663107f2ad029565d3c15bdfbd47c Gerrit-Change-Number: 10503 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR](2.x) IMPALA-5216: Make admission control queuing async
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10715 ) Change subject: IMPALA-5216: Make admission control queuing async .. IMPALA-5216: Make admission control queuing async Implement asynchronous admission control queuing. This is achieved by running the admission control code-path in a separate thread. Major changes include: propagating cancellation to the admission control thread and dequeuing thread, and adding a new Query Operation State called "PENDING" that represents the state between completion of planning and starting of query execution. Testing: - Added a deterministic end to end test and a session expiry test. - Ran multiple stress tests successfully with a cancellation probability of 60% and with different values for the following parameters: max_requests, queue_wait_timeout_ms. Ensured that the impalad was in a valid state afterwards (no orphan fragments or wrong metrics). - Ran all exhaustive tests and ASAN core tests successfully. - Ran data load successfully. Change-Id: I989cf5b259afb8f5bc5c35590c94961c81ce88bf Reviewed-on: http://gerrit.cloudera.org:8080/10715 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M be/src/common/atomic.h M be/src/common/logging.h M be/src/rpc/authentication.cc M be/src/runtime/fragment-instance-state.cc M be/src/runtime/query-state.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/query-schedule.cc M be/src/scheduling/query-schedule.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-http-handler.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M be/src/util/promise-test.cc M be/src/util/promise.h M common/thrift/ImpalaService.thrift M tests/authorization/test_authorization.py M tests/beeswax/impala_beeswax.py M tests/common/impala_connection.py M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_krpc_mem_usage.py M tests/custom_cluster/test_session_expiration.py M tests/hs2/hs2_test_suite.py M tests/hs2/test_hs2.py M tests/query_test/test_observability.py M tests/query_test/test_udfs.py M www/query_backends.tmpl 32 files changed, 692 insertions(+), 247 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/10715 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: merged Gerrit-Change-Id: I989cf5b259afb8f5bc5c35590c94961c81ce88bf Gerrit-Change-Number: 10715 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR](2.x) IMPALA-5216: Make admission control queuing async
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10715 ) Change subject: IMPALA-5216: Make admission control queuing async .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10715 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I989cf5b259afb8f5bc5c35590c94961c81ce88bf Gerrit-Change-Number: 10715 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 14 Jun 2018 20:42:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3816: (prep) Move TupleSorter to sorter-ir.cc
Michael Ho has posted comments on this change. (
http://gerrit.cloudera.org:8080/10679 )
Change subject: IMPALA-3816: (prep) Move TupleSorter to sorter-ir.cc
..
Patch Set 3:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/10679/3/be/src/runtime/sorter-internal.h
File be/src/runtime/sorter-internal.h:
http://gerrit.cloudera.org:8080/#/c/10679/3/be/src/runtime/sorter-internal.h@392
PS3, Line 392: //
nit: ///
http://gerrit.cloudera.org:8080/#/c/10679/3/be/src/runtime/sorter-internal.h@396
PS3, Line 396: //
nit: ///
http://gerrit.cloudera.org:8080/#/c/10679/3/be/src/runtime/sorter-internal.h@470
PS3, Line 470: /// if 'lhs' is less than 'rhs'.
May need to add a remark in the follow up patch that comparator_.Less() is
replaced with handcrafted IR during codegen.
http://gerrit.cloudera.org:8080/#/c/10679/3/be/src/runtime/sorter-internal.h@485
PS3, Line 485: const Tuple* pivot, TupleIterator* cut);
WARN_UNUSED_RESULT
http://gerrit.cloudera.org:8080/#/c/10679/3/be/src/runtime/sorter-ir.cc
File be/src/runtime/sorter-ir.cc:
http://gerrit.cloudera.org:8080/#/c/10679/3/be/src/runtime/sorter-ir.cc@209
PS3, Line 209: Tuple* Sorter::TupleSorter::MedianOfThree(Tuple* t1, Tuple* t2,
Tuple* t3) {
> This calls Less(). After my change Less() will always call CompareInterpret
This makes sense. Missed seeing the Less() below in the first pass.
http://gerrit.cloudera.org:8080/#/c/10679/3/be/src/runtime/sorter.cc
File be/src/runtime/sorter.cc:
http://gerrit.cloudera.org:8080/#/c/10679/3/be/src/runtime/sorter.cc@681
PS3, Line 681: Status
: Sorter::Run::AddIntermediateBatch(RowBatch* batch, int
start_index, int* num_processed) {
nit: we usually break the argument lists into multiple rows if it doesn't fit
in a single row.
http://gerrit.cloudera.org:8080/#/c/10679/3/be/src/runtime/sorter.cc@749
PS3, Line 749:
nit: indent off
--
To view, visit http://gerrit.cloudera.org:8080/10679
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaaf2b75c2f789002c42939865c018f728d29a113
Gerrit-Change-Number: 10679
Gerrit-PatchSet: 3
Gerrit-Owner: Tianyi Wang
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Tianyi Wang
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Thu, 14 Jun 2018 20:37:56 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6942: Reword error message to say "Failed" rather than "Cancelled"
Dan Hecht has posted comments on this change. ( http://gerrit.cloudera.org:8080/10717 ) Change subject: IMPALA-6942: Reword error message to say "Failed" rather than "Cancelled" .. Patch Set 2: Code-Review+2 (1 comment) Carry +2 http://gerrit.cloudera.org:8080/#/c/10717/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/10717/1//COMMIT_MSG@7 PS1, Line 7: sa > delete Done -- To view, visit http://gerrit.cloudera.org:8080/10717 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4d8e755aef196e5c25205094af9c8486eb899344 Gerrit-Change-Number: 10717 Gerrit-PatchSet: 2 Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 14 Jun 2018 20:35:14 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [DOCS] Wording changes in DPP examples for clarity
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10718 ) Change subject: [DOCS] Wording changes in DPP examples for clarity .. [DOCS] Wording changes in DPP examples for clarity Change-Id: If786fc3d3064b26b213afb685a2f310cebe904fe Reviewed-on: http://gerrit.cloudera.org:8080/10718 Reviewed-by: Alex Rodoni Tested-by: Impala Public Jenkins --- M docs/shared/impala_common.xml M docs/topics/impala_partitioning.xml M docs/topics/impala_runtime_filtering.xml 3 files changed, 10 insertions(+), 10 deletions(-) Approvals: Alex Rodoni: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/10718 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: If786fc3d3064b26b213afb685a2f310cebe904fe Gerrit-Change-Number: 10718 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-6942: Reword error message to say "Failed" rather than "Cancelled"
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10717 ) Change subject: IMPALA-6942: Reword error message to say "Failed" rather than "Cancelled" .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10717 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4d8e755aef196e5c25205094af9c8486eb899344 Gerrit-Change-Number: 10717 Gerrit-PatchSet: 3 Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 14 Jun 2018 20:35:26 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Wording changes in DPP examples for clarity
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10718 ) Change subject: [DOCS] Wording changes in DPP examples for clarity .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10718 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If786fc3d3064b26b213afb685a2f310cebe904fe Gerrit-Change-Number: 10718 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 14 Jun 2018 20:35:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6942: Reword error message to say "Failed" rather than "Cancelled"
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10717 ) Change subject: IMPALA-6942: Reword error message to say "Failed" rather than "Cancelled" .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2674/ -- To view, visit http://gerrit.cloudera.org:8080/10717 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4d8e755aef196e5c25205094af9c8486eb899344 Gerrit-Change-Number: 10717 Gerrit-PatchSet: 3 Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 14 Jun 2018 20:35:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6942: Reword error message to say "Failed" rather than "Cancelled"
Hello Tim Armstrong, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10717 to look at the new patch set (#2). Change subject: IMPALA-6942: Reword error message to say "Failed" rather than "Cancelled" .. IMPALA-6942: Reword error message to say "Failed" rather than "Cancelled" In this case, the query is failing. It happens to use the cancellation path to cleanup, but from a user's perspective this is a query failure not a cancellation. Reword the message to reflect that. Change-Id: I4d8e755aef196e5c25205094af9c8486eb899344 --- M be/src/service/impala-server.cc 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/17/10717/2 -- To view, visit http://gerrit.cloudera.org:8080/10717 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4d8e755aef196e5c25205094af9c8486eb899344 Gerrit-Change-Number: 10717 Gerrit-PatchSet: 2 Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6942: Reword error message to be say "Failed" rather than "Cancelled"
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10717 ) Change subject: IMPALA-6942: Reword error message to be say "Failed" rather than "Cancelled" .. Patch Set 1: Code-Review+2 I agree with your reasoning -- To view, visit http://gerrit.cloudera.org:8080/10717 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4d8e755aef196e5c25205094af9c8486eb899344 Gerrit-Change-Number: 10717 Gerrit-PatchSet: 1 Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 14 Jun 2018 20:29:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6942: Reword error message to be say "Failed" rather than "Cancelled"
Dan Hecht has posted comments on this change. ( http://gerrit.cloudera.org:8080/10717 ) Change subject: IMPALA-6942: Reword error message to be say "Failed" rather than "Cancelled" .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/10717/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/10717/1//COMMIT_MSG@7 PS1, Line 7: be delete -- To view, visit http://gerrit.cloudera.org:8080/10717 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4d8e755aef196e5c25205094af9c8486eb899344 Gerrit-Change-Number: 10717 Gerrit-PatchSet: 1 Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 14 Jun 2018 20:22:15 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [DOCS] Wording changes in DPP examples for clarity
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10718 ) Change subject: [DOCS] Wording changes in DPP examples for clarity .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-docs-submit/318/ -- To view, visit http://gerrit.cloudera.org:8080/10718 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If786fc3d3064b26b213afb685a2f310cebe904fe Gerrit-Change-Number: 10718 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 14 Jun 2018 20:15:44 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Wording changes in DPP examples for clarity
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10718 Change subject: [DOCS] Wording changes in DPP examples for clarity .. [DOCS] Wording changes in DPP examples for clarity Change-Id: If786fc3d3064b26b213afb685a2f310cebe904fe --- M docs/shared/impala_common.xml M docs/topics/impala_partitioning.xml M docs/topics/impala_runtime_filtering.xml 3 files changed, 10 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/10718/1 -- To view, visit http://gerrit.cloudera.org:8080/10718 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: If786fc3d3064b26b213afb685a2f310cebe904fe Gerrit-Change-Number: 10718 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] IMPALA-7128 (part 2): add an interface for data sources
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10626 ) Change subject: IMPALA-7128 (part 2): add an interface for data sources .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10626 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe704197dc2ad7c09b8340865f17567096aa630e Gerrit-Change-Number: 10626 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 14 Jun 2018 20:14:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7128 (part 2): add an interface for data sources
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10626 ) Change subject: IMPALA-7128 (part 2): add an interface for data sources .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2673/ -- To view, visit http://gerrit.cloudera.org:8080/10626 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe704197dc2ad7c09b8340865f17567096aa630e Gerrit-Change-Number: 10626 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 14 Jun 2018 20:14:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6034: Add Cpu and scanned bytes limits per query
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10415 ) Change subject: IMPALA-6034: Add Cpu and scanned bytes limits per query .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2671/ -- To view, visit http://gerrit.cloudera.org:8080/10415 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c6015e21da684bb9f33e236d71309dd4c178a20 Gerrit-Change-Number: 10415 Gerrit-PatchSet: 5 Gerrit-Owner: Mostafa Mokhtar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 14 Jun 2018 20:13:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7141 (part 1): clean up handling of default/dummy partition
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10711 ) Change subject: IMPALA-7141 (part 1): clean up handling of default/dummy partition .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10711 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I15e91b50eb7c2a5e0bac8c33d603d6cd8cbaca2e Gerrit-Change-Number: 10711 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 14 Jun 2018 20:12:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7141 (part 2). Extract interfaces for partition pruning without loading
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10631 ) Change subject: IMPALA-7141 (part 2). Extract interfaces for partition pruning without loading .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10631 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa Gerrit-Change-Number: 10631 Gerrit-PatchSet: 7 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 14 Jun 2018 20:09:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7140 (part 1). Support fetching schema info in LocalCatalog
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10630 ) Change subject: IMPALA-7140 (part 1). Support fetching schema info in LocalCatalog .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I640f27e36198955e057da62a3ce25a858406e496 Gerrit-Change-Number: 10630 Gerrit-PatchSet: 7 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 14 Jun 2018 20:09:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7135. Skeleton implementation of LocalCatalog
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10627 ) Change subject: IMPALA-7135. Skeleton implementation of LocalCatalog .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10627 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iab653371188b21c72f50ee1ec4e94950aa6fb9ee Gerrit-Change-Number: 10627 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 14 Jun 2018 20:08:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5168: Codegen HASH PARTITIONED KrpcDataStreamSender::Send()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10421 ) Change subject: IMPALA-5168: Codegen HASH_PARTITIONED KrpcDataStreamSender::Send() .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2672/ -- To view, visit http://gerrit.cloudera.org:8080/10421 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1c44cc9312c062cc7a5a4ac9156ceaa31fb887ff Gerrit-Change-Number: 10421 Gerrit-PatchSet: 9 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 14 Jun 2018 20:08:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7137. Support configuring Frontend to use LocalCatalog
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10629 ) Change subject: IMPALA-7137. Support configuring Frontend to use LocalCatalog .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10629 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8c9665bd031d23608740b23eef301970af9aa764 Gerrit-Change-Number: 10629 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 14 Jun 2018 20:08:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6942: Reword error message to be say "Failed" rather than "Cancelled"
Dan Hecht has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10717 Change subject: IMPALA-6942: Reword error message to be say "Failed" rather than "Cancelled" .. IMPALA-6942: Reword error message to be say "Failed" rather than "Cancelled" In this case, the query is failing. It happens to use the cancellation path to cleanup, but from a user's perspective this is a query failure not a cancellation. Reword the message to reflect that. Change-Id: I4d8e755aef196e5c25205094af9c8486eb899344 --- M be/src/service/impala-server.cc 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/17/10717/1 -- To view, visit http://gerrit.cloudera.org:8080/10717 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I4d8e755aef196e5c25205094af9c8486eb899344 Gerrit-Change-Number: 10717 Gerrit-PatchSet: 1 Gerrit-Owner: Dan Hecht
[Impala-ASF-CR] IMPALA-5168: Codegen HASH PARTITIONED KrpcDataStreamSender::Send()
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/10421 ) Change subject: IMPALA-5168: Codegen HASH_PARTITIONED KrpcDataStreamSender::Send() .. Patch Set 9: Code-Review+2 Fixed another clang-tidy error missed in previous patch. -- To view, visit http://gerrit.cloudera.org:8080/10421 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1c44cc9312c062cc7a5a4ac9156ceaa31fb887ff Gerrit-Change-Number: 10421 Gerrit-PatchSet: 9 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 14 Jun 2018 20:04:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5168: Codegen HASH PARTITIONED KrpcDataStreamSender::Send()
Hello Tim Armstrong, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10421 to look at the new patch set (#8). Change subject: IMPALA-5168: Codegen HASH_PARTITIONED KrpcDataStreamSender::Send() .. IMPALA-5168: Codegen HASH_PARTITIONED KrpcDataStreamSender::Send() This change codegens the hash partitioning logic of KrpcDataStreamSender::Send() when the partitioning strategy is HASH_PARTITIONED. It does so by unrolling the loop which evaluates each row against the partitioning expressions and hashes the result. It also replaces the number of channels of that sender with a constant at runtime. With this change, we get reasonable speedup with some benchmarks: ++---+-++++ | Workload | File Format | Avg (s) | Delta(Avg) | GeoMean(s) | Delta(GeoMean) | ++---+-++++ | TPCH(_300) | parquet / none / none | 20.03 | -6.44% | 13.56 | -7.15% | ++---+-++++ +-+---+-++++ | Workload| File Format | Avg (s) | Delta(Avg) | GeoMean(s) | Delta(GeoMean) | +-+---+-++++ | TARGETED-PERF(_300) | parquet / none / none | 58.59 | -5.56% | 12.28 | -5.30% | +-+---+-++++ +-+---+-++++ | Workload| File Format | Avg (s) | Delta(Avg) | GeoMean(s) | Delta(GeoMean) | +-+---+-++++ | TPCDS-UNMODIFIED(_1000) | parquet / none / none | 15.60 | -3.10% | 7.16 | -4.33% | +-+---+-++++ +---+---+-++++ | Workload | File Format | Avg (s) | Delta(Avg) | GeoMean(s) | Delta(GeoMean) | +---+---+-++++ | TPCH_NESTED(_300) | parquet / none / none | 30.93 | -3.02% | 17.46 | -4.71% | +---+---+-++++ Change-Id: I1c44cc9312c062cc7a5a4ac9156ceaa31fb887ff --- M be/src/codegen/gen_ir_descriptions.py M be/src/codegen/impala-ir.cc M be/src/exec/data-sink.cc M be/src/exec/data-sink.h M be/src/exec/exchange-node.cc M be/src/exec/exec-node.cc M be/src/exec/exec-node.h M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/partial-sort-node.cc M be/src/exec/partitioned-aggregation-node.cc M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-builder.h M be/src/exec/partitioned-hash-join-node.cc M be/src/exec/sort-node.cc M be/src/exec/topn-node.cc M be/src/runtime/CMakeLists.txt M be/src/runtime/fragment-instance-state.cc A be/src/runtime/krpc-data-stream-sender-ir.cc M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/krpc-data-stream-sender.h M be/src/runtime/raw-value-ir.cc M be/src/runtime/raw-value.cc M be/src/runtime/runtime-state.h M be/src/util/runtime-profile.h A testdata/workloads/functional-query/queries/QueryTest/datastream-sender-codegen.test M tests/query_test/test_codegen.py 26 files changed, 425 insertions(+), 101 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/10421/8 -- To view, visit http://gerrit.cloudera.org:8080/10421 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1c44cc9312c062cc7a5a4ac9156ceaa31fb887ff Gerrit-Change-Number: 10421 Gerrit-PatchSet: 8 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7016: Implement ALTER DATABASE SET OWNER
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10471 ) Change subject: IMPALA-7016: Implement ALTER DATABASE SET OWNER .. Patch Set 12: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10471 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie3b923021ebce5192d2d64784e7ddb952ba82bc3 Gerrit-Change-Number: 10471 Gerrit-PatchSet: 12 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 14 Jun 2018 19:30:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7016: Implement ALTER DATABASE SET OWNER
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10471 ) Change subject: IMPALA-7016: Implement ALTER DATABASE SET OWNER .. IMPALA-7016: Implement ALTER DATABASE SET OWNER Alter the database owner to either user or role. On database creation, the database owner will be set to the current user, which can be viewed via DESCRIBE DATABASE db command. Having an owner information allows implementing a feature where an owner can be given certain privileges automatically upon a database creation. See IMPALA-7075. The ALTER DATABASE SET OWNER will be a useful command for transferring ownership (a set of owner privileges) from the current owner to another owner. Syntax: ALTER DATABASE db SET OWNER USER user ALTER DATABASE db SET OWNER ROLE role Testing: - Added new front-end tests - Added new end-to-end tests Change-Id: Ie3b923021ebce5192d2d64784e7ddb952ba82bc3 Reviewed-on: http://gerrit.cloudera.org:8080/10471 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M common/thrift/CatalogService.thrift M common/thrift/JniCatalog.thrift M fe/src/main/cup/sql-parser.cup A fe/src/main/java/org/apache/impala/analysis/AlterDbSetOwnerStmt.java A fe/src/main/java/org/apache/impala/analysis/AlterDbStmt.java M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java A fe/src/main/java/org/apache/impala/analysis/Owner.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M tests/metadata/test_ddl.py M tests/metadata/test_ddl_base.py 15 files changed, 361 insertions(+), 7 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/10471 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie3b923021ebce5192d2d64784e7ddb952ba82bc3 Gerrit-Change-Number: 10471 Gerrit-PatchSet: 13 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-6034: Add Cpu and scanned bytes limits per query
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10415 ) Change subject: IMPALA-6034: Add Cpu and scanned bytes limits per query .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2671/ -- To view, visit http://gerrit.cloudera.org:8080/10415 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c6015e21da684bb9f33e236d71309dd4c178a20 Gerrit-Change-Number: 10415 Gerrit-PatchSet: 5 Gerrit-Owner: Mostafa Mokhtar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 14 Jun 2018 19:10:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6034: Add Cpu and scanned bytes limits per query
Hello Tim Armstrong, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10415 to look at the new patch set (#5). Change subject: IMPALA-6034: Add Cpu and scanned bytes limits per query .. IMPALA-6034: Add Cpu and scanned bytes limits per query To protect Impala from runaway queries add per query limits for Cpu and scanned bytes, limits are enforced via query options and applied to the entire query, not per backend like mem_limit. If a query exceeds any of the limits it will get cancelled. Query profile is updated to include query wide and per backend metrics for Cpu and scanned bytes. Testing: Added tests for various permutations for MAX_CPU_TIME_S and MAX_SCAN_BYTES Change-Id: I4c6015e21da684bb9f33e236d71309dd4c178a20 --- M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/service/query-options-test.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift A testdata/workloads/functional-query/queries/QueryTest/query-resource-limits.test A tests/query_test/test_query_resource_limits.py 13 files changed, 374 insertions(+), 22 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/10415/5 -- To view, visit http://gerrit.cloudera.org:8080/10415 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4c6015e21da684bb9f33e236d71309dd4c178a20 Gerrit-Change-Number: 10415 Gerrit-PatchSet: 5 Gerrit-Owner: Mostafa Mokhtar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7046: introduce "global" debug actions
Hello Bikramjeet Vig, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10690 to look at the new patch set (#5). Change subject: IMPALA-7046: introduce "global" debug_actions .. IMPALA-7046: introduce "global" debug_actions The motivation is to add jitter to backend startup in test_failpoints. The race in IMPALA-7033 can be reproduced by adding jitter to the exec rpcs when some backends fail. Let's add jitter to test_failpoints to get better coverage of exec startup races. This builds on top of the debug action extensions added in the async admission control patch by allowing the new "global" debug actions (i.e. actions that can be used in points outside of the ExecNodes). See the code comments for details. For now, we're only using the SLEEP and JITTER commands, but I've included a FAIL command as well since I'll want to use that to write a test for IMPALA-6788 to simulate exec rpc failure. Note that I don't bother resolving the actions ahead of time (like we do for ExecNode actions). It doesn't seem worth it since the resolution only needs to occur after we've matched the label and I don't expect the same label to be hit many times within a single thread. We can always optimize this later if needed. Instead, just continue compiling it out of Release builds. Testing: - Verified that test_failpoints can reproduce the race in IMPALA-7033 by reverting that fix and testing. - Ran the modified tests and grepped the impalad log to see that the sleeps are still occuring. - Manually verify global FAIL command (in a build with another patch). Change-Id: I77663a539be18711a4f12c470ffd7474e3d69388 --- M be/src/runtime/coordinator.cc M be/src/runtime/debug-options.cc M be/src/scheduling/admission-controller.cc M be/src/service/client-request-state.cc M be/src/util/debug-util.cc M be/src/util/debug-util.h M common/thrift/ImpalaService.thrift M tests/custom_cluster/test_admission_controller.py M tests/failure/test_failpoints.py M tests/query_test/test_observability.py 10 files changed, 196 insertions(+), 73 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/10690/5 -- To view, visit http://gerrit.cloudera.org:8080/10690 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I77663a539be18711a4f12c470ffd7474e3d69388 Gerrit-Change-Number: 10690 Gerrit-PatchSet: 5 Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Bikramjeet Vig
[Impala-ASF-CR] IMPALA-7141 (part 2). Extract interfaces for partition pruning without loading
Hello Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10631 to look at the new patch set (#7). Change subject: IMPALA-7141 (part 2). Extract interfaces for partition pruning without loading .. IMPALA-7141 (part 2). Extract interfaces for partition pruning without loading This adds a new PrunablePartition interface which HdfsPartition implements. The interface includes only the partition ID and the partition key/values. For the case of the current catalog implementation, this provides no benefit. However, for LocalCatalog, we want to defer loading partition information until after pruning. With this interface, we can construct PrunablePartition objects using just the partition names, and then load the partitions once pruning is complete. Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa --- M fe/src/main/java/org/apache/impala/analysis/AlterTableSetCachedStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetLocationStmt.java M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/analysis/PartitionSet.java M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java A fe/src/main/java/org/apache/impala/catalog/PrunablePartition.java M fe/src/main/java/org/apache/impala/planner/HdfsPartitionFilter.java M fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java M fe/src/test/java/org/apache/impala/testutil/BlockIdGenerator.java 18 files changed, 175 insertions(+), 52 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/10631/7 -- To view, visit http://gerrit.cloudera.org:8080/10631 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa Gerrit-Change-Number: 10631 Gerrit-PatchSet: 7 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7141 (part 1): clean up handling of default/dummy partition
Hello Gabor Kaszab, Zoltan Borok-Nagy, Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10711 to look at the new patch set (#2). Change subject: IMPALA-7141 (part 1): clean up handling of default/dummy partition .. IMPALA-7141 (part 1): clean up handling of default/dummy partition Currently, HdfsTable inconsistently uses the term "default partition" to refer to two different concepts: 1) For unpartitioned tables, a single partition with ID 1 and no partition keys is created and added to the partition map. 2) All tables have an additional partition added with partition ID -1 which acts as a sort of prototype for partition creation: when new partitions are created during an INSERT operation, the file format and other related options are copied out of this special partition. This partition is inconsistently referred to as either the "default partition" or the "dummy partition". The handling of this second case (the partition with id -1) was somewhat messy: - the partition shows up in the partitionMap_ member, but does not show up in the partitionIds_ member. - almost all of the call sites that iterate through the partitions of an HdfsTable instance ended up skipping over the dummy partition. - several call sites called getPartitions().size() but then had to adjust the result by subtracting one in order to actually count the number of partitions in a table. - similarly, test assertions had to assert that tables with 24 partitions had an expected partition map size of 25. In order to address the above, this patch makes the following changes: - getPartitions() and getPartitionMap() no longer include the dummy partition. This removes a bunch of special case checks to skip over the dummy partition or to adjust partition counts based on it. - to clarify the purpose of this partition, references to it are renamed to "prototype partition" instead of "default partition". - when converting the HdfsTable to/from Thrift, the prototype partition is included in its own field in the struct, instead of being stuffed into the same map with the true partitions of the table. This reflects the fact that this partition is special (eg missing fields like 'location' which otherwise are required for real partitions) This change should should be entirely internal with no functional differences. As such, the only testing changes are some fixes for assertions on the Thrift serialized structures and other internals. Change-Id: I15e91b50eb7c2a5e0bac8c33d603d6cd8cbaca2e --- M be/src/exec/hdfs-table-sink.cc M be/src/exec/hdfs-table-sink.h M be/src/runtime/descriptors.cc M be/src/runtime/descriptors.h M common/thrift/CatalogObjects.thrift M common/thrift/ImpalaInternalService.thrift M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/catalog/FeFsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/testutil/BlockIdGenerator.java 18 files changed, 162 insertions(+), 204 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/10711/2 -- To view, visit http://gerrit.cloudera.org:8080/10711 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I15e91b50eb7c2a5e0bac8c33d603d6cd8cbaca2e Gerrit-Change-Number: 10711 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-7141 (part 1): clean up handling of default/dummy partition
Todd Lipcon has posted comments on this change. (
http://gerrit.cloudera.org:8080/10711 )
Change subject: IMPALA-7141 (part 1): clean up handling of default/dummy
partition
..
Patch Set 1:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/10711/1/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
File fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java:
http://gerrit.cloudera.org:8080/#/c/10711/1/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java@1234
PS1, Line 1234: {
: continue;
: }
> nit: single line
Done
http://gerrit.cloudera.org:8080/#/c/10711/1/fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java
File
fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java:
http://gerrit.cloudera.org:8080/#/c/10711/1/fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java@a115
PS1, Line 115:
> In the old code we had this assert.
Done
http://gerrit.cloudera.org:8080/#/c/10711/1/fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java@215
PS1, Line 215: HdfsPartition part =
(HdfsPartition)Iterables.getFirst(hdfsTable.getPartitions(), null);
> nit: too long line
Done
http://gerrit.cloudera.org:8080/#/c/10711/1/fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java@216
PS1, Line 216: ;;
> nit: double ;
Done
--
To view, visit http://gerrit.cloudera.org:8080/10711
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15e91b50eb7c2a5e0bac8c33d603d6cd8cbaca2e
Gerrit-Change-Number: 10711
Gerrit-PatchSet: 1
Gerrit-Owner: Todd Lipcon
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Thu, 14 Jun 2018 18:28:27 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7046: introduce "global" debug actions
Dan Hecht has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10690 Change subject: IMPALA-7046: introduce "global" debug_actions .. IMPALA-7046: introduce "global" debug_actions The motivation is to add jitter to backend startup in test_failpoints. The race in IMPALA-7033 can be reproduced by adding jitter to the exec rpcs when some backends fail. Let's add jitter to test_failpoints to get better coverage of exec startup races. This builds on top of the debug action extensions added in the async admission control patch by allowing the new "global" debug actions (i.e. actions that can be used in points outside of the ExecNodes). See the code comments for details. For now, we're only using the SLEEP and JITTER commands, but I've included a FAIL command as well since I'll want to use that to write a test for IMPALA-6788 to simulate exec rpc failure. Testing: - Verified that test_failpoints can reproduce the race in IMPALA-7033 by reverting that fix and testing. - Ran the modified tests and grepped the impalad log to see that the sleeps are still occuring. - Manually verify global FAIL command (in a build with another patch). Change-Id: I77663a539be18711a4f12c470ffd7474e3d69388 --- M be/src/runtime/coordinator.cc M be/src/runtime/debug-options.cc M be/src/scheduling/admission-controller.cc M be/src/service/client-request-state.cc M be/src/util/debug-util.cc M be/src/util/debug-util.h M common/thrift/ImpalaService.thrift M tests/custom_cluster/test_admission_controller.py M tests/failure/test_failpoints.py M tests/query_test/test_observability.py 10 files changed, 196 insertions(+), 73 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/10690/4 -- To view, visit http://gerrit.cloudera.org:8080/10690 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I77663a539be18711a4f12c470ffd7474e3d69388 Gerrit-Change-Number: 10690 Gerrit-PatchSet: 4 Gerrit-Owner: Dan Hecht
[Impala-ASF-CR] IMPALA-7141 (part 1): clean up handling of default/dummy partition
Vuk Ercegovac has posted comments on this change. (
http://gerrit.cloudera.org:8080/10711 )
Change subject: IMPALA-7141 (part 1): clean up handling of default/dummy
partition
..
Patch Set 1: Code-Review+2
(1 comment)
nice clean-up! small clean-ups remaining.. lgtm.
http://gerrit.cloudera.org:8080/#/c/10711/1/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
File fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java:
http://gerrit.cloudera.org:8080/#/c/10711/1/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java@1234
PS1, Line 1234: {
: continue;
: }
nit: single line
--
To view, visit http://gerrit.cloudera.org:8080/10711
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15e91b50eb7c2a5e0bac8c33d603d6cd8cbaca2e
Gerrit-Change-Number: 10711
Gerrit-PatchSet: 1
Gerrit-Owner: Todd Lipcon
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Thu, 14 Jun 2018 18:08:20 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6812: Fix flaky Kudu scan tests
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/10503 ) Change subject: IMPALA-6812: Fix flaky Kudu scan tests .. Patch Set 2: In order to reduce flakiness while we wait for a fix for the READ_YOUR_WRITES issue, I have put together a version of this patch that adds a query option for kudu_read_mode and then sets this option to READ_AT_SNAPSHOT for all of the flaky tests. -- To view, visit http://gerrit.cloudera.org:8080/10503 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70df84f2cbc663107f2ad029565d3c15bdfbd47c Gerrit-Change-Number: 10503 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 14 Jun 2018 18:05:56 + Gerrit-HasComments: No
