[Impala-ASF-CR] IMPALA-9071: Handle translated external HDFS table in CATS
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14527 ) Change subject: IMPALA-9071: Handle translated external HDFS table in CATS .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4866/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I460a57dc877ef68ad7dd0864a33b1599b1e9a8d9 Gerrit-Change-Number: 14527 Gerrit-PatchSet: 6 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 24 Oct 2019 05:35:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9071: Handle translated external HDFS table in CATS
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14527 to look at the new patch set (#6). Change subject: IMPALA-9071: Handle translated external HDFS table in CATS .. IMPALA-9071: Handle translated external HDFS table in CATS After upgrading Hive-3 to a version containing HIVE-22158, it's not allowed for managed tables to be non transactional. Creating non ACID tables will result in creating an external table with table property 'external.table.purge' set to true. In Hive-3, the default location of external HDFS tables will locate in 'metastore.warehouse.external.dir' if it's set. This property is added by HIVE-19837 in Hive 2.7, but hasn't been added to Hive in cdh6 yet. In CTAS statement, we create a temporary HMS Table for the analysis on the Insert part. The table path is created assuming it's a managed table, and the Insert part will use this path for insertion. However, in Hive-3, the created table is translated to an external table. It's not the same as we passed to the HMS API. The created table is located in 'metastore.warehouse.external.dir', while the table path we assumed is in 'metastore.warehouse.dir'. This introduces bugs when these two properties are different. CTAS statement will create table in one place and insert data in another place. This patch adds a new method in MetastoreShim to wrap the difference for getting the default table path for non transactional tables between Hive-2 and Hive-3. Changes in the infra: - To support customizing hive configuration, add a env var, CUSTOM_CLASSPATH in bin/set-classpath.sh to be put in front of existing CLASSPATH. The customized hive-site.xml should be put inside CUSTOM_CLASSPATH. - Change hive-site.xml.py to generate a hive-site.xml with non default 'metastore.warehouse.external.dir' - Add an option, --env_vars, in bin/start-impala-cluster.py to pass down CUSTOM_CLASSPATH. Tests: - Add a custom cluster test to start Hive with metastore.warehouse.external.dir being set to non default value. Run it locally using CDP components with HIVE-22158. - Run CORE tests using CDH components Change-Id: I460a57dc877ef68ad7dd0864a33b1599b1e9a8d9 --- M bin/create-test-configuration.sh M bin/set-classpath.sh M bin/start-impala-cluster.py M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/test/resources/hive-site.xml.py M tests/common/custom_cluster_test_suite.py A tests/custom_cluster/test_custom_hive_configs.py 9 files changed, 156 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/14527/6 -- To view, visit http://gerrit.cloudera.org:8080/14527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I460a57dc877ef68ad7dd0864a33b1599b1e9a8d9 Gerrit-Change-Number: 14527 Gerrit-PatchSet: 6 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-8999: make union scheduling work with mt dop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14384 ) Change subject: IMPALA-8999: make union scheduling work with mt_dop .. Patch Set 14: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14384 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d2e9c86b530da3053e49d42b837dca0b1348ff2 Gerrit-Change-Number: 14384 Gerrit-PatchSet: 14 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 24 Oct 2019 04:41:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8999: make union scheduling work with mt dop
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14384 ) Change subject: IMPALA-8999: make union scheduling work with mt_dop .. IMPALA-8999: make union scheduling work with mt_dop This change unifies mt_dop scheduling between the union and scan cases. Testing: Manually checked that fragments with unions get parallelised to the correct degree, both as a result of scans within the fragment and input fragments. Extend TestMtDopAdmissionSlots (renamed to TestMtDopScheduling) to confirm that queries that were not parallelised before are now parallelised. These tests verify the number of instances of each operator using the ExecSummary embedded in the profile. Change-Id: I0d2e9c86b530da3053e49d42b837dca0b1348ff2 Reviewed-on: http://gerrit.cloudera.org:8080/14384 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/scheduling/query-schedule.cc M be/src/scheduling/scheduler-test.cc M be/src/scheduling/scheduler.cc M be/src/scheduling/scheduler.h M fe/src/main/java/org/apache/impala/planner/UnionNode.java D testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-admission-slots.test A testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-scheduling.test M tests/query_test/test_mt_dop.py 8 files changed, 382 insertions(+), 219 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14384 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I0d2e9c86b530da3053e49d42b837dca0b1348ff2 Gerrit-Change-Number: 14384 Gerrit-PatchSet: 15 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9071: Handle translated external HDFS table in CATS
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14527 ) Change subject: IMPALA-9071: Handle translated external HDFS table in CATS .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4865/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I460a57dc877ef68ad7dd0864a33b1599b1e9a8d9 Gerrit-Change-Number: 14527 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 24 Oct 2019 01:40:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8995: Fix synchronization in dequeue thread
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14539 ) Change subject: IMPALA-8995: Fix synchronization in dequeue thread .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4864/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14539 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I91080ce54e59cc7e6361f7c50d6b2156a8a180c8 Gerrit-Change-Number: 14539 Gerrit-PatchSet: 1 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 24 Oct 2019 01:39:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7984: Port runtime filter from Thrift RPC to KRPC
Michael Ho has posted comments on this change. (
http://gerrit.cloudera.org:8080/13882 )
Change subject: IMPALA-7984: Port runtime filter from Thrift RPC to KRPC
..
Patch Set 25:
(26 comments)
http://gerrit.cloudera.org:8080/#/c/13882/25/be/src/runtime/coordinator-backend-state.cc
File be/src/runtime/coordinator-backend-state.cc:
http://gerrit.cloudera.org:8080/#/c/13882/25/be/src/runtime/coordinator-backend-state.cc@556
PS25, Line 556: LOG(ERROR) << "PublishFilter() rpc failed: " <<
rpc_status.ToString();
return here as there is no point in continuing to print the result status if
RPC failed.
http://gerrit.cloudera.org:8080/#/c/13882/24/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:
http://gerrit.cloudera.org:8080/#/c/13882/24/be/src/runtime/coordinator.cc@1134
PS24, Line 1134: r is neither an always true filter nor an
: // always false filter, then it must be the case that a
non-empty sidecar slice
> I actually found out that if we are using "std::move(sidecar_slice.ToString
Sorry for the wrong advice. I believe the RVO should take care of the
copy-elision so no need for the std::move. In other words, it should be
sufficient to just call sidecar_slice.ToString();
http://gerrit.cloudera.org:8080/#/c/13882/25/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:
http://gerrit.cloudera.org:8080/#/c/13882/25/be/src/runtime/coordinator.cc@1010
PS25, Line 1010: // >>> IMPALA-7984: Port runtime filter from Thrift RPC to
KRPC
To be removed ?!
http://gerrit.cloudera.org:8080/#/c/13882/25/be/src/runtime/coordinator.cc@1022
PS25, Line 1022: std::
nit: no need for std::
Also, may make sense to move this declaration to somewhere closer to its use
(e.g. line 1068 below).
Same for rpc_params and target_fragment_idxs above.
http://gerrit.cloudera.org:8080/#/c/13882/25/be/src/runtime/coordinator.cc@1072
PS25, Line 1072: BloomFilterPB& aggregated_filter = state->bloom_filter();
:
aggregated_filter.Swap(rpc_params.mutable_bloom_filter());
This swapping tricks seems unnecessary now that the bloom_filter_directory is a
separate structure.
http://gerrit.cloudera.org:8080/#/c/13882/25/be/src/runtime/coordinator.cc@1074
PS25, Line 1074: bloom_filter_directory
Is there a reason to assign state->bloom_filter_directory() to this local
variable ? Why cannot we use state->bloom_filter_directory() directly below ?
http://gerrit.cloudera.org:8080/#/c/13882/25/be/src/runtime/coordinator.cc@1098
PS25, Line 1098: reinterpret_cast(&(bloom_filter_directory[0])
state->bloom_filter_directory().data()
http://gerrit.cloudera.org:8080/#/c/13882/25/be/src/runtime/coordinator.cc@1153
PS25, Line 1153: bloom_filter_ = BloomFilterPB(params.bloom_filter());
bloom_filter_ = params.bloom_filter();
The above should be sufficient.
http://gerrit.cloudera.org:8080/#/c/13882/25/be/src/runtime/coordinator.cc@1155
PS25, Line 1155: std::string(
: reinterpret_cast(sidecar_slice.data()), sidecar_slice.size())
Can keep the original suggestion of sidecar_slice.ToString(). C++ ROV should
take care of avoiding the copy so the TODO can be removed. There is no easy way
to avoid copying the sidecar from the network buffer so we have to copy at
least once.
http://gerrit.cloudera.org:8080/#/c/13882/25/be/src/runtime/runtime-filter-bank.cc
File be/src/runtime/runtime-filter-bank.cc:
http://gerrit.cloudera.org:8080/#/c/13882/25/be/src/runtime/runtime-filter-bank.cc@190
PS25, Line 190: std::
nit: no need for std::
http://gerrit.cloudera.org:8080/#/c/13882/25/be/src/runtime/runtime-filter-bank.cc@204
PS25, Line 204: std::
nit: no need for std::
http://gerrit.cloudera.org:8080/#/c/13882/25/be/src/runtime/runtime-filter-bank.cc@205
PS25, Line 205: ++num_inflight_rpcs_;
DCHECK_GE(num_inflight_rpcs_, 0); before increment.
http://gerrit.cloudera.org:8080/#/c/13882/25/be/src/runtime/runtime-filter-bank.cc@323
PS25, Line 323: std::
nit: no need for std::
http://gerrit.cloudera.org:8080/#/c/13882/25/be/src/runtime/timestamp-value.h
File be/src/runtime/timestamp-value.h:
http://gerrit.cloudera.org:8080/#/c/13882/25/be/src/runtime/timestamp-value.h@170
PS25, Line 170: // Store the binary representation of this TimestampValue in
'tvalue'.
: void ToTColumnValue(TColumnValue* tvalue) const {
: const uint8_t* data = reinterpret_cast(this);
: tvalue->timestamp_val.assign(data, data + Size());
: tvalue->__isset.timestamp_val = true;
: }
Is this not used anymore ?
http://gerrit.cloudera.org:8080/#/c/13882/25/be/src/runtime/timestamp-value.h@183
PS25, Line 183: // Returns a new TimestampValue created from the value in
'tvalue'.
: static TimestampValue FromTColumnValue(const TColumnValue&
tvalue) {
: TimestampValue value;
[Impala-ASF-CR] IMPALA-8995: Fix synchronization in dequeue thread
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/14539 )
Change subject: IMPALA-8995: Fix synchronization in dequeue thread
..
Patch Set 1:
(1 comment)
Fix makes sense, just want to be sure that we're not leaving things in a state
where more subtle bugs will go un-noticed.
http://gerrit.cloudera.org:8080/#/c/14539/1/be/src/scheduling/admission-controller.cc
File be/src/scheduling/admission-controller.cc:
http://gerrit.cloudera.org:8080/#/c/14539/1/be/src/scheduling/admission-controller.cc@1375
PS1, Line 1375: dequeue_cv_.Wait(lock);
I feel like this usage pattern of condition variables is really asking for
trouble - the pattern of
while (!condition) {
cv_.wait(lock);
}
Is a lot more robust. Can we switch to using this pattern so that it's more
explicit, e.g. have a pending_dequeue_ variable that is explicitly set when
signalling the variable to make sure that the wakeup doesn't get lost.
--
To view, visit http://gerrit.cloudera.org:8080/14539
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I91080ce54e59cc7e6361f7c50d6b2156a8a180c8
Gerrit-Change-Number: 14539
Gerrit-PatchSet: 1
Gerrit-Owner: Bikramjeet Vig
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Thu, 24 Oct 2019 01:10:19 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9071: Handle translated external HDFS table in CATS
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14527 to look at the new patch set (#5). Change subject: IMPALA-9071: Handle translated external HDFS table in CATS .. IMPALA-9071: Handle translated external HDFS table in CATS After upgrading Hive-3 to a version containing HIVE-22158, it's not allowed for managed tables to be non transactional. Creating non ACID tables will result in creating an external table with table property 'external.table.purge' set to true. In Hive-3, the default location of external HDFS tables will locate in 'metastore.warehouse.external.dir' if it's set. This property is added by HIVE-19837 in Hive 2.7, but hasn't been added to Hive in cdh6 yet. In CTAS statement, we create a temporary HMS Table for the analysis on the Insert part. The table path is created assuming it's a managed table, and the Insert part will use this path for insertion. However, in Hive-3, the created table is translated to an external table. It's not the same as we passed to the HMS API. The created table is located in 'metastore.warehouse.external.dir', while the table path we assumed is in 'metastore.warehouse.dir'. This introduces bugs when these two properties are different. CTAS statement will create table in one place and insert data in another place. This patch adds a new method in MetastoreShim to wrap the difference for getting the default table path for non transactional tables between Hive-2 and Hive-3. This patch also bumps the CDP version to contain HIVE-22158. Note that Ranger version also bumps from 1.2 to 2.0 due to this CDP upgrade, which causes some test failures. They are disabled for follow-up fixs in IMPALA-9047. Changes in the infra: - To support customizing hive configuration, add a env var, CUSTOM_CLASSPATH in bin/set-classpath.sh to be put in front of existing CLASSPATH. The customized hive-site.xml should be put inside CUSTOM_CLASSPATH. - Change hive-site.xml.py to generate a hive-site.xml with non default 'metastore.warehouse.external.dir' - Add an option, --env_vars, in bin/start-impala-cluster.py to pass down CUSTOM_CLASSPATH. Tests: - Add a custom cluster test to start Hive with metastore.warehouse.external.dir being set to non default value. Run it locally using CDP components with HIVE-22158. - Run CORE tests using CDH components Change-Id: I460a57dc877ef68ad7dd0864a33b1599b1e9a8d9 --- M bin/create-test-configuration.sh M bin/impala-config.sh M bin/set-classpath.sh M bin/start-impala-cluster.py M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java M fe/src/test/resources/hive-site.xml.py M testdata/pom.xml M tests/authorization/test_authorized_proxy.py M tests/authorization/test_ranger.py M tests/common/custom_cluster_test_suite.py A tests/custom_cluster/test_custom_hive_configs.py 15 files changed, 255 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/14527/5 -- To view, visit http://gerrit.cloudera.org:8080/14527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I460a57dc877ef68ad7dd0864a33b1599b1e9a8d9 Gerrit-Change-Number: 14527 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-8995: Fix synchronization in dequeue thread
Bikramjeet Vig has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14539 Change subject: IMPALA-8995: Fix synchronization in dequeue thread .. IMPALA-8995: Fix synchronization in dequeue thread The admission controller's dequeue thread currently wakes up either when queries release their admission resources or when a statestore update is received. The dequeue loop releases the admission lock at the end of the loop, then acquires it back and calls wait on it. In this small window, a query can complete, update the admission stats by acquiring the admission lock, and then call send a notify to wake the dequeue thread. But since the dequeue thread has not called wait yet, it can miss this notify. Moreover if the statestore is down there is no way of waking it up. This will cause the queued queries to eventually timeout. This patch attempts to fix this by removing that window. Testing: Was able to trigger this manually by adding a sleep right before the dequeue loop acquires the admission lock. Change-Id: I91080ce54e59cc7e6361f7c50d6b2156a8a180c8 --- M be/src/scheduling/admission-controller.cc 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/14539/1 -- To view, visit http://gerrit.cloudera.org:8080/14539 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I91080ce54e59cc7e6361f7c50d6b2156a8a180c8 Gerrit-Change-Number: 14539 Gerrit-PatchSet: 1 Gerrit-Owner: Bikramjeet Vig
[Impala-ASF-CR] IMPALA-8999: make union scheduling work with mt dop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14384 ) Change subject: IMPALA-8999: make union scheduling work with mt_dop .. Patch Set 14: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5129/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14384 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d2e9c86b530da3053e49d42b837dca0b1348ff2 Gerrit-Change-Number: 14384 Gerrit-PatchSet: 14 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 24 Oct 2019 00:23:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8999: make union scheduling work with mt dop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14384 ) Change subject: IMPALA-8999: make union scheduling work with mt_dop .. Patch Set 14: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14384 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d2e9c86b530da3053e49d42b837dca0b1348ff2 Gerrit-Change-Number: 14384 Gerrit-PatchSet: 14 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 24 Oct 2019 00:23:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8999: make union scheduling work with mt dop
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/14384 ) Change subject: IMPALA-8999: make union scheduling work with mt_dop .. Patch Set 13: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14384 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d2e9c86b530da3053e49d42b837dca0b1348ff2 Gerrit-Change-Number: 14384 Gerrit-PatchSet: 13 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 23 Oct 2019 23:37:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8999: make union scheduling work with mt dop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14384 ) Change subject: IMPALA-8999: make union scheduling work with mt_dop .. Patch Set 13: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4863/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14384 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d2e9c86b530da3053e49d42b837dca0b1348ff2 Gerrit-Change-Number: 14384 Gerrit-PatchSet: 13 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 23 Oct 2019 22:46:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8999: make union scheduling work with mt dop
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14384 ) Change subject: IMPALA-8999: make union scheduling work with mt_dop .. Patch Set 12: (3 comments) http://gerrit.cloudera.org:8080/#/c/14384/12/be/src/scheduling/scheduler.cc File be/src/scheduling/scheduler.cc: http://gerrit.cloudera.org:8080/#/c/14384/12/be/src/scheduling/scheduler.cc@338 PS12, Line 338: for the scan range > nit: remove? Done http://gerrit.cloudera.org:8080/#/c/14384/12/be/src/scheduling/scheduler.cc@344 PS12, Line 344: Note that this takes into account all of the input fragments, : // not just the leftmost because we expect unions to be symmetrical for purposes of : // planning, unlike joins. > nit: how about: "Note that step 1 is modified to run on fragments with unio Done http://gerrit.cloudera.org:8080/#/c/14384/12/be/src/scheduling/scheduler.cc@392 PS12, Line 392: vector scan_hosts; > nit: maybe add a Dcheck(scan_node_ids.size() == 1 || has_union) so that we Done -- To view, visit http://gerrit.cloudera.org:8080/14384 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d2e9c86b530da3053e49d42b837dca0b1348ff2 Gerrit-Change-Number: 14384 Gerrit-PatchSet: 12 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 23 Oct 2019 22:01:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8999: make union scheduling work with mt dop
Hello Andrew Sherman, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14384 to look at the new patch set (#13). Change subject: IMPALA-8999: make union scheduling work with mt_dop .. IMPALA-8999: make union scheduling work with mt_dop This change unifies mt_dop scheduling between the union and scan cases. Testing: Manually checked that fragments with unions get parallelised to the correct degree, both as a result of scans within the fragment and input fragments. Extend TestMtDopAdmissionSlots (renamed to TestMtDopScheduling) to confirm that queries that were not parallelised before are now parallelised. These tests verify the number of instances of each operator using the ExecSummary embedded in the profile. Change-Id: I0d2e9c86b530da3053e49d42b837dca0b1348ff2 --- M be/src/scheduling/query-schedule.cc M be/src/scheduling/scheduler-test.cc M be/src/scheduling/scheduler.cc M be/src/scheduling/scheduler.h M fe/src/main/java/org/apache/impala/planner/UnionNode.java D testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-admission-slots.test A testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-scheduling.test M tests/query_test/test_mt_dop.py 8 files changed, 382 insertions(+), 219 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/84/14384/13 -- To view, visit http://gerrit.cloudera.org:8080/14384 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0d2e9c86b530da3053e49d42b837dca0b1348ff2 Gerrit-Change-Number: 14384 Gerrit-PatchSet: 13 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9047: Bump CDP BUILD NUMBER to 1507246
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14529 ) Change subject: IMPALA-9047: Bump CDP_BUILD_NUMBER to 1507246 .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4862/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14529 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifbc42a73fe98c437edc20a8f57aba51e3096a09d Gerrit-Change-Number: 14529 Gerrit-PatchSet: 2 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 23 Oct 2019 18:02:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9047: Bump CDP BUILD NUMBER to 1507246
Fang-Yu Rao has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/14529 ) Change subject: IMPALA-9047: Bump CDP_BUILD_NUMBER to 1507246 .. IMPALA-9047: Bump CDP_BUILD_NUMBER to 1507246 This patch bumps CDP_BUILD_NUMBER to 1507246. Some test cases would fail due to this change. For example, some Ranger related E2E tests and FE tests would fail when we are using a newer version of Ranger (e.g., ranger-2.0.0.7.0.2.0-108 instead of ranger-1.2.0.7.1.0.0-33) due to the changes to the default Ranger policies described at https://issues.apache.org/jira/browse/RANGER-2536. To address this issue, this patch temporarily disables those affected Ranger tests. Specifically, the affected tests in the following test files are disabled for now. 1. test_authorized_proxy.py 2. test_ranger.py 3. AuthorizationStmtTest.java 4. RangerAuditLogTest.java Testing: - This patch passes the affected Ranger tests listed above on a local machine. Change-Id: Ifbc42a73fe98c437edc20a8f57aba51e3096a09d --- M bin/impala-config.sh M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java M tests/authorization/test_authorized_proxy.py M tests/authorization/test_ranger.py 5 files changed, 101 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/14529/2 -- To view, visit http://gerrit.cloudera.org:8080/14529 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifbc42a73fe98c437edc20a8f57aba51e3096a09d Gerrit-Change-Number: 14529 Gerrit-PatchSet: 2 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-7504/KUDU-2979 ParseKerberosPrincipal() should use krb5 parse name() instead
Andrew Sherman has posted comments on this change. (
http://gerrit.cloudera.org:8080/14433 )
Change subject: IMPALA-7504/KUDU-2979 ParseKerberosPrincipal() should use
krb5_parse_name() instead
..
Patch Set 6:
(3 comments)
Thanks for this update
http://gerrit.cloudera.org:8080/#/c/14433/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/14433/5//COMMIT_MSG@21
PS5, Line 21: format principal, new error code is 2 instead of original 112
> What kind of end-to-end test? The one included in impala-private-parameteri
yes, maybe there's a better name but I think this name is generally understood
http://gerrit.cloudera.org:8080/#/c/14433/6//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/14433/6//COMMIT_MSG@19
PS6, Line 19: Add two authentication-test
Add two tests to authentication-test
http://gerrit.cloudera.org:8080/#/c/14433/6/be/src/rpc/authentication-test.cc
File be/src/rpc/authentication-test.cc:
http://gerrit.cloudera.org:8080/#/c/14433/6/be/src/rpc/authentication-test.cc@200
PS6, Line 200: EXPECT_ERROR(sa.InitKerberos(" ", "/etc/hosts"), 2);
This says we will get an error, but do we know it is the right error?
We should check the return code or message or something more specific
--
To view, visit http://gerrit.cloudera.org:8080/14433
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0e64ebdc10f102dbdc5b87f6fe3f2a0310b1be24
Gerrit-Change-Number: 14433
Gerrit-PatchSet: 6
Gerrit-Owner: Xiaomeng Zhang
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Xiaomeng Zhang
Gerrit-Comment-Date: Wed, 23 Oct 2019 17:03:19 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9071: Handle translated external HDFS table in CATS
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/14527 ) Change subject: IMPALA-9071: Handle translated external HDFS table in CATS .. Patch Set 4: This is still WIP. Still have errors in data-loading when USE_CDP_HIVE=true. -- To view, visit http://gerrit.cloudera.org:8080/14527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I460a57dc877ef68ad7dd0864a33b1599b1e9a8d9 Gerrit-Change-Number: 14527 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 23 Oct 2019 15:01:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9071: Handle translated external HDFS table in CATS
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14527 ) Change subject: IMPALA-9071: Handle translated external HDFS table in CATS .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4861/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I460a57dc877ef68ad7dd0864a33b1599b1e9a8d9 Gerrit-Change-Number: 14527 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 23 Oct 2019 13:05:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14398 ) Change subject: IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4860/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14398 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iacc53801a660c033869cb4747910c98a80e08297 Gerrit-Change-Number: 14398 Gerrit-PatchSet: 7 Gerrit-Owner: wangsheng Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Wed, 23 Oct 2019 12:13:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8974: Fixed a bug when create kudu managed table without HMS config
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/14398 ) Change subject: IMPALA-8974: Fixed a bug when create kudu managed table without HMS config .. Patch Set 8: (4 comments) > (5 comments) > > Thank you for adding the test. I have some concerns about how it > changes config files around, so I proposed an alternative solution. Thanks for your review, Tim. I've already solved the problems you mentioned above. If there is a chance, I will try to write custom cluster test in Python instead of Java. http://gerrit.cloudera.org:8080/#/c/14398/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/14398/5//COMMIT_MSG@7 PS5, Line 7: managed > spelling: managed (also a few cases below). Done http://gerrit.cloudera.org:8080/#/c/14398/5/fe/src/test/java/org/apache/impala/customcluster/CreateKuduTableWithoutHMSTest.java File fe/src/test/java/org/apache/impala/customcluster/CreateKuduTableWithoutHMSTest.java: http://gerrit.cloudera.org:8080/#/c/14398/5/fe/src/test/java/org/apache/impala/customcluster/CreateKuduTableWithoutHMSTest.java@35 PS5, Line 35: managed > spelling: managed Done http://gerrit.cloudera.org:8080/#/c/14398/5/fe/src/test/java/org/apache/impala/customcluster/CreateKuduTableWithoutHMSTest.java@66 PS5, Line 66: sultSet > spelling: managed Done http://gerrit.cloudera.org:8080/#/c/14398/5/fe/src/test/java/org/apache/impala/customcluster/CreateKuduTableWithoutHMSTest.java@84 PS5, Line 84: } > I don't really like this test modifying global test configs, it really risk Done -- To view, visit http://gerrit.cloudera.org:8080/14398 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iacc53801a660c033869cb4747910c98a80e08297 Gerrit-Change-Number: 14398 Gerrit-PatchSet: 8 Gerrit-Owner: wangsheng Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Wed, 23 Oct 2019 12:21:04 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9071: Handle translated external HDFS table in CATS
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14527 to look at the new patch set (#4). Change subject: IMPALA-9071: Handle translated external HDFS table in CATS .. IMPALA-9071: Handle translated external HDFS table in CATS After upgrading Hive-3 to a version containing HIVE-22158, it's not allowed for managed tables to be non transactional. Creating non ACID tables will result in creating an external table with table property 'external.table.purge' set to true. In Hive-3, the default location of external HDFS tables will locate in 'metastore.warehouse.external.dir' if it's set. This property is added by HIVE-19837 in Hive 2.7, but hasn't been added to Hive in cdh6 yet. In CTAS statement, we create a temporary HMS Table for the analysis on the Insert part. The table path is created assuming it's a managed table, and the Insert part will use this path for insertion. However, in Hive-3, the created table is translated to an external table. It's not the same as we passed to the HMS API. The created table is located in 'metastore.warehouse.external.dir', while the table path we assumed is in 'metastore.warehouse.dir'. This introduces bugs when these two properties are different. CTAS statement will create table in one place and insert data in another place. This patch adds a new method in MetastoreShim to wrap the difference for getting the default table path for non transactional tables between Hive-2 and Hive-3. This patch also bumps the CDP version to contain HIVE-22158. Note that Ranger version also bumps from 1.2 to 2.0 due to this CDP upgrade, which causes some test failures. They are disabled for follow-up fixs in IMPALA-9047. Changes in the infra: - To support customizing hive configuration, add a env var, CUSTOM_CLASSPATH in bin/set-classpath.sh to be put in front of existing CLASSPATH. The customized hive-site.xml should be put inside CUSTOM_CLASSPATH. - Change hive-site.xml.py to generate a hive-site.xml with non default 'metastore.warehouse.external.dir' - Add an option, --env_vars, in bin/start-impala-cluster.py to pass down CUSTOM_CLASSPATH. Tests: - Add a custom cluster test to start Hive with metastore.warehouse.external.dir being set to non default value. Run it locally using CDP components with HIVE-22158. - Run CORE tests using CDH components Change-Id: I460a57dc877ef68ad7dd0864a33b1599b1e9a8d9 --- M bin/create-test-configuration.sh M bin/impala-config.sh M bin/set-classpath.sh M bin/start-impala-cluster.py M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java M fe/src/test/resources/hive-site.xml.py M testdata/pom.xml M tests/authorization/test_authorized_proxy.py M tests/authorization/test_ranger.py M tests/common/custom_cluster_test_suite.py A tests/custom_cluster/test_custom_hive_configs.py 15 files changed, 253 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/14527/4 -- To view, visit http://gerrit.cloudera.org:8080/14527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I460a57dc877ef68ad7dd0864a33b1599b1e9a8d9 Gerrit-Change-Number: 14527 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8974: Fixed a bug when create kudu managed table without HMS config
Hello Quanlong Huang, Hao Hao, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14398 to look at the new patch set (#8). Change subject: IMPALA-8974: Fixed a bug when create kudu managed table without HMS config .. IMPALA-8974: Fixed a bug when create kudu managed table without HMS config When catalogd connected to mysql/postgresql directly instead of use HMS, a kudu managerd table created DDL would failed due to Preconditions checked failed. The patch fixed the bug by using the if condition instead of the Preconditions functions which lead to exception when lack of HMS config. Tests: * Add test for create/drop kudu managerd table without HMS config * Ran all front-end tests Change-Id: Iacc53801a660c033869cb4747910c98a80e08297 --- M bin/start-daemon.sh M bin/start-impala-cluster.py M fe/src/main/java/org/apache/impala/catalog/KuduTable.java A fe/src/test/java/org/apache/impala/customcluster/CreateKuduTableWithoutHMSTest.java A fe/src/test/resources/hive-site-without-hms.xml.py 5 files changed, 177 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/14398/8 -- To view, visit http://gerrit.cloudera.org:8080/14398 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iacc53801a660c033869cb4747910c98a80e08297 Gerrit-Change-Number: 14398 Gerrit-PatchSet: 8 Gerrit-Owner: wangsheng Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: wangsheng
[Impala-ASF-CR] IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14398 ) Change subject: IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config .. Patch Set 6: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/4859/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/14398 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iacc53801a660c033869cb4747910c98a80e08297 Gerrit-Change-Number: 14398 Gerrit-PatchSet: 6 Gerrit-Owner: wangsheng Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Wed, 23 Oct 2019 12:13:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config
Hello Quanlong Huang, Hao Hao, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14398 to look at the new patch set (#7). Change subject: IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config .. IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config When catalogd connected to mysql/postgresql directly instead of use HMS, a kudu managerd table created DDL would failed due to Preconditions checked failed. The patch fixed the bug by using the if condition instead of the Preconditions functions which lead to exception when lack of HMS config. Tests: * Add test for create/drop kudu managerd table without HMS config * Ran all front-end tests Change-Id: Iacc53801a660c033869cb4747910c98a80e08297 --- M bin/start-daemon.sh M bin/start-impala-cluster.py M fe/src/main/java/org/apache/impala/catalog/KuduTable.java A fe/src/test/java/org/apache/impala/customcluster/CreateKuduTableWithoutHMSTest.java A fe/src/test/resources/hive-site-without-hms.xml.py 5 files changed, 177 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/14398/7 -- To view, visit http://gerrit.cloudera.org:8080/14398 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iacc53801a660c033869cb4747910c98a80e08297 Gerrit-Change-Number: 14398 Gerrit-PatchSet: 7 Gerrit-Owner: wangsheng Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: wangsheng
[Impala-ASF-CR] IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14398 ) Change subject: IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config .. Patch Set 6: (2 comments) http://gerrit.cloudera.org:8080/#/c/14398/6/bin/start-impala-cluster.py File bin/start-impala-cluster.py: http://gerrit.cloudera.org:8080/#/c/14398/6/bin/start-impala-cluster.py@136 PS6, Line 136: , flake8: E231 missing whitespace after ',' http://gerrit.cloudera.org:8080/#/c/14398/6/bin/start-impala-cluster.py@136 PS6, Line 136: , flake8: E999 SyntaxError: invalid syntax -- To view, visit http://gerrit.cloudera.org:8080/14398 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iacc53801a660c033869cb4747910c98a80e08297 Gerrit-Change-Number: 14398 Gerrit-PatchSet: 6 Gerrit-Owner: wangsheng Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Wed, 23 Oct 2019 11:32:13 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config
Hello Quanlong Huang, Hao Hao, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14398 to look at the new patch set (#6). Change subject: IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config .. IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config When catalogd connected to mysql/postgresql directly instead of use HMS, a kudu managerd table created DDL would failed due to Preconditions checked failed. The patch fixed the bug by using the if condition instead of the Preconditions functions which lead to exception when lack of HMS config. Tests: * Add test for create/drop kudu managerd table without HMS config * Ran all front-end tests Change-Id: Iacc53801a660c033869cb4747910c98a80e08297 --- M bin/start-daemon.sh M bin/start-impala-cluster.py M fe/src/main/java/org/apache/impala/catalog/KuduTable.java A fe/src/test/java/org/apache/impala/customcluster/CreateKuduTableWithoutHMSTest.java A fe/src/test/resources/hive-site-without-hms.xml.py 5 files changed, 177 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/14398/6 -- To view, visit http://gerrit.cloudera.org:8080/14398 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iacc53801a660c033869cb4747910c98a80e08297 Gerrit-Change-Number: 14398 Gerrit-PatchSet: 6 Gerrit-Owner: wangsheng Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: wangsheng
[Impala-ASF-CR] IMPALA-9071: Handle translated external HDFS table in CATS
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14527 ) Change subject: IMPALA-9071: Handle translated external HDFS table in CATS .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4858/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I460a57dc877ef68ad7dd0864a33b1599b1e9a8d9 Gerrit-Change-Number: 14527 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 23 Oct 2019 11:01:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9071: Handle translated external HDFS table in CATS
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14527 ) Change subject: IMPALA-9071: Handle translated external HDFS table in CATS .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/14527/3/tests/custom_cluster/test_custom_hive_configs.py File tests/custom_cluster/test_custom_hive_configs.py: http://gerrit.cloudera.org:8080/#/c/14527/3/tests/custom_cluster/test_custom_hive_configs.py@25 PS3, Line 25: class TestCustomHiveConfigs(CustomClusterTestSuite): flake8: E302 expected 2 blank lines, found 1 -- To view, visit http://gerrit.cloudera.org:8080/14527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I460a57dc877ef68ad7dd0864a33b1599b1e9a8d9 Gerrit-Change-Number: 14527 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 23 Oct 2019 10:15:13 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9071: Handle translated external HDFS table in CATS
Quanlong Huang has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/14527 ) Change subject: IMPALA-9071: Handle translated external HDFS table in CATS .. IMPALA-9071: Handle translated external HDFS table in CATS After upgrading Hive-3 to a version containing HIVE-22158, it's not allowed for managed tables to be non transactional. Creating non ACID tables will result in creating an external table with table property 'external.table.purge' set to true. In Hive-3, the default location of external HDFS tables will locate in 'metastore.warehouse.external.dir' if it's set. This property is added by HIVE-19837 in Hive 2.7, but hasn't been added to Hive in cdh6 yet. In CTAS statement, we create a temporary HMS Table for the analysis on the Insert part. The table path is created assuming it's a managed table, and the Insert part will use this path for insertion. However, in Hive-3, the created table is translated to an external table. It's not the same as we passed to the HMS API. The created table is located in 'metastore.warehouse.external.dir', while the table path we assumed is in 'metastore.warehouse.dir'. This introduces bugs when these two properties are different. CTAS statement will create table in one place and insert data in another place. This patch adds a new method in MetastoreShim to wrap the difference for getting the default table path for non transactional tables between Hive-2 and Hive-3. This patch also bumps the CDP version to contain HIVE-22158. Note that Ranger version also bumps from 1.2 to 2.0 due to this CDP upgrade, which causes some test failures. They are disabled for follow-up fixs in IMPALA-9047. Changes in the infra: - To support customizing hive configuration, add a env var, CUSTOM_CLASSPATH in bin/set-classpath.sh to be put in front of existing CLASSPATH. The customized hive-site.xml should be put inside CUSTOM_CLASSPATH. - Change hive-site.xml.py to generate a hive-site.xml with non default 'metastore.warehouse.external.dir' - Add an option, --env_vars, in bin/start-impala-cluster.py to pass down CUSTOM_CLASSPATH. Tests: - Add a custom cluster test to start Hive with metastore.warehouse.external.dir being set to non default value. Run it locally using CDP components with HIVE-22158. - Run CORE tests using CDH components Change-Id: I460a57dc877ef68ad7dd0864a33b1599b1e9a8d9 --- M bin/create-test-configuration.sh M bin/impala-config.sh M bin/set-classpath.sh M bin/start-impala-cluster.py M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java M fe/src/test/resources/hive-site.xml.py M testdata/pom.xml M tests/authorization/test_authorized_proxy.py M tests/authorization/test_ranger.py M tests/common/custom_cluster_test_suite.py A tests/custom_cluster/test_custom_hive_configs.py 15 files changed, 253 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/14527/3 -- To view, visit http://gerrit.cloudera.org:8080/14527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I460a57dc877ef68ad7dd0864a33b1599b1e9a8d9 Gerrit-Change-Number: 14527 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang
