[Impala-ASF-CR] IMPALA-9047: Bump CDP BUILD NUMBER to 1471450
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14576 ) Change subject: IMPALA-9047: Bump CDP_BUILD_NUMBER to 1471450 .. IMPALA-9047: Bump CDP_BUILD_NUMBER to 1471450 This patch bumps CDP_BUILD_NUMBER to 1471450. The new GBN upgrades Ranger from 1.2 to 2.0, which includes the change to the default Ranger policies described in https://issues.apache.org/jira/browse/RANGER-2536. Some of the Ranger tests fail, because they assume the older behavior. To address this issue, this patch temporarily disables those affected Ranger tests. Specifically, the affected tests in the following test files are disabled for now. 1. test_authorized_proxy.py 2. test_ranger.py 3. AuthorizationStmtTest.java 4. RangerAuditLogTest.java IMPALA-8842 part 2: (Hive3) Use 'engine' field in HMS stat API The new CDP GBN includes the fix for HIVE-22046. HIVE-22046 added 'engine' column to TAB_COL_STATS and PART_COL_STATS HMS tables. The new column is used to differentiate among column stats computed by different engines. The related HMS API calls were changed accordingly. Part of this patch is Step 4 in a series of steps to coordinate the introduction of HMS API changes to Hive3 and Impala. For more information see IMPALA-8842 part 1. Step 4 replaces *V2 calls with *. The *V2 names were introduced temporarily and will be removed from the HMS API in the near future. Testing: - This patch passes the affected Ranger tests listed above on a local machine. - E2E tests were added to make sure that column statistics are differentiated by engine for partitioned and non-partitioned tables. The tests are executed for transactional and non-transactional tables. Change-Id: I962423cf202ad632b5817669500b3e3479f1a454 Reviewed-on: http://gerrit.cloudera.org:8080/14576 Reviewed-by: Joe McDonnell Tested-by: Impala Public Jenkins --- M bin/impala-config.sh M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java M tests/authorization/test_authorized_proxy.py M tests/authorization/test_ranger.py M tests/common/skip.py M tests/metadata/test_hms_integration.py 8 files changed, 227 insertions(+), 10 deletions(-) Approvals: Joe McDonnell: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/14576 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I962423cf202ad632b5817669500b3e3479f1a454 Gerrit-Change-Number: 14576 Gerrit-PatchSet: 5 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-9047: Bump CDP BUILD NUMBER to 1471450
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14576 ) Change subject: IMPALA-9047: Bump CDP_BUILD_NUMBER to 1471450 .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14576 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I962423cf202ad632b5817669500b3e3479f1a454 Gerrit-Change-Number: 14576 Gerrit-PatchSet: 4 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Sat, 02 Nov 2019 00:32:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9047: Bump CDP BUILD NUMBER to 1471450
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/14576 ) Change subject: IMPALA-9047: Bump CDP_BUILD_NUMBER to 1471450 .. Patch Set 4: Code-Review+2 Carry +2 -- To view, visit http://gerrit.cloudera.org:8080/14576 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I962423cf202ad632b5817669500b3e3479f1a454 Gerrit-Change-Number: 14576 Gerrit-PatchSet: 4 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 01 Nov 2019 20:07:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9047: Bump CDP BUILD NUMBER to 1471450
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/14576 ) Change subject: IMPALA-9047: Bump CDP_BUILD_NUMBER to 1471450 .. Patch Set 4: Running a gerrit-verify-dryrun-external to see if the issue reproduces. We haven't been able to reproduce locally. -- To view, visit http://gerrit.cloudera.org:8080/14576 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I962423cf202ad632b5817669500b3e3479f1a454 Gerrit-Change-Number: 14576 Gerrit-PatchSet: 4 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 01 Nov 2019 20:07:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9047: Bump CDP BUILD NUMBER to 1471450
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14576 ) Change subject: IMPALA-9047: Bump CDP_BUILD_NUMBER to 1471450 .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5164/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/14576 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I962423cf202ad632b5817669500b3e3479f1a454 Gerrit-Change-Number: 14576 Gerrit-PatchSet: 4 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 01 Nov 2019 20:07:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9047: Bump CDP BUILD NUMBER to 1471450
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14576 ) Change subject: IMPALA-9047: Bump CDP_BUILD_NUMBER to 1471450 .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5152/ -- To view, visit http://gerrit.cloudera.org:8080/14576 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I962423cf202ad632b5817669500b3e3479f1a454 Gerrit-Change-Number: 14576 Gerrit-PatchSet: 3 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 30 Oct 2019 22:33:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9047: Bump CDP BUILD NUMBER to 1471450
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14576 ) Change subject: IMPALA-9047: Bump CDP_BUILD_NUMBER to 1471450 .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5152/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14576 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I962423cf202ad632b5817669500b3e3479f1a454 Gerrit-Change-Number: 14576 Gerrit-PatchSet: 3 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 30 Oct 2019 18:13:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9047: Bump CDP BUILD NUMBER to 1471450
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14576 ) Change subject: IMPALA-9047: Bump CDP_BUILD_NUMBER to 1471450 .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14576 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I962423cf202ad632b5817669500b3e3479f1a454 Gerrit-Change-Number: 14576 Gerrit-PatchSet: 3 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 30 Oct 2019 18:13:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9047: Bump CDP BUILD NUMBER to 1471450
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/14576 ) Change subject: IMPALA-9047: Bump CDP_BUILD_NUMBER to 1471450 .. Patch Set 2: Code-Review+2 This looks good to me. -- To view, visit http://gerrit.cloudera.org:8080/14576 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I962423cf202ad632b5817669500b3e3479f1a454 Gerrit-Change-Number: 14576 Gerrit-PatchSet: 2 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 30 Oct 2019 17:53:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9047: Bump CDP BUILD NUMBER to 1471450
Attila Jeges has posted comments on this change. ( http://gerrit.cloudera.org:8080/14576 ) Change subject: IMPALA-9047: Bump CDP_BUILD_NUMBER to 1471450 .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/14576 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I962423cf202ad632b5817669500b3e3479f1a454 Gerrit-Change-Number: 14576 Gerrit-PatchSet: 2 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 30 Oct 2019 14:43:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9047: Bump CDP BUILD NUMBER to 1471450
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14576 ) Change subject: IMPALA-9047: Bump CDP_BUILD_NUMBER to 1471450 .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4905/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14576 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I962423cf202ad632b5817669500b3e3479f1a454 Gerrit-Change-Number: 14576 Gerrit-PatchSet: 2 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 30 Oct 2019 04:55:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9047: Bump CDP BUILD NUMBER to 1471450
Fang-Yu Rao has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/14576 ) Change subject: IMPALA-9047: Bump CDP_BUILD_NUMBER to 1471450 .. IMPALA-9047: Bump CDP_BUILD_NUMBER to 1471450 This patch bumps CDP_BUILD_NUMBER to 1471450. The new GBN upgrades Ranger from 1.2 to 2.0, which includes the change to the default Ranger policies described in https://issues.apache.org/jira/browse/RANGER-2536. Some of the Ranger tests fail, because they assume the older behavior. To address this issue, this patch temporarily disables those affected Ranger tests. Specifically, the affected tests in the following test files are disabled for now. 1. test_authorized_proxy.py 2. test_ranger.py 3. AuthorizationStmtTest.java 4. RangerAuditLogTest.java IMPALA-8842 part 2: (Hive3) Use 'engine' field in HMS stat API The new CDP GBN includes the fix for HIVE-22046. HIVE-22046 added 'engine' column to TAB_COL_STATS and PART_COL_STATS HMS tables. The new column is used to differentiate among column stats computed by different engines. The related HMS API calls were changed accordingly. Part of this patch is Step 4 in a series of steps to coordinate the introduction of HMS API changes to Hive3 and Impala. For more information see IMPALA-8842 part 1. Step 4 replaces *V2 calls with *. The *V2 names were introduced temporarily and will be removed from the HMS API in the near future. Testing: - This patch passes the affected Ranger tests listed above on a local machine. - E2E tests were added to make sure that column statistics are differentiated by engine for partitioned and non-partitioned tables. The tests are executed for transactional and non-transactional tables. Change-Id: I962423cf202ad632b5817669500b3e3479f1a454 --- M bin/impala-config.sh M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java M tests/authorization/test_authorized_proxy.py M tests/authorization/test_ranger.py M tests/common/skip.py M tests/metadata/test_hms_integration.py 8 files changed, 227 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/14576/2 -- To view, visit http://gerrit.cloudera.org:8080/14576 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I962423cf202ad632b5817669500b3e3479f1a454 Gerrit-Change-Number: 14576 Gerrit-PatchSet: 2 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-9047: Bump CDP BUILD NUMBER to 1471450
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/14576 ) Change subject: IMPALA-9047: Bump CDP_BUILD_NUMBER to 1471450 .. Patch Set 1: (2 comments) A couple comments on your commit comment, but I think otherwise, this seems reasonable. http://gerrit.cloudera.org:8080/#/c/14576/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/14576/1//COMMIT_MSG@9 PS1, Line 9: Some test cases would fail : due to this change. Specifically, some Ranger related E2E tests and FE : tests would fail when we are using a newer version of Ranger (i.e., : ranger-2.0.0.7.0.2.0-50 instead of ranger-1.2.0.7.1.0.0-33) due to the : changes to the default Ranger policies described at : https://issues.apache.org/jira/browse/RANGER-2536. I would describe this as the new GBN having a Ranger upgrade from 1.2 to 2.0. Something like: The new GBN upgrades Ranger from 1.2 to 2.0, which includes the change to the default Ranger policies describe in https://issues.apache.org/jira/browse/RANGER-2536. The Ranger tests fail, because they assume the older behavior. http://gerrit.cloudera.org:8080/#/c/14576/1//COMMIT_MSG@25 PS1, Line 25: Moreover, two HMS related E2E tests in test_hms_integration.py fail when : we are using a newer version of Hive (i.e., hive-3.1.2000.7.0.2.0-50 : instead of hive-2.1.1 by setting USE_CDP_HIVE to true when building : Impala). To address this issue, this patch temporarily skips those tests. USE_CDP_HIVE=true is a known configuration that uses Hive 3. The difference here is not a difference between Hive 2 and Hive 3, but between this latest Hive 3 and Hive 3 from the previous GBN. The behavior has permanently changed, so the skip is not temporary. I think I would drop this paragraph, and add a sentence to the start of the next paragraph that the new CDP GBN contains HIVE-22046. I would also include a line saying this is IMPALA-8842 part 2. So, this would end up looking like: IMPALA-8842 part 2: (Hive3) Use 'engine' field in HMS stat API The new CDP GBN includes the fix for HIVE-22046. HIVE-22046 added 'engine' column... (rest of paragraph) -- To view, visit http://gerrit.cloudera.org:8080/14576 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I962423cf202ad632b5817669500b3e3479f1a454 Gerrit-Change-Number: 14576 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 30 Oct 2019 00:36:57 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9047: Bump CDP BUILD NUMBER to 1471450
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14576 ) Change subject: IMPALA-9047: Bump CDP_BUILD_NUMBER to 1471450 .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4902/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14576 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I962423cf202ad632b5817669500b3e3479f1a454 Gerrit-Change-Number: 14576 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 29 Oct 2019 22:38:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9047: Bump CDP BUILD NUMBER to 1471450
Fang-Yu Rao has abandoned this change. ( http://gerrit.cloudera.org:8080/14529 ) Change subject: IMPALA-9047: Bump CDP_BUILD_NUMBER to 1471450 .. Abandoned I have submitted a revised patch so this old patch is no longer needed. -- To view, visit http://gerrit.cloudera.org:8080/14529 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: Ifbc42a73fe98c437edc20a8f57aba51e3096a09d Gerrit-Change-Number: 14529 Gerrit-PatchSet: 4 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-9047: Bump CDP BUILD NUMBER to 1471450
Fang-Yu Rao has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14576 Change subject: IMPALA-9047: Bump CDP_BUILD_NUMBER to 1471450 .. IMPALA-9047: Bump CDP_BUILD_NUMBER to 1471450 This patch bumps CDP_BUILD_NUMBER to 1471450. Some test cases would fail due to this change. Specifically, some Ranger related E2E tests and FE tests would fail when we are using a newer version of Ranger (i.e., ranger-2.0.0.7.0.2.0-50 instead of ranger-1.2.0.7.1.0.0-33) due to the changes to the default Ranger policies described at https://issues.apache.org/jira/browse/RANGER-2536. To address this issue, this patch temporarily disables those affected Ranger tests. Specifically, the affected tests in the following test files are disabled for now. 1. test_authorized_proxy.py 2. test_ranger.py 3. AuthorizationStmtTest.java 4. RangerAuditLogTest.java Moreover, two HMS related E2E tests in test_hms_integration.py fail when we are using a newer version of Hive (i.e., hive-3.1.2000.7.0.2.0-50 instead of hive-2.1.1 by setting USE_CDP_HIVE to true when building Impala). To address this issue, this patch temporarily skips those tests. On the other hand, HIVE-22046 added 'engine' column to TAB_COL_STATS and PART_COL_STATS HMS tables. The new column is used to differentiate among column stats computed by different engines. The related HMS API calls were changed accordingly. Part of this patch is Step 4 in a series of steps to coordinate the introduction of HMS API changes to Hive3 and Impala. For more information see IMPALA-8842 part 1. Step 4 replaces *V2 calls with *. The *V2 names were introduced temporarily and will be removed from the HMS API in the near future. Testing: - This patch passes the affected Ranger tests listed above on a local machine. - E2E tests were added to make sure that column statistics are differentiated by engine for partitioned and non-partitioned tables. The tests are executed for transactional and non-transactional tables. Change-Id: I962423cf202ad632b5817669500b3e3479f1a454 --- M bin/impala-config.sh M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java M tests/authorization/test_authorized_proxy.py M tests/authorization/test_ranger.py M tests/common/skip.py M tests/metadata/test_hms_integration.py 8 files changed, 227 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/14576/1 -- To view, visit http://gerrit.cloudera.org:8080/14576 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I962423cf202ad632b5817669500b3e3479f1a454 Gerrit-Change-Number: 14576 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-9047: Bump CDP BUILD NUMBER to 1471450
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14529 ) Change subject: IMPALA-9047: Bump CDP_BUILD_NUMBER to 1471450 .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4880/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14529 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifbc42a73fe98c437edc20a8f57aba51e3096a09d Gerrit-Change-Number: 14529 Gerrit-PatchSet: 4 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 24 Oct 2019 23:30:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9047: Bump CDP BUILD NUMBER to 1471450
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14529 ) Change subject: IMPALA-9047: Bump CDP_BUILD_NUMBER to 1471450 .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4879/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14529 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifbc42a73fe98c437edc20a8f57aba51e3096a09d Gerrit-Change-Number: 14529 Gerrit-PatchSet: 3 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 24 Oct 2019 23:29:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9047: Bump CDP BUILD NUMBER to 1471450
Fang-Yu Rao has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/14529 ) Change subject: IMPALA-9047: Bump CDP_BUILD_NUMBER to 1471450 .. IMPALA-9047: Bump CDP_BUILD_NUMBER to 1471450 This patch bumps CDP_BUILD_NUMBER to 1471450. Some test cases would fail due to this change. For example, some Ranger related E2E tests and FE tests would fail when we are using a newer version of Ranger (e.g., ranger-2.0.0.7.0.2.0-50 instead of ranger-1.2.0.7.1.0.0-33) due to the changes to the default Ranger policies described at https://issues.apache.org/jira/browse/RANGER-2536. To address this issue, this patch temporarily disables those affected Ranger tests. Specifically, the affected tests in the following test files are disabled for now. 1. test_authorized_proxy.py 2. test_ranger.py 3. AuthorizationStmtTest.java 4. RangerAuditLogTest.java Testing: - This patch passes the affected Ranger tests listed above on a local machine. Change-Id: Ifbc42a73fe98c437edc20a8f57aba51e3096a09d --- M bin/impala-config.sh M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java M tests/authorization/test_authorized_proxy.py M tests/authorization/test_ranger.py 5 files changed, 119 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/14529/4 -- To view, visit http://gerrit.cloudera.org:8080/14529 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifbc42a73fe98c437edc20a8f57aba51e3096a09d Gerrit-Change-Number: 14529 Gerrit-PatchSet: 4 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-9047: Bump CDP BUILD NUMBER to 1471450
Fang-Yu Rao has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/14529 ) Change subject: IMPALA-9047: Bump CDP_BUILD_NUMBER to 1471450 .. IMPALA-9047: Bump CDP_BUILD_NUMBER to 1471450 This patch bumps CDP_BUILD_NUMBER to 1471450. Some test cases would fail due to this change. For example, some Ranger related E2E tests and FE tests would fail when we are using a newer version of Ranger (e.g., ranger-2.0.0.7.0.2.0-50 instead of ranger-1.2.0.7.1.0.0-33) due to the changes to the default Ranger policies described at https://issues.apache.org/jira/browse/RANGER-2536. To address this issue, this patch temporarily disables those affected Ranger tests. Specifically, the affected tests in the following test files are disabled for now. 1. test_authorized_proxy.py 2. test_ranger.py 3. AuthorizationStmtTest.java 4. RangerAuditLogTest.java Testing: - This patch passes the affected Ranger tests listed above on a local machine. Change-Id: Ifbc42a73fe98c437edc20a8f57aba51e3096a09d --- M bin/impala-config.sh M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java M tests/authorization/test_authorized_proxy.py M tests/authorization/test_ranger.py 5 files changed, 119 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/14529/3 -- To view, visit http://gerrit.cloudera.org:8080/14529 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifbc42a73fe98c437edc20a8f57aba51e3096a09d Gerrit-Change-Number: 14529 Gerrit-PatchSet: 3 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang
