[Impala-ASF-CR] IMPALA-8454 (part 1): Refactor file descriptor loading code
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12950 ) Change subject: IMPALA-8454 (part 1): Refactor file descriptor loading code .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4078/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12950 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I59edf493b9ba38be5f556b4795a7684d9c9e3a07 Gerrit-Change-Number: 12950 Gerrit-PatchSet: 7 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Fri, 26 Apr 2019 05:56:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8454 (part 1): Refactor file descriptor loading code
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12950 ) Change subject: IMPALA-8454 (part 1): Refactor file descriptor loading code .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12950 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I59edf493b9ba38be5f556b4795a7684d9c9e3a07 Gerrit-Change-Number: 12950 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Fri, 26 Apr 2019 05:56:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8454 (part 1): Refactor file descriptor loading code
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12950 ) Change subject: IMPALA-8454 (part 1): Refactor file descriptor loading code .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12950 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I59edf493b9ba38be5f556b4795a7684d9c9e3a07 Gerrit-Change-Number: 12950 Gerrit-PatchSet: 7 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Fri, 26 Apr 2019 05:56:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8446: Create a unit test for Admission Controller.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13078 ) Change subject: IMPALA-8446: Create a unit test for Admission Controller. .. IMPALA-8446: Create a unit test for Admission Controller. This test allows construction of white box tests that exercise Admission Controller code. The initial test cases are (1) a test which does the following: + creates a RequestPoolService which reads some Admission Controller configuration files + creates an Admission Controller object + creates a QuerySchedule for a query + tests if the query can be admitted by the Admission Controller + simulates activity in the cluster which consumes memory + tests that the earlier query cannot now be admitted by the Admission Controller (2) a test which verifies that the configuration files are read correctly. TESTING: Ran end-to-end tests cleanly and checked that the new tests were executed. Change-Id: I8a840590b868f2df1a06f3f397b7b0fc2b29462c Reviewed-on: http://gerrit.cloudera.org:8080/13078 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/scheduling/CMakeLists.txt A be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/query-schedule.cc M be/src/scheduling/query-schedule.h 6 files changed, 263 insertions(+), 17 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13078 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I8a840590b868f2df1a06f3f397b7b0fc2b29462c Gerrit-Change-Number: 13078 Gerrit-PatchSet: 7 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8454 (part 3): enable recursive file listing by default
Hello Vihang Karajgaonkar, Sudhanshu Arora, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/13127 to review the following change. Change subject: IMPALA-8454 (part 3): enable recursive file listing by default .. IMPALA-8454 (part 3): enable recursive file listing by default This enables recursive listing of files within partition directories by default. This is a behavior change, but in fact makes Impala consistent with modern versions of Hive, Spark, Presto, etc. In fact, this is necessary for querying certain Hive tables which have been written out by a query containing a UNION ALL clause if that query is executed by Tez (see HIVE-12812 for example). Technically, this is an incompatible change. Although it's unlikely people were relying on the non-recursive listing, this patch offers two escape hatches: - an individual table may be marked with the 'impala.disable.recursive.listing' property - impala may be globally configured with --recursively_list_partitions=false Given that we know this behavior is inconsistent with other SQL engines, and that there is no performance benefit to not recursing in the common case that there _are_ no subdirectories, I made the flag "hidden" and did not document the new table property. These are only "chicken bit" flags. Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8 --- M be/src/common/global-flags.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java A tests/metadata/test_recursive_listing.py 8 files changed, 150 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/13127/1 -- To view, visit http://gerrit.cloudera.org:8080/13127 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ib30e2bcaf820210f2faa8f159d1af2f947a4d0e8 Gerrit-Change-Number: 13127 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8446: Create a unit test for Admission Controller.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13078 ) Change subject: IMPALA-8446: Create a unit test for Admission Controller. .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13078 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8a840590b868f2df1a06f3f397b7b0fc2b29462c Gerrit-Change-Number: 13078 Gerrit-PatchSet: 6 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 26 Apr 2019 05:47:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8454 (part 1): Refactor file descriptor loading code
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12950 ) Change subject: IMPALA-8454 (part 1): Refactor file descriptor loading code .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4077/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/12950 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I59edf493b9ba38be5f556b4795a7684d9c9e3a07 Gerrit-Change-Number: 12950 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Fri, 26 Apr 2019 04:30:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8454 (part 2): Initial support for recursive file listing within a partition
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12991 ) Change subject: IMPALA-8454 (part 2): Initial support for recursive file listing within a partition .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4076/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/12991 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9b151d7abb8443c0d9de0a0d82a9f13e07ad5109 Gerrit-Change-Number: 12991 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 26 Apr 2019 04:30:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4075/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 9 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 26 Apr 2019 04:30:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8419 : Validate event processing related configurations
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13019 ) Change subject: IMPALA-8419 : Validate event processing related configurations .. Patch Set 9: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4072/ -- To view, visit http://gerrit.cloudera.org:8080/13019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I94c2783e36287a65122003aa55d8075a806bc606 Gerrit-Change-Number: 13019 Gerrit-PatchSet: 9 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 26 Apr 2019 04:31:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2928/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 9 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 26 Apr 2019 02:50:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8119: document how to set heap size in docker
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13119 ) Change subject: IMPALA-8119: document how to set heap size in docker .. IMPALA-8119: document how to set heap size in docker JAVA_TOOL_OPTIONS is a standard mechanism to pass arguments to a JVM. Let's just document this as the canonical way to pass in the heap size. Change-Id: Ie6ddba3c42a698b52d7c4e2ff6a9c73068e198b2 Reviewed-on: http://gerrit.cloudera.org:8080/13119 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M bin/start-impala-cluster.py M docker/README.md M docker/daemon_entrypoint.sh 3 files changed, 9 insertions(+), 9 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13119 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie6ddba3c42a698b52d7c4e2ff6a9c73068e198b2 Gerrit-Change-Number: 13119 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2927/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 3 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 26 Apr 2019 02:31:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8119: document how to set heap size in docker
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13119 ) Change subject: IMPALA-8119: document how to set heap size in docker .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13119 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie6ddba3c42a698b52d7c4e2ff6a9c73068e198b2 Gerrit-Change-Number: 13119 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Fri, 26 Apr 2019 02:28:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/13005 )
Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive
3.1.0
..
Patch Set 9:
(25 comments)
http://gerrit.cloudera.org:8080/#/c/13005/9/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/13005/9/bin/impala-config.sh@196
PS9, Line 196: # When USE_CDP_HIVE is set we use the latest hive version
available to deply in minicluster
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/13005/9/bin/impala-config.sh@200
PS9, Line 200: # TODO(Vihang) we should repackage the tarballs so that the
src and binaries are extracted
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/13005/9/bin/impala-config.sh@209
PS9, Line 209: export
HIVE_HOME="$IMPALA_TOOLCHAIN/cdh_components-${CDH_BUILD_NUMBER}/hive-${MINICLUSTER_HIVE_VERSION}"
line too long (106 > 90)
http://gerrit.cloudera.org:8080/#/c/13005/9/bin/impala-config.sh@546
PS9, Line 546: export
HIVE_METASTORE_THRIFT_DIR=$CDP_COMPONENTS_HOME/apache-hive-${IMPALA_HIVE_VERSION}-src/standalone-metastore/src/main/thrift
line too long (129 > 90)
http://gerrit.cloudera.org:8080/#/c/13005/9/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/13005/9/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1073
PS9, Line 1073:
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/13005/9/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1169
PS9, Line 1169:
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/13005/9/testdata/bin/run-hive-server.sh
File testdata/bin/run-hive-server.sh:
http://gerrit.cloudera.org:8080/#/c/13005/9/testdata/bin/run-hive-server.sh@66
PS9, Line 66: export HIVE_METASTORE_HADOOP_OPTS="-verbose:class -Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=30010"
line too long (121 > 90)
http://gerrit.cloudera.org:8080/#/c/13005/9/tests/authorization/test_owner_privileges.py
File tests/authorization/test_owner_privileges.py:
http://gerrit.cloudera.org:8080/#/c/13005/9/tests/authorization/test_owner_privileges.py@103
PS9, Line 103: )
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/9/tests/authorization/test_owner_privileges.py@179
PS9, Line 179: =
flake8: E501 line too long (93 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/9/tests/authorization/test_owner_privileges.py@182
PS9, Line 182: =
flake8: E501 line too long (93 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/9/tests/authorization/test_owner_privileges.py@186
PS9, Line 186: )
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/9/tests/authorization/test_owner_privileges.py@193
PS9, Line 193: ,
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/9/tests/authorization/test_owner_privileges.py@213
PS9, Line 213: )
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/9/tests/authorization/test_owner_privileges.py@226
PS9, Line 226: )
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/9/tests/authorization/test_owner_privileges.py@236
PS9, Line 236: )
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/9/tests/authorization/test_owner_privileges.py@282
PS9, Line 282: _
flake8: E501 line too long (100 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/9/tests/authorization/test_owner_privileges.py@287
PS9, Line 287: t
flake8: E501 line too long (101 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/9/tests/authorization/test_owner_privileges.py@293
PS9, Line 293: 2
flake8: E501 line too long (92 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/9/tests/authorization/test_owner_privileges.py@355
PS9, Line 355: _
flake8: E501 line too long (100 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/9/tests/authorization/test_owner_privileges.py@360
PS9, Line 360: t
flake8: E501 line too long (101 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/9/tests/authorization/test_owner_privileges.py@364
PS9, Line 364: 2
flake8: E501 line too long (92 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/9/tests/authorization/test_owner_privileges.py@368
PS9, Line 368: )
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/9/tests/authorization/test_owner_privileges.py@412
PS9, Line 412: t
flake8: E501 line too long (101 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/9/tests/authorization/test_owner_privileges.py@421
PS9, Line 421: i
flake8: E501 line too long (102 > 90
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Vihang Karajgaonkar has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 This change upgrades the hive dependencies of Impala to use Hive 3.1.0 based binaries. Most of the changes in this patch are based off patches provided by Todd (links available in JIRA). Upgrading the dependencies allows us to work with both Hive 3.1.0 and Hive 2.1.0 in the same code line. In order to do this, the patch trims down a lot of unnecessary hive dependencies of the front end code by creating a shaded-deps module. The pom.xml of shaded-deps includes only the files from Hive source which Impala depends for compilation. Additionally, it also uses a custom build of Hive which is based of Hive 3.1.0. This custom build includes patches for HIVE-21596 and HIVE-21586 which are needed by Impala so that it can compile against Hive-3 libraries and be able to talk to HMS-2.x metastore. Once these patches are merged we can get rid of this custom build and rely on more official sources of Hive builds. The patch also changes impala-config.sh so that it always downloads the Hive-3 libraries from the toolchain. The code is always built using Hive-3 jars. However, based on the value of USE_CDP_HIVE, the minicluster is deployed using Hive-3 or Hive-2 binaries. Since Impala implements HiveServer2's TCLIService.thrift interface, it requires us to use the existing mechanism of copying the hive-2/api TCLIService.thrift to hive-3/api. It also adds a few environment variables which point to the metastore's thrift file and the CDH Hive version. Testing: 1. Code compiles and runs against both HMS-3 and HMS-2 2. Ran full-suite of tests using the private jenkins job against HMS-2 3. Running full-tests against HMS-3 will need more work like supporting Tez in the mini-cluster (for dataloading) and HMS transaction support since HMS3 create transactional tables by default. This will be taken up in subsequent change. Notes: 1. Patch uses a custom build of Hive to be deployed in mini-cluster. This build has the fixes for HIVE-21596 2. The Hive 3.1 does not include some of the patches like HIVE-21077. Some of the recent code in Impala depends on that patch to exist in hive. We will need to port HIVE-21077 to Hive 3.1.0 in order to compile against Hive 3.1.0 jars. I will continue working on getting this merged along with HIVE-21586 3. Some of the existing tests rely on the fact the UDFs implement the UDF interface in Hive (UDFLength, UDFHour, UDFYear). These built-in hive functions have been moved to use GenericUDF interface in Hive 3. Impala currently only supports UDFExecutor. In order to have a full compatibility with all the functions in Hive 2.x we should support GenericUDFs too. That would be taken up as a separate patch. 4. The Sentry object ownership tests are flaky. There are race-conditions in that code which get exposed for some by this patch. For example, when a database is created, the object privileges are immediately updated in the catalog cache. If Sentry has not been updated and there is a refresh authorization call in between it can clear the privilege from the cache causing the subsequent statement using that privilege to fail. The patch adds sleep statements in the test to work-around these issues. Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 --- M CMakeLists.txt M README.md M bin/bootstrap_toolchain.py M bin/impala-config.sh M bin/set-classpath.sh M common/thrift/.gitignore M common/thrift/CMakeLists.txt M fe/CMakeLists.txt M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java M fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java M fe/src/main/java/org/apache/impala/catalog/TableLoader.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java M fe/src/main/java/org/apache/impala/service/MetadataOp.java A fe/src/main/java/org/apache/impala/util/MetadataFormatUtils.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java M fe/src/test/java/org/apache/impala/testutil/EmbeddedMetastoreClientPool.java M impala-parent/pom.xml A shaded-deps/.gitignore A
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Anurag Mantripragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained events processing for
partition level HMS events.
..
Patch Set 3:
(15 comments)
Thanks for your comments Vihang and Bharath. Please review my comments.
http://gerrit.cloudera.org:8080/#/c/13111/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13111/2//COMMIT_MSG@7
PS2, Line 7: IMPALA-7973: Add support for fine grained events processing for
> nit: mention about "events" in the title? Otherwise it looks very general .
Done
http://gerrit.cloudera.org:8080/#/c/13111/2//COMMIT_MSG@13
PS2, Line 13:
> nit: ..affected..
Done
http://gerrit.cloudera.org:8080/#/c/13111/2//COMMIT_MSG@14
PS2, Line 14: refresh affected partitions
: in case of add/d
> Not sure I understand this, can you please clarify?
Done
http://gerrit.cloudera.org:8080/#/c/13111/2//COMMIT_MSG@19
PS2, Line 19: on to
> nit:typo
Done
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2085
PS2, Line 2085:
> nit: ..it..
Done
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2085
PS2, Line 2085: s. Returns true if reload of the partition succeeds,
:* false othe
> Instead of referring to specific methods, rephrase it to something like re
Done
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2089
PS2, Line 2089: reloadP
> nit:call "reload" to be consistent ?
Done
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1102
PS2, Line 1102: per(catalog, metrics, event);
:
Preconditions.checkState(eventType_.equals(MetastoreEventType.ADD_PARTITION));
: if (event.getMessage() == null) {
: throw new IllegalStateException(debugString("Event
messag
> nit: Not super clear what this is. How are HMS transactional semantics rela
Done
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1106
PS2, Line 1106:
> Would be good if you can log a useful information about the event like numb
Done
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1107
PS2, Line 1107: {
: AddPartitionMessage addPartitionMessage_ =
: MetastoreEventsProcessor.getMessageFactory()
: .getDeserializer()
: .getAddPartitionMessage(event.getMessage());
: addedPartitions_ =
:
Lists.newArrayList(addPartitionMessage_.getPartitionObjs());
: Preconditions.checkState(addedPartitions_.size() > 0);
: //
> We can avoid one unnecessary conversion from Partition -> partSpecMap -> TP
Created static method in base class to do this.
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1118
PS2, Line 1118:
> nit: include the partition name too
This scenario is only if the table is not present in the catalog, Next load of
the table will refresh all the partitions anyway. Do you think logging which
partition failed is useful to the user in this case?
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1125
PS2, Line 1125:
> include the partition name
In success case, it can be a really log list of partitions. (eg:- dynamically
loaded partitions)
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1194
PS2, Line 1194: */
: private AlterPartitionEvent(CatalogServiceCatalog catalog,
Metrics metrics,
: NotificationEvent event) throws
MetastoreNotificationException {
: super(catalog, metrics, event);
:
Preconditions.checkState(eventType_.equals(MetastoreEventType.ALTER_PARTITION));
: Preconditions.checkNotNull(event.getMessage());
: AlterPartitionMessage alterPartitionMessage =
:
MetastoreEventsProcessor.getMessageFactory().getDeserializer()
:
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Anurag Mantripragada has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. IMPALA-7973: Add support for fine grained events processing for partition level HMS events. This patch adds support for fine grained updates for add/drop/alter partition events. Currently, partition events invalidate the table. This can be expensive for large tables. Here, we refresh affected partitions in case of add/drop/alter partition events. HMS processes add/drop partitions in a transaction, which means there may be multiple partitions affetced in a single add/drop event. We try to refresh all these partitions in a loop. If any of the partition refresh fails, we throw MetastoreNotificationNeedsInvalidateException to mandate a manual invalidate for event processing to continue. Testing: Modified pre-existing tests for partition events to instead test if partitions are added/dropped/altered when event processing is enabled. Change-Id: I213401329f3965dd81055197792ccf8a05368af5 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 3 files changed, 208 insertions(+), 44 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/13111/3 -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 3 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. Patch Set 8: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4071/ -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 8 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 26 Apr 2019 01:23:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8446: Create a unit test for Admission Controller.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13078 ) Change subject: IMPALA-8446: Create a unit test for Admission Controller. .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4074/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13078 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8a840590b868f2df1a06f3f397b7b0fc2b29462c Gerrit-Change-Number: 13078 Gerrit-PatchSet: 6 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 26 Apr 2019 01:23:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-966: Type errors are attributed to wrong expression with insert
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13050 ) Change subject: IMPALA-966: Type errors are attributed to wrong expression with insert .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2925/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13050 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88718fc2cbe1a492165435a542fd2d91d8693a39 Gerrit-Change-Number: 13050 Gerrit-PatchSet: 2 Gerrit-Owner: Alice Fan Gerrit-Reviewer: Alice Fan Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Comment-Date: Fri, 26 Apr 2019 00:47:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8454 (part 2): Initial support for recursive file listing within a partition
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12991 ) Change subject: IMPALA-8454 (part 2): Initial support for recursive file listing within a partition .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2926/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12991 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9b151d7abb8443c0d9de0a0d82a9f13e07ad5109 Gerrit-Change-Number: 12991 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 26 Apr 2019 00:44:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8446: Create a unit test for Admission Controller.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13078 ) Change subject: IMPALA-8446: Create a unit test for Admission Controller. .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4073/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13078 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8a840590b868f2df1a06f3f397b7b0fc2b29462c Gerrit-Change-Number: 13078 Gerrit-PatchSet: 6 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 26 Apr 2019 00:32:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8446: Create a unit test for Admission Controller.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13078 ) Change subject: IMPALA-8446: Create a unit test for Admission Controller. .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13078 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8a840590b868f2df1a06f3f397b7b0fc2b29462c Gerrit-Change-Number: 13078 Gerrit-PatchSet: 6 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 26 Apr 2019 00:32:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8446: Create a unit test for Admission Controller.
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/13078 ) Change subject: IMPALA-8446: Create a unit test for Admission Controller. .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13078 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8a840590b868f2df1a06f3f397b7b0fc2b29462c Gerrit-Change-Number: 13078 Gerrit-PatchSet: 5 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 26 Apr 2019 00:32:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 19: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4068/ -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 19 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Fri, 26 Apr 2019 00:28:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8454 (part 2): Initial support for recursive file listing within a partition
Hello Bharath Vissapragada, Vihang Karajgaonkar, Sudhanshu Arora, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12991 to look at the new patch set (#3). Change subject: IMPALA-8454 (part 2): Initial support for recursive file listing within a partition .. IMPALA-8454 (part 2): Initial support for recursive file listing within a partition This adds support to FileMetadataLoader to recursively list a directory and create file descriptors. The changes are as follows: * FileMetadataLoader can now take a 'recursive' argument to trigger the new behavior. All the non-test code paths still use non-recursive (i.e. this new feature isn't exposed for real tables as of yet). * FileSystemUtil has some functionality for recursive directory listing. There are a few notes there around unexpected optimizations for S3 vs HDFS. * Renamed the 'file_name' field to 'relative_path' for FileDescriptor and HDFS splits, since now the file descriptors may be more than a single path component. The new functionality is just unit tested at the moment. Later, this functionality will be tied into the actual table code paths to solve issues with Hive interop, along with end-to-end tests. Change-Id: I9b151d7abb8443c0d9de0a0d82a9f13e07ad5109 --- M be/src/exec/hdfs-scan-node-base.cc M be/src/scheduling/scheduler-test-util.cc M be/src/scheduling/scheduler.cc M common/fbs/CatalogObjects.fbs M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java M fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java A fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java M fe/src/test/java/org/apache/impala/catalog/HdfsPartitionTest.java M fe/src/test/java/org/apache/impala/planner/ExplainTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java M fe/src/test/java/org/apache/impala/testutil/BlockIdGenerator.java 17 files changed, 267 insertions(+), 59 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/12991/3 -- To view, visit http://gerrit.cloudera.org:8080/12991 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9b151d7abb8443c0d9de0a0d82a9f13e07ad5109 Gerrit-Change-Number: 12991 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-966: Type errors are attributed to wrong expression with insert
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/13050 )
Change subject: IMPALA-966: Type errors are attributed to wrong expression with
insert
..
Patch Set 2:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/13050/2/fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
File fe/src/main/java/org/apache/impala/analysis/InsertStmt.java:
http://gerrit.cloudera.org:8080/#/c/13050/2/fe/src/main/java/org/apache/impala/analysis/InsertStmt.java@690
PS2, Line 690: if (queryStmt_.widestExprs_ == null ||
queryStmt_.widestExprs_.size() != selectListExprs.size()) {
line too long (104 > 90)
http://gerrit.cloudera.org:8080/#/c/13050/2/fe/src/main/java/org/apache/impala/analysis/StatementBase.java
File fe/src/main/java/org/apache/impala/analysis/StatementBase.java:
http://gerrit.cloudera.org:8080/#/c/13050/2/fe/src/main/java/org/apache/impala/analysis/StatementBase.java@195
PS2, Line 195:*
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/13050/2/fe/src/main/java/org/apache/impala/analysis/StatementBase.java@204
PS2, Line 204: Expr srcExpr, boolean strictDecimal, Expr
widestTypeSrcExpr) throws AnalysisException {
line too long (93 > 90)
http://gerrit.cloudera.org:8080/#/c/13050/2/fe/src/main/java/org/apache/impala/analysis/StatementBase.java@224
PS2, Line 224: dstTableName, widestTypeSrcExpr.toSql(),
srcExprType.toSql(), dstColType.toSql(),
line too long (91 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/13050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I88718fc2cbe1a492165435a542fd2d91d8693a39
Gerrit-Change-Number: 13050
Gerrit-PatchSet: 2
Gerrit-Owner: Alice Fan
Gerrit-Reviewer: Alice Fan
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Paul Rogers
Gerrit-Comment-Date: Fri, 26 Apr 2019 00:00:47 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-966: Type errors are attributed to wrong expression with insert
Alice Fan has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13050 Change subject: IMPALA-966: Type errors are attributed to wrong expression with insert .. IMPALA-966: Type errors are attributed to wrong expression with insert When multiple incompatible values insert into a table, error should blame associated expression. Testing: - Added a test to AnalyzeStmtsTest.java Change-Id: I88718fc2cbe1a492165435a542fd2d91d8693a39 --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java M fe/src/main/java/org/apache/impala/analysis/StatementBase.java M fe/src/main/java/org/apache/impala/analysis/UnionStmt.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java 6 files changed, 55 insertions(+), 17 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/13050/2 -- To view, visit http://gerrit.cloudera.org:8080/13050 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I88718fc2cbe1a492165435a542fd2d91d8693a39 Gerrit-Change-Number: 13050 Gerrit-PatchSet: 2 Gerrit-Owner: Alice Fan Gerrit-Reviewer: Alice Fan Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Paul Rogers
[Impala-ASF-CR] IMPALA-8419 : Validate event processing related configurations
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13019 ) Change subject: IMPALA-8419 : Validate event processing related configurations .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4072/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I94c2783e36287a65122003aa55d8075a806bc606 Gerrit-Change-Number: 13019 Gerrit-PatchSet: 9 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 25 Apr 2019 23:47:45 + Gerrit-HasComments: No
[Impala-ASF-CR] Configure Hive 3's HS2 to execute queries using Tez local mode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12931 ) Change subject: Configure Hive 3's HS2 to execute queries using Tez local mode .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4070/ -- To view, visit http://gerrit.cloudera.org:8080/12931 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I76e47fbd1d6ff5103d81a8de430d5465dba284cd Gerrit-Change-Number: 12931 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 25 Apr 2019 23:47:43 + Gerrit-HasComments: No
[Impala-ASF-CR] Remove hive-benchmark test workload
Todd Lipcon has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13117 ) Change subject: Remove hive-benchmark test workload .. Remove hive-benchmark test workload This workload was added in 2012 or so, and hasn't actually been able to run in years: - fails to load because the name has a '-' in it, and we use unquoted database names in the load script - the SQL to load it seems to refer to testdata files which don't exist Even if it did load, it looks like it's only a set of ~10 trivial queries which I doubt have any real benchmark interest in modern days. Change-Id: I383638ef4acce59792fee287da0016b3d949e693 Reviewed-on: http://gerrit.cloudera.org:8080/13117 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M testdata/bin/generate-schema-statements.py M testdata/datasets/README D testdata/datasets/hive-benchmark/hive-benchmark_schema_template.sql M testdata/datasets/tpcds/tpcds_schema_template.sql M testdata/datasets/tpch/tpch_schema_template.sql D testdata/workloads/hive-benchmark/hive-benchmark_core.csv D testdata/workloads/hive-benchmark/hive-benchmark_dimensions.csv D testdata/workloads/hive-benchmark/hive-benchmark_exhaustive.csv D testdata/workloads/hive-benchmark/hive-benchmark_pairwise.csv D testdata/workloads/hive-benchmark/queries/hive-benchmark.test 10 files changed, 34 insertions(+), 215 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/13117 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I383638ef4acce59792fee287da0016b3d949e693 Gerrit-Change-Number: 13117 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-2990: timeout unresponsive queries in coordinator
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12299 ) Change subject: IMPALA-2990: timeout unresponsive queries in coordinator .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2924/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12299 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I196c8c6a5633b1960e2c3a3884777be9b3824987 Gerrit-Change-Number: 12299 Gerrit-PatchSet: 10 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 25 Apr 2019 22:51:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-2990: timeout unresponsive queries in coordinator
Hello Michael Ho, Philip Zeyliger, Todd Lipcon, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12299 to look at the new patch set (#10). Change subject: IMPALA-2990: timeout unresponsive queries in coordinator .. IMPALA-2990: timeout unresponsive queries in coordinator The coordinator currently waits indefinitely if it does not receive a status report from a backend. This could cause a query to hang indefinitely in certain situations, for example if the backend decides to cancel itself as a result of failed status report rpcs. This patch adds a thread to ImpalaServer which periodically iterates over all queries for which that server is the coordinator and cancels any that haven't had a report from a backend in a certain amount of time. This patch adds two flags: --status_report_max_retry_s: the maximum number of seconds a backend will attempt to send status reports before giving up. This is used in place of --status_report_max_retries which is now deprecated. --status_report_cancellation_padding: the coordinator will wait --status_report_max_retry_s * (1 + --status_report_cancellation_padding / 100) before concluding a backend is not responding and cancelling the query. Testing: - Added a functional test that runs a query that is cancelled through the new mechanism. - Passed a full set of exhaustive tests. Ran tests on a 10 node cluster loaded with tpch 500: - Ran the stress test for 1000 queries with the debug actions: 'REPORT_EXEC_STATUS_DELAY:JITTER@1000' Prior to this patch, this setup results in hanging queries. With this patch, no hangs were observed. - Ran perf tests with 4 concurrent streams, 3 iterations per query. Found no change in performance. Change-Id: I196c8c6a5633b1960e2c3a3884777be9b3824987 --- M be/src/common/global-flags.cc M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/runtime/query-state.cc M be/src/runtime/query-state.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M common/thrift/ImpalaInternalService.thrift M common/thrift/generate_error_codes.py M tests/custom_cluster/test_rpc_timeout.py 12 files changed, 190 insertions(+), 46 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/12299/10 -- To view, visit http://gerrit.cloudera.org:8080/12299 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I196c8c6a5633b1960e2c3a3884777be9b3824987 Gerrit-Change-Number: 12299 Gerrit-PatchSet: 10 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4071/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 8 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 25 Apr 2019 22:29:17 + Gerrit-HasComments: No
[Impala-ASF-CR] Remove hive-benchmark test workload
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13117 ) Change subject: Remove hive-benchmark test workload .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13117 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I383638ef4acce59792fee287da0016b3d949e693 Gerrit-Change-Number: 13117 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 25 Apr 2019 22:29:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-2990: timeout unresponsive queries in coordinator
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/12299 ) Change subject: IMPALA-2990: timeout unresponsive queries in coordinator .. Patch Set 10: Code-Review+1 rebased, carrying forward -- To view, visit http://gerrit.cloudera.org:8080/12299 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I196c8c6a5633b1960e2c3a3884777be9b3824987 Gerrit-Change-Number: 12299 Gerrit-PatchSet: 10 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 25 Apr 2019 22:28:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8419 : Validate event processing related configurations
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13019 ) Change subject: IMPALA-8419 : Validate event processing related configurations .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2923/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I94c2783e36287a65122003aa55d8075a806bc606 Gerrit-Change-Number: 13019 Gerrit-PatchSet: 9 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 25 Apr 2019 22:17:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8446: Create a unit test for Admission Controller.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13078 ) Change subject: IMPALA-8446: Create a unit test for Admission Controller. .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2922/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13078 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8a840590b868f2df1a06f3f397b7b0fc2b29462c Gerrit-Change-Number: 13078 Gerrit-PatchSet: 5 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 25 Apr 2019 22:15:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8444: Fix performance regression when building privilege name
Csaba Ringhofer has posted comments on this change. (
http://gerrit.cloudera.org:8080/13095 )
Change subject: IMPALA-8444: Fix performance regression when building privilege
name
..
Patch Set 8:
(4 comments)
Thanks for the changes! Few more missing bits.
http://gerrit.cloudera.org:8080/#/c/13095/8//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13095/8//COMMIT_MSG@8
PS8, Line 8:
The commit message could also reflect that there are also some fixes related to
case sensitivity.
http://gerrit.cloudera.org:8080/#/c/13095/8//COMMIT_MSG@33
PS8, Line 33: While fixing this, I found a bug where Principal
: stores the PrincipalPrivilege in case insensitive way. This is
true for
: all privilege scopes, except URI. This patch fixes the issue by
storing
: URI privilege in a case sensitive manner.
Is this still correct?
http://gerrit.cloudera.org:8080/#/c/13095/8/fe/src/main/java/org/apache/impala/catalog/CatalogObjectCache.java
File fe/src/main/java/org/apache/impala/catalog/CatalogObjectCache.java:
http://gerrit.cloudera.org:8080/#/c/13095/8/fe/src/main/java/org/apache/impala/catalog/CatalogObjectCache.java@158
PS8, Line 158: }
Is there some kind of whitespace change here?
http://gerrit.cloudera.org:8080/#/c/13095/2/fe/src/main/java/org/apache/impala/catalog/Principal.java
File fe/src/main/java/org/apache/impala/catalog/Principal.java:
http://gerrit.cloudera.org:8080/#/c/13095/2/fe/src/main/java/org/apache/impala/catalog/Principal.java@63
PS2, Line 63:*/
: public boolean addPrivilege(PrincipalPrivilege privilege) {
: return principalPrivileges_.add(privilege);
> I decided to change the whole thing to make the cache stores the privilege
Can you add tests for the bugs that were fixed?
--
To view, visit http://gerrit.cloudera.org:8080/13095
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I942d9b55f07c8972f69e532567d9b7d80fceb6e5
Gerrit-Change-Number: 13095
Gerrit-PatchSet: 8
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Thu, 25 Apr 2019 22:12:03 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8454 (part 1): Refactor file descriptor loading code
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12950 ) Change subject: IMPALA-8454 (part 1): Refactor file descriptor loading code .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4066/ -- To view, visit http://gerrit.cloudera.org:8080/12950 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I59edf493b9ba38be5f556b4795a7684d9c9e3a07 Gerrit-Change-Number: 12950 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Thu, 25 Apr 2019 22:02:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. Patch Set 8: > (3 comments) > > Before more work goes into this, this is changing the Hive major > version number and there should be a discussion on dev@ about this > approach and how it fits into Impala releases. Just sent a note the dev list. Thanks for the suggestion. -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 8 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 25 Apr 2019 22:02:29 + Gerrit-HasComments: No
[Impala-ASF-CR] Configure Hive 3's HS2 to execute queries using Tez local mode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12931 ) Change subject: Configure Hive 3's HS2 to execute queries using Tez local mode .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4070/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/12931 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I76e47fbd1d6ff5103d81a8de430d5465dba284cd Gerrit-Change-Number: 12931 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 25 Apr 2019 22:02:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8419 : Validate event processing related configurations
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13019 ) Change subject: IMPALA-8419 : Validate event processing related configurations .. Patch Set 9: (6 comments) http://gerrit.cloudera.org:8080/#/c/13019/9/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java: http://gerrit.cloudera.org:8080/#/c/13019/9/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@20 PS9, Line 20: import static org.apache.impala.catalog.events.EventProcessorValidator.hasValidMetastoreConfigs; line too long (96 > 90) http://gerrit.cloudera.org:8080/#/c/13019/9/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@21 PS9, Line 21: import static org.apache.impala.catalog.events.EventProcessorValidator.verifyParametersNotFiltered; line too long (99 > 90) http://gerrit.cloudera.org:8080/#/c/13019/9/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/13019/9/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@21 PS9, Line 21: import static org.apache.impala.catalog.events.EventProcessorValidator.DEFAULT_METASTORE_CONFIG_VALUE; line too long (102 > 90) http://gerrit.cloudera.org:8080/#/c/13019/9/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@22 PS9, Line 22: import static org.apache.impala.catalog.events.EventProcessorValidator.validateMetastoreConfigs; line too long (96 > 90) http://gerrit.cloudera.org:8080/#/c/13019/9/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@23 PS9, Line 23: import static org.apache.impala.catalog.events.EventProcessorValidator.validateMetastoreEventParameters; line too long (104 > 90) http://gerrit.cloudera.org:8080/#/c/13019/9/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@72 PS9, Line 72: import org.apache.impala.catalog.events.EventProcessorValidator.MetastoreEventConfigsToValidate; line too long (96 > 90) -- To view, visit http://gerrit.cloudera.org:8080/13019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I94c2783e36287a65122003aa55d8075a806bc606 Gerrit-Change-Number: 13019 Gerrit-PatchSet: 9 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 25 Apr 2019 21:53:57 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8419 : Validate event processing related configurations
Bharath Krishna has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/13019 ) Change subject: IMPALA-8419 : Validate event processing related configurations .. IMPALA-8419 : Validate event processing related configurations Using the Metastore API to get the configuration values, verify that the configurations needed for event processing are set correctly. Also check that the parameters required for event processing is not filtered out by the Hive config METASTORE_PARAMETER_EXCLUDE_PATTERNS. This validation is done while creating the event processor and throws CatalogException if the configuration is incorrect. Testing - Added unit tests Change-Id: I94c2783e36287a65122003aa55d8075a806bc606 --- A fe/src/main/java/org/apache/impala/catalog/events/EventProcessorValidator.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/java/org/apache/impala/catalog/events/SynchronousHMSEventProcessorForTests.java 7 files changed, 507 insertions(+), 59 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/13019/9 -- To view, visit http://gerrit.cloudera.org:8080/13019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I94c2783e36287a65122003aa55d8075a806bc606 Gerrit-Change-Number: 13019 Gerrit-PatchSet: 9 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8446: Create a unit test for Admission Controller.
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/13078 ) Change subject: IMPALA-8446: Create a unit test for Admission Controller. .. Patch Set 5: One more fix to remove an 'inline' which gets optimized away in a release build -- To view, visit http://gerrit.cloudera.org:8080/13078 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8a840590b868f2df1a06f3f397b7b0fc2b29462c Gerrit-Change-Number: 13078 Gerrit-PatchSet: 5 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 25 Apr 2019 21:29:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8446: Create a unit test for Admission Controller.
Andrew Sherman has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/13078 ) Change subject: IMPALA-8446: Create a unit test for Admission Controller. .. IMPALA-8446: Create a unit test for Admission Controller. This test allows construction of white box tests that exercise Admission Controller code. The initial test cases are (1) a test which does the following: + creates a RequestPoolService which reads some Admission Controller configuration files + creates an Admission Controller object + creates a QuerySchedule for a query + tests if the query can be admitted by the Admission Controller + simulates activity in the cluster which consumes memory + tests that the earlier query cannot now be admitted by the Admission Controller (2) a test which verifies that the configuration files are read correctly. TESTING: Ran end-to-end tests cleanly and checked that the new tests were executed. Change-Id: I8a840590b868f2df1a06f3f397b7b0fc2b29462c --- M be/src/scheduling/CMakeLists.txt A be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/query-schedule.cc M be/src/scheduling/query-schedule.h 6 files changed, 263 insertions(+), 17 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/13078/5 -- To view, visit http://gerrit.cloudera.org:8080/13078 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8a840590b868f2df1a06f3f397b7b0fc2b29462c Gerrit-Change-Number: 13078 Gerrit-PatchSet: 5 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2921/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 8 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 25 Apr 2019 21:27:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8444: Fix performance regression when building privilege name
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13095 ) Change subject: IMPALA-8444: Fix performance regression when building privilege name .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2920/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13095 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I942d9b55f07c8972f69e532567d9b7d80fceb6e5 Gerrit-Change-Number: 13095 Gerrit-PatchSet: 8 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 25 Apr 2019 21:22:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2919/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 7 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 25 Apr 2019 21:19:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8119: document how to set heap size in docker
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13119 ) Change subject: IMPALA-8119: document how to set heap size in docker .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4069/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13119 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie6ddba3c42a698b52d7c4e2ff6a9c73068e198b2 Gerrit-Change-Number: 13119 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Thu, 25 Apr 2019 21:12:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8119: document how to set heap size in docker
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13119 ) Change subject: IMPALA-8119: document how to set heap size in docker .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13119 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie6ddba3c42a698b52d7c4e2ff6a9c73068e198b2 Gerrit-Change-Number: 13119 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Thu, 25 Apr 2019 21:12:25 + Gerrit-HasComments: No
[Impala-ASF-CR] Make infra/python compatible with both Python 2 & 3
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13070 ) Change subject: Make infra/python compatible with both Python 2 & 3 .. Patch Set 5: Thanks for the contribution Akshesh! -- To view, visit http://gerrit.cloudera.org:8080/13070 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4285a021bb581f88425daa52ef8a3f844017d82 Gerrit-Change-Number: 13070 Gerrit-PatchSet: 5 Gerrit-Owner: Akshesh Doshi Gerrit-Reviewer: Akshesh Doshi Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 25 Apr 2019 21:12:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8419 : Validate event processing related configurations
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13019 ) Change subject: IMPALA-8419 : Validate event processing related configurations .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2918/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I94c2783e36287a65122003aa55d8075a806bc606 Gerrit-Change-Number: 13019 Gerrit-PatchSet: 7 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 25 Apr 2019 21:05:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7892: [DOCS] Described the new network I/O throughput in query profiles
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/13006 ) Change subject: IMPALA-7892: [DOCS] Described the new network I/O throughput in query profiles .. Patch Set 2: Lars, Could you take a look at the new patch? Thank you! -- To view, visit http://gerrit.cloudera.org:8080/13006 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I25b128bc23f418347b400ca9e694d9d591935592 Gerrit-Change-Number: 13006 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Thu, 25 Apr 2019 20:55:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8149 : Add support for alter database events
Bharath Krishna has posted comments on this change. (
http://gerrit.cloudera.org:8080/13049 )
Change subject: IMPALA-8149 : Add support for alter_database events
..
Patch Set 4:
(4 comments)
Overall looks good, leaving a few comments..
http://gerrit.cloudera.org:8080/#/c/13049/4/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/13049/4/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@757
PS4, Line 757:* If tblName is null, applies to database.
We can say:
if tblName is null, removes version number from database.
if tblName is not null and Table is not incomplete, removes version number from
Table.
http://gerrit.cloudera.org:8080/#/c/13049/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/13049/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3805
PS1, Line 3805: if (db == null) {
> Remove it should be fine?
I think removing is fine. Probably we can change the Preconditions.checkState()
in catalog_.getDb to PreConditions.checkArguments() as Fredy suggested?
http://gerrit.cloudera.org:8080/#/c/13049/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:
http://gerrit.cloudera.org:8080/#/c/13049/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@345
PS4, Line 345:
I feel we should add an additional test, here we are testing if the
Notification Processor is successfully processing the notification events
generated from Hive.
We are not triggering the AlterDatabase operation from Impala in this test. (We
need to do something similar to
MetastoreEventsProcessorTest#alterTableAddColsFromImpala).
http://gerrit.cloudera.org:8080/#/c/13049/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@1481
PS4, Line 1481: eventsProcessor_.processEvents();
Can we check if possible that the counter
MetastoreEventsProcessor.EVENTS_SKIPPED_METRIC is updated correctly here, as we
are skipping some events?
--
To view, visit http://gerrit.cloudera.org:8080/13049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf020e85cae04163bf32e31363eb4119d624640b
Gerrit-Change-Number: 13049
Gerrit-PatchSet: 4
Gerrit-Owner: Anonymous Coward
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Anurag Mantripragada
Gerrit-Reviewer: Bharath Krishna
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Thu, 25 Apr 2019 20:46:33 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8454 (part 1): Refactor file descriptor loading code
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/12950 ) Change subject: IMPALA-8454 (part 1): Refactor file descriptor loading code .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/12950/5/fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java File fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java: http://gerrit.cloudera.org:8080/#/c/12950/5/fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java@86 PS5, Line 86: "); > nit: worth including the partition name? (same with other logs) would rather not add code for this since it's just a developer-facing assertion. This shouldn't ever fire in real life -- To view, visit http://gerrit.cloudera.org:8080/12950 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I59edf493b9ba38be5f556b4795a7684d9c9e3a07 Gerrit-Change-Number: 12950 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Thu, 25 Apr 2019 20:50:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Make infra/python compatible with both Python 2 & 3
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13070 ) Change subject: Make infra/python compatible with both Python 2 & 3 .. Make infra/python compatible with both Python 2 & 3 Change-Id: If4285a021bb581f88425daa52ef8a3f844017d82 Reviewed-on: http://gerrit.cloudera.org:8080/13070 Reviewed-by: Impala Public Jenkins Reviewed-by: Akshesh Doshi Tested-by: Impala Public Jenkins --- M infra/python/bootstrap_virtualenv.py M infra/python/deps/find_py26.py M infra/python/deps/pip_download.py 3 files changed, 25 insertions(+), 15 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified Akshesh Doshi: Looks good to me, but someone else must approve -- To view, visit http://gerrit.cloudera.org:8080/13070 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: If4285a021bb581f88425daa52ef8a3f844017d82 Gerrit-Change-Number: 13070 Gerrit-PatchSet: 5 Gerrit-Owner: Akshesh Doshi Gerrit-Reviewer: Akshesh Doshi Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] Make infra/python compatible with both Python 2 & 3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13070 ) Change subject: Make infra/python compatible with both Python 2 & 3 .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13070 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4285a021bb581f88425daa52ef8a3f844017d82 Gerrit-Change-Number: 13070 Gerrit-PatchSet: 4 Gerrit-Owner: Akshesh Doshi Gerrit-Reviewer: Akshesh Doshi Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 25 Apr 2019 20:43:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8444: Fix performance regression when building privilege name
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/13095 )
Change subject: IMPALA-8444: Fix performance regression when building privilege
name
..
Patch Set 8:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/13095/7/fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationPolicy.java
File
fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationPolicy.java:
http://gerrit.cloudera.org:8080/#/c/13095/7/fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationPolicy.java@66
PS7, Line 66: privileges.addAll(role.getPrivilegeNames());
: }
: }
> I think that this could be further simplified to privileges.addAll(role.get
Done
http://gerrit.cloudera.org:8080/#/c/13095/7/fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationPolicy.java@84
PS7, Line 84: }
: return privileges;
: }
> Same as line 66.
Done
http://gerrit.cloudera.org:8080/#/c/13095/2/fe/src/main/java/org/apache/impala/catalog/Principal.java
File fe/src/main/java/org/apache/impala/catalog/Principal.java:
http://gerrit.cloudera.org:8080/#/c/13095/2/fe/src/main/java/org/apache/impala/catalog/Principal.java@63
PS2, Line 63:*/
: public boolean addPrivilege(PrincipalPrivilege privilege) {
: return principalPrivileges_.add(privilege);
> > > if non-URI parts of the name are always lower case, then it
I decided to change the whole thing to make the cache stores the privilege name
in a case sensitive way as per your original suggestion and I do agree that
this is actually a better solution. Interestingly I found various bugs while
fixing this, which will be updated in the next patchset.
http://gerrit.cloudera.org:8080/#/c/13095/2/fe/src/main/java/org/apache/impala/catalog/Principal.java@89
PS2, Line 89:
> Don't we need to handle URIs case sensitively in this function? principalPr
Done.
http://gerrit.cloudera.org:8080/#/c/13095/2/fe/src/main/java/org/apache/impala/catalog/Principal.java@97
PS2, Line 97: return principalPrivileges_.remove(privilegeName);
: }
> Same as above.
Done
--
To view, visit http://gerrit.cloudera.org:8080/13095
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I942d9b55f07c8972f69e532567d9b7d80fceb6e5
Gerrit-Change-Number: 13095
Gerrit-PatchSet: 8
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Thu, 25 Apr 2019 20:28:11 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8119: document how to set heap size in docker
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13119 ) Change subject: IMPALA-8119: document how to set heap size in docker .. Patch Set 1: Code-Review+2 Good to get this cleaned up -- To view, visit http://gerrit.cloudera.org:8080/13119 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie6ddba3c42a698b52d7c4e2ff6a9c73068e198b2 Gerrit-Change-Number: 13119 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Thu, 25 Apr 2019 20:33:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/13005 )
Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive
3.1.0
..
Patch Set 8:
(25 comments)
http://gerrit.cloudera.org:8080/#/c/13005/8/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/13005/8/bin/impala-config.sh@196
PS8, Line 196: # When USE_CDP_HIVE is set we use the latest hive version
available to deply in minicluster
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/13005/8/bin/impala-config.sh@200
PS8, Line 200: # TODO(Vihang) we should repackage the tarballs so that the
src and binaries are extracted
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/13005/8/bin/impala-config.sh@209
PS8, Line 209: export
HIVE_HOME="$IMPALA_TOOLCHAIN/cdh_components-${CDH_BUILD_NUMBER}/hive-${MINICLUSTER_HIVE_VERSION}"
line too long (106 > 90)
http://gerrit.cloudera.org:8080/#/c/13005/8/bin/impala-config.sh@546
PS8, Line 546: export
HIVE_METASTORE_THRIFT_DIR=$CDP_COMPONENTS_HOME/apache-hive-${IMPALA_HIVE_VERSION}-src/standalone-metastore/src/main/thrift
line too long (129 > 90)
http://gerrit.cloudera.org:8080/#/c/13005/8/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/13005/8/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1073
PS8, Line 1073:
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/13005/8/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1169
PS8, Line 1169:
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/13005/8/testdata/bin/run-hive-server.sh
File testdata/bin/run-hive-server.sh:
http://gerrit.cloudera.org:8080/#/c/13005/8/testdata/bin/run-hive-server.sh@66
PS8, Line 66: export HIVE_METASTORE_HADOOP_OPTS="-verbose:class -Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=30010"
line too long (121 > 90)
http://gerrit.cloudera.org:8080/#/c/13005/8/tests/authorization/test_owner_privileges.py
File tests/authorization/test_owner_privileges.py:
http://gerrit.cloudera.org:8080/#/c/13005/8/tests/authorization/test_owner_privileges.py@103
PS8, Line 103: )
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/8/tests/authorization/test_owner_privileges.py@179
PS8, Line 179: =
flake8: E501 line too long (93 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/8/tests/authorization/test_owner_privileges.py@182
PS8, Line 182: =
flake8: E501 line too long (93 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/8/tests/authorization/test_owner_privileges.py@186
PS8, Line 186: )
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/8/tests/authorization/test_owner_privileges.py@193
PS8, Line 193: ,
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/8/tests/authorization/test_owner_privileges.py@213
PS8, Line 213: )
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/8/tests/authorization/test_owner_privileges.py@226
PS8, Line 226: )
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/8/tests/authorization/test_owner_privileges.py@236
PS8, Line 236: )
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/8/tests/authorization/test_owner_privileges.py@282
PS8, Line 282: _
flake8: E501 line too long (100 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/8/tests/authorization/test_owner_privileges.py@287
PS8, Line 287: t
flake8: E501 line too long (101 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/8/tests/authorization/test_owner_privileges.py@293
PS8, Line 293: 2
flake8: E501 line too long (92 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/8/tests/authorization/test_owner_privileges.py@355
PS8, Line 355: _
flake8: E501 line too long (100 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/8/tests/authorization/test_owner_privileges.py@360
PS8, Line 360: t
flake8: E501 line too long (101 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/8/tests/authorization/test_owner_privileges.py@364
PS8, Line 364: 2
flake8: E501 line too long (92 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/8/tests/authorization/test_owner_privileges.py@368
PS8, Line 368: )
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/8/tests/authorization/test_owner_privileges.py@412
PS8, Line 412: t
flake8: E501 line too long (101 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/8/tests/authorization/test_owner_privileges.py@421
PS8, Line 421: i
flake8: E501 line too long (102 > 90
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Vihang Karajgaonkar has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 This change upgrades the hive dependencies of Impala to use Hive 3.1.0 based binaries. Most of the changes in this patch are based off patches provided by Todd (links available in JIRA). Upgrading the dependencies allows us to work with both Hive 3.1.0 and Hive 2.1.0 in the same code line. In order to do this, the patch trims down a lot of unnecessary hive dependencies of the front end code by creating a shaded-deps module. The pom.xml of shaded-deps includes only the files from Hive source which Impala depends for compilation. Additionally, it also uses a custom build of Hive which is based of Hive 3.1.0. This custom build includes patches for HIVE-21596 and HIVE-21586 which are needed by Impala so that it can compile against Hive-3 libraries and be able to talk to HMS-2.x metastore. Once these patches are merged we can get rid of this custom build and rely on more official sources of Hive builds. The patch also changes impala-config.sh so that it always downloads the Hive-3 libraries from the toolchain. The code is always built using Hive-3 jars. However, based on the value of USE_CDP_HIVE, the minicluster is deployed using Hive-3 or Hive-2 binaries. Since Impala implements HiveServer2's TCLIService.thrift interface, it requires us to use the existing mechanism of copying the hive-2/api TCLIService.thrift to hive-3/api. It also adds a few environment variables which point to the metastore's thrift file and the CDH Hive version. Testing: 1. Code compiles and runs against both HMS-3 and HMS-2 2. Ran full-suite of tests using the private jenkins job against HMS-2 3. Running full-tests against HMS-3 will need more work like supporting Tez in the mini-cluster (for dataloading) and HMS transaction support since HMS3 create transactional tables by default. This will be taken up in subsequent change. Notes: 1. Patch uses a custom build of Hive to be deployed in mini-cluster. This build has the fixes for HIVE-21596 2. The Hive 3.1 does not include some of the patches like HIVE-21077. Some of the recent code in Impala depends on that patch to exist in hive. We will need to port HIVE-21077 to Hive 3.1.0 in order to compile against Hive 3.1.0 jars. I will continue working on getting this merged along with HIVE-21586 3. Some of the existing tests rely on the fact the UDFs implement the UDF interface in Hive (UDFLength, UDFHour, UDFYear). These built-in hive functions have been moved to use GenericUDF interface in Hive 3. Impala currently only supports UDFExecutor. In order to have a full compatibility with all the functions in Hive 2.x we should support GenericUDFs too. That would be taken up as a separate patch. 4. The Sentry object ownership tests are flaky. There are race-conditions in that code which get exposed for some by this patch. For example, when a database is created, the object privileges are immediately updated in the catalog cache. If Sentry has not been updated and there is a refresh authorization call in between it can clear the privilege from the cache causing the subsequent statement using that privilege to fail. The patch adds sleep statements in the test to work-around these issues. Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 --- M CMakeLists.txt M README.md M bin/bootstrap_toolchain.py M bin/impala-config.sh M bin/set-classpath.sh M common/thrift/.gitignore M common/thrift/CMakeLists.txt M fe/CMakeLists.txt M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java M fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java M fe/src/main/java/org/apache/impala/catalog/TableLoader.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java M fe/src/main/java/org/apache/impala/service/MetadataOp.java A fe/src/main/java/org/apache/impala/util/MetadataFormatUtils.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java M fe/src/test/java/org/apache/impala/testutil/EmbeddedMetastoreClientPool.java M impala-parent/pom.xml A shaded-deps/.gitignore A
[Impala-ASF-CR] IMPALA-8419 : Validate event processing related configurations
Bharath Krishna has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/13019 ) Change subject: IMPALA-8419 : Validate event processing related configurations .. IMPALA-8419 : Validate event processing related configurations Using the Metastore API to get the configuration values, verify that the configurations needed for event processing are set correctly. Also check that the parameters required for event processing is not filtered out by the Hive config METASTORE_PARAMETER_EXCLUDE_PATTERNS. This validation is done while creating the event processor and throws CatalogException if the configuration is incorrect. Testing - Added unit tests Change-Id: I94c2783e36287a65122003aa55d8075a806bc606 --- A fe/src/main/java/org/apache/impala/catalog/events/EventProcessorValidation.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/java/org/apache/impala/catalog/events/SynchronousHMSEventProcessorForTests.java 7 files changed, 493 insertions(+), 59 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/13019/7 -- To view, visit http://gerrit.cloudera.org:8080/13019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I94c2783e36287a65122003aa55d8075a806bc606 Gerrit-Change-Number: 13019 Gerrit-PatchSet: 7 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8444: Fix performance regression when building privilege name
Fredy Wijaya has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/13095 ) Change subject: IMPALA-8444: Fix performance regression when building privilege name .. IMPALA-8444: Fix performance regression when building privilege name This patch fixes the performance regression when building privilege name by rewriting PrincipalPrivilege.buildPrivilegeName() with a simple string concatentation instead of using a list that gets converted into a string. Below is the result of running a benchmark using JMH comparing the old and new implementations: Result "org.apache.impala.BuildPrivilegeNameBenchmark.fast": 0.344 ±(99.9%) 0.004 us/op [Average] (min, avg, max) = (0.336, 0.344, 0.355), stdev = 0.005 CI (99.9%): [0.339, 0.348] (assumes normal distribution) Result "org.apache.impala.BuildPrivilegeNameBenchmark.slow": 0.831 ±(99.9%) 0.011 us/op [Average] (min, avg, max) = (0.807, 0.831, 0.856), stdev = 0.015 CI (99.9%): [0.820, 0.842] (assumes normal distribution) Benchmark Mode Cnt Score Error Units BuildPrivilegeNameBenchmark.fast avgt 25 0.344 ± 0.004 us/op BuildPrivilegeNameBenchmark.slow avgt 25 0.831 ± 0.011 us/op This patch also updates SentryAuthorizationPolicy.listPrivileges() to reuse the privilege names that have already been built instead of building them again. While fixing this, I found a bug where Principal stores the PrincipalPrivilege in case insensitive way. This is true for all privilege scopes, except URI. This patch fixes the issue by storing URI privilege in a case sensitive manner. This patch removes incorrect synchronization in SentryAuthorizationPolicy.listPrivileges() that can cause the operation to run in serial in a highly concurrent workload. Testing: - Ran all FE tests - Ran all E2E authorization tests - Added E2E test for URI privilege bug Change-Id: I942d9b55f07c8972f69e532567d9b7d80fceb6e5 --- M fe/src/main/java/org/apache/impala/authorization/AuthorizationPolicy.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationPolicy.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryProxy.java M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogObjectCache.java M fe/src/main/java/org/apache/impala/catalog/Principal.java M fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilege.java M tests/authorization/test_grant_revoke.py 8 files changed, 128 insertions(+), 126 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/13095/8 -- To view, visit http://gerrit.cloudera.org:8080/13095 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I942d9b55f07c8972f69e532567d9b7d80fceb6e5 Gerrit-Change-Number: 13095 Gerrit-PatchSet: 8 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8444: Fix performance regression when building privilege name
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13095 ) Change subject: IMPALA-8444: Fix performance regression when building privilege name .. Patch Set 8: Csaba, this is ready for review. -- To view, visit http://gerrit.cloudera.org:8080/13095 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I942d9b55f07c8972f69e532567d9b7d80fceb6e5 Gerrit-Change-Number: 13095 Gerrit-PatchSet: 8 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 25 Apr 2019 20:28:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/13005 )
Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive
3.1.0
..
Patch Set 7:
(25 comments)
http://gerrit.cloudera.org:8080/#/c/13005/7/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/13005/7/bin/impala-config.sh@196
PS7, Line 196: # When USE_CDP_HIVE is set we use the latest hive version
available to deply in minicluster
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/13005/7/bin/impala-config.sh@200
PS7, Line 200: # TODO(Vihang) we should repackage the tarballs so that the
src and binaries are extracted
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/13005/7/bin/impala-config.sh@209
PS7, Line 209: export
HIVE_HOME="$IMPALA_TOOLCHAIN/cdh_components-${CDH_BUILD_NUMBER}/hive-${MINICLUSTER_HIVE_VERSION}"
line too long (106 > 90)
http://gerrit.cloudera.org:8080/#/c/13005/7/bin/impala-config.sh@546
PS7, Line 546: export
HIVE_METASTORE_THRIFT_DIR=$CDP_COMPONENTS_HOME/apache-hive-${IMPALA_HIVE_VERSION}-src/standalone-metastore/src/main/thrift
line too long (129 > 90)
http://gerrit.cloudera.org:8080/#/c/13005/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/13005/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1073
PS7, Line 1073:
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/13005/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1169
PS7, Line 1169:
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/13005/7/testdata/bin/run-hive-server.sh
File testdata/bin/run-hive-server.sh:
http://gerrit.cloudera.org:8080/#/c/13005/7/testdata/bin/run-hive-server.sh@66
PS7, Line 66: export HIVE_METASTORE_HADOOP_OPTS="-verbose:class -Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=30010"
line too long (121 > 90)
http://gerrit.cloudera.org:8080/#/c/13005/7/tests/authorization/test_owner_privileges.py
File tests/authorization/test_owner_privileges.py:
http://gerrit.cloudera.org:8080/#/c/13005/7/tests/authorization/test_owner_privileges.py@103
PS7, Line 103: )
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/7/tests/authorization/test_owner_privileges.py@179
PS7, Line 179: =
flake8: E501 line too long (93 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/7/tests/authorization/test_owner_privileges.py@182
PS7, Line 182: =
flake8: E501 line too long (93 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/7/tests/authorization/test_owner_privileges.py@186
PS7, Line 186: )
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/7/tests/authorization/test_owner_privileges.py@193
PS7, Line 193: ,
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/7/tests/authorization/test_owner_privileges.py@213
PS7, Line 213: )
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/7/tests/authorization/test_owner_privileges.py@226
PS7, Line 226: )
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/7/tests/authorization/test_owner_privileges.py@236
PS7, Line 236: )
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/7/tests/authorization/test_owner_privileges.py@282
PS7, Line 282: _
flake8: E501 line too long (100 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/7/tests/authorization/test_owner_privileges.py@287
PS7, Line 287: t
flake8: E501 line too long (101 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/7/tests/authorization/test_owner_privileges.py@293
PS7, Line 293: 2
flake8: E501 line too long (92 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/7/tests/authorization/test_owner_privileges.py@355
PS7, Line 355: _
flake8: E501 line too long (100 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/7/tests/authorization/test_owner_privileges.py@360
PS7, Line 360: t
flake8: E501 line too long (101 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/7/tests/authorization/test_owner_privileges.py@364
PS7, Line 364: 2
flake8: E501 line too long (92 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/7/tests/authorization/test_owner_privileges.py@368
PS7, Line 368: )
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/7/tests/authorization/test_owner_privileges.py@412
PS7, Line 412: t
flake8: E501 line too long (101 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/7/tests/authorization/test_owner_privileges.py@421
PS7, Line 421: i
flake8: E501 line too long (102 > 90
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Vihang Karajgaonkar has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 This change upgrades the hive dependencies of Impala to use Hive 3.1.0 based binaries. Most of the changes in this patch are based off patches provided by Todd (links available in JIRA). Upgrading the dependencies allows us to work with both Hive 3.1.0 and Hive 2.1.0 in the same code line. In order to do this, the patch trims down a lot of unnecessary hive dependencies of the front end code by creating a shaded-deps module. The pom.xml of shaded-deps includes only the files from Hive source which Impala depends for compilation. Additionally, it also uses a custom build of Hive which is based of Hive 3.1.0. This custom build includes patches for HIVE-21596 and HIVE-21586 which are needed by Impala so that it can compile against Hive-3 libraries and be able to talk to HMS-2.x metastore. Once these patches are merged we can get rid of this custom build and rely on more official sources of Hive builds. The patch also changes impala-config.sh so that it always downloads the Hive-3 libraries from the toolchain. The code is always built using Hive-3 jars. However, based on the value of USE_CDP_HIVE, the minicluster is deployed using Hive-3 or Hive-2 binaries. Since Impala implements HiveServer2's TCLIService.thrift interface, it requires us to use the existing mechanism of copying the hive-2/api TCLIService.thrift to hive-3/api. It also adds a few environment variables which point to the metastore's thrift file and the CDH Hive version. Testing: 1. Code compiles and runs against both HMS-3 and HMS-2 2. Ran full-suite of tests using the private jenkins job against HMS-2 3. Running full-tests against HMS-3 will need more work like supporting Tez in the mini-cluster (for dataloading) and HMS transaction support since HMS3 create transactional tables by default. Notes: 1. Patch uses a custom build of Hive to be deployed in mini-cluster. This build has the fixes for HIVE-21596 2. The Hive 3.1 does not include some of the patches like HIVE-21077. Some of the recent code in Impala depends on that patch to exist in hive. We will need to port HIVE-21077 to Hive 3.1.0 in order to compile against Hive 3.1.0 jars. I will continue working on getting this merged along with HIVE-21586 Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 --- M CMakeLists.txt M README.md M bin/bootstrap_toolchain.py M bin/impala-config.sh M bin/set-classpath.sh M common/thrift/.gitignore M common/thrift/CMakeLists.txt M fe/CMakeLists.txt M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java M fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java M fe/src/main/java/org/apache/impala/catalog/TableLoader.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java M fe/src/main/java/org/apache/impala/service/MetadataOp.java A fe/src/main/java/org/apache/impala/util/MetadataFormatUtils.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java M fe/src/test/java/org/apache/impala/testutil/EmbeddedMetastoreClientPool.java M impala-parent/pom.xml A shaded-deps/.gitignore A shaded-deps/CMakeLists.txt A shaded-deps/pom.xml M testdata/bin/run-hive-server.sh M tests/authorization/test_owner_privileges.py M tests/common/sentry_cache_test_suite.py M tests/custom_cluster/test_permanent_udfs.py 34 files changed, 1,208 insertions(+), 294 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/13005/7 -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 7 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan
[Impala-ASF-CR] IMPALA-8419 : Validate event processing related configurations
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13019 ) Change subject: IMPALA-8419 : Validate event processing related configurations .. Patch Set 7: (12 comments) Overall looks good. Few comments below and then good to go from my side. http://gerrit.cloudera.org:8080/#/c/13019/7/fe/src/main/java/org/apache/impala/catalog/events/EventProcessorValidation.java File fe/src/main/java/org/apache/impala/catalog/events/EventProcessorValidation.java: http://gerrit.cloudera.org:8080/#/c/13019/7/fe/src/main/java/org/apache/impala/catalog/events/EventProcessorValidation.java@42 PS7, Line 42: EventProcessorValidation a better name could be EventProcessorConfigValidator http://gerrit.cloudera.org:8080/#/c/13019/7/fe/src/main/java/org/apache/impala/catalog/events/EventProcessorValidation.java@42 PS7, Line 42: interface general convention in Impala is not to use package-private classes (although I think its a good idea to have them) For consistency, please change these to public. Same with the members of this class. Try to keep the members private unless its not possible. http://gerrit.cloudera.org:8080/#/c/13019/7/fe/src/main/java/org/apache/impala/catalog/events/EventProcessorValidation.java@59 PS7, Line 59: getExpectedValue method doc http://gerrit.cloudera.org:8080/#/c/13019/7/fe/src/main/java/org/apache/impala/catalog/events/EventProcessorValidation.java@74 PS7, Line 74: String private static final http://gerrit.cloudera.org:8080/#/c/13019/7/fe/src/main/java/org/apache/impala/catalog/events/EventProcessorValidation.java@115 PS7, Line 115: MetastoreEventParameters nit, formatting is off http://gerrit.cloudera.org:8080/#/c/13019/7/fe/src/main/java/org/apache/impala/catalog/events/EventProcessorValidation.java@120 PS7, Line 120: filtered s/filtered/filtered out/ http://gerrit.cloudera.org:8080/#/c/13019/7/fe/src/main/java/org/apache/impala/catalog/events/EventProcessorValidation.java@157 PS7, Line 157: Found would be good to provided the expected value here as well http://gerrit.cloudera.org:8080/#/c/13019/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/13019/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@68 PS7, Line 68: MetastoreEventParameters using parameters and properties interchangebly is confusing. may be rename this to MetastoreEventPropertyKey http://gerrit.cloudera.org:8080/#/c/13019/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@70 PS7, Line 70: CATALOG_VERSION_PROP_KEY _PROP_KEY is redundant if you name the enum as MetastoreEventPropertyKey. SAme for others. http://gerrit.cloudera.org:8080/#/c/13019/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java: http://gerrit.cloudera.org:8080/#/c/13019/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@253 PS7, Line 253: Runtime error.. redundant to have this text here. can be removed. http://gerrit.cloudera.org:8080/#/c/13019/7/fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java File fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java: http://gerrit.cloudera.org:8080/#/c/13019/7/fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java@88 PS7, Line 88: DEFAULT_METASTORE_CONFIG_VALUE I am not convinced of the value of having this. Why can't we just use empty string which could be private to the config validator? http://gerrit.cloudera.org:8080/#/c/13019/7/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/13019/7/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@336 PS7, Line 336: impala add a test case for ^impala as well since it matches the disableHmsSync paramter -- To view, visit http://gerrit.cloudera.org:8080/13019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I94c2783e36287a65122003aa55d8075a806bc606 Gerrit-Change-Number: 13019 Gerrit-PatchSet: 7 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 25 Apr 2019 20:23:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8419 : Validate event processing related configurations
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13019 ) Change subject: IMPALA-8419 : Validate event processing related configurations .. Patch Set 7: (9 comments) http://gerrit.cloudera.org:8080/#/c/13019/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/13019/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@73 PS7, Line 73: // flag to be set in the table/database parameters to disable event based metadata sync line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/13019/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java: http://gerrit.cloudera.org:8080/#/c/13019/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@20 PS7, Line 20: import static org.apache.impala.catalog.events.EventProcessorValidation.hasValidMetastoreConfigs; line too long (97 > 90) http://gerrit.cloudera.org:8080/#/c/13019/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@21 PS7, Line 21: import static org.apache.impala.catalog.events.EventProcessorValidation.verifyParametersNotFiltered; line too long (100 > 90) http://gerrit.cloudera.org:8080/#/c/13019/7/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/13019/7/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@21 PS7, Line 21: import static org.apache.impala.catalog.events.EventProcessorValidation.validateMetastoreConfigs; line too long (97 > 90) http://gerrit.cloudera.org:8080/#/c/13019/7/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@22 PS7, Line 22: import static org.apache.impala.catalog.events.EventProcessorValidation.validateMetastoreEventParameters; line too long (105 > 90) http://gerrit.cloudera.org:8080/#/c/13019/7/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@71 PS7, Line 71: import org.apache.impala.catalog.events.EventProcessorValidation.MetastoreEventConfigsToValidate; line too long (97 > 90) http://gerrit.cloudera.org:8080/#/c/13019/7/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@1100 PS7, Line 1100: dbParams.put(MetastoreEventParameters.DISABLE_EVENT_HMS_SYNC_KEY.getKey(), dbFlag); line too long (93 > 90) http://gerrit.cloudera.org:8080/#/c/13019/7/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@1114 PS7, Line 1114: MetastoreEventParameters.DISABLE_EVENT_HMS_SYNC_KEY.getKey(), tblTransition.second); line too long (96 > 90) http://gerrit.cloudera.org:8080/#/c/13019/7/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@1621 PS7, Line 1621: alteredDb.putToParameters(MetastoreEventParameters.DISABLE_EVENT_HMS_SYNC_KEY.getKey(), line too long (91 > 90) -- To view, visit http://gerrit.cloudera.org:8080/13019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I94c2783e36287a65122003aa55d8075a806bc606 Gerrit-Change-Number: 13019 Gerrit-PatchSet: 7 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 25 Apr 2019 20:22:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8454 (part 1): Refactor file descriptor loading code
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12950 ) Change subject: IMPALA-8454 (part 1): Refactor file descriptor loading code .. Patch Set 5: Code-Review+2 (2 comments) http://gerrit.cloudera.org:8080/#/c/12950/5/fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java File fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java: http://gerrit.cloudera.org:8080/#/c/12950/5/fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java@86 PS5, Line 86: "); nit: worth including the partition name? (same with other logs) http://gerrit.cloudera.org:8080/#/c/12950/4/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: http://gerrit.cloudera.org:8080/#/c/12950/4/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@545 PS4, Line 545:* @param isRefresh whether this is a refresh operation or an initial load. This only > The problem with that is that the FileMetadataLoader.load() logging is only cool -- To view, visit http://gerrit.cloudera.org:8080/12950 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I59edf493b9ba38be5f556b4795a7684d9c9e3a07 Gerrit-Change-Number: 12950 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Thu, 25 Apr 2019 20:02:12 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 18: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2917/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 18 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Thu, 25 Apr 2019 19:40:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 19: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4068/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 19 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Thu, 25 Apr 2019 18:59:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 19: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 19 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Thu, 25 Apr 2019 18:59:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 18: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 18 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Thu, 25 Apr 2019 18:59:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Austin Nobis has uploaded a new patch set (#18) to the change originally created by radford nguyen. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. IMPALA-8309: add user authorization_provider flag This commit adds a `authorization_provider` user-facing flag in order to provide a more human-readable alternative to the `authorization_factory_class` for internally-provided authorization strategies. The `authorization_factory_class` flag is retained, but no longer takes a default value if not specified. The default for `authorization_provider` is "sentry" in order to retain backwards-compatibility. If specified, `authorization_factory_class` will take precedence. Testing: - Manually started minicluster with each of following flags and verified correct authorization strategy chosen: - provider='' factory='' => sentry - provider=sentry factory='' => sentry - provider=ranger factory='' => ranger - provider='' factory=sentry => sentry - provider='' factory=ranger => ranger - provider=sentry factory=sentry => sentry - provider=ranger factory=sentry => sentry - provider=sentry factory=ranger => ranger - provider=ranger factory=ranger => ranger - Wrote unit tests to capture above assertions - Ran fe unit and e2e tests - Wrote e2e test to verify new flag behavior Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 --- M be/src/service/frontend.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/authorization/AuthorizationConfig.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationConfig.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationConfig.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A fe/src/main/java/org/apache/impala/util/AuthorizationUtil.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java A fe/src/test/java/org/apache/impala/util/AuthorizationUtilTest.java A tests/authorization/test_provider.py M tests/authorization/test_ranger.py 16 files changed, 338 insertions(+), 62 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/12901/18 -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 18 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-2990: timeout unresponsive queries in coordinator
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/12299 ) Change subject: IMPALA-2990: timeout unresponsive queries in coordinator .. Patch Set 9: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/12299 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I196c8c6a5633b1960e2c3a3884777be9b3824987 Gerrit-Change-Number: 12299 Gerrit-PatchSet: 9 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 25 Apr 2019 18:43:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8341: Data cache for remote reads
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12987 ) Change subject: IMPALA-8341: Data cache for remote reads .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/12987/4/be/src/runtime/io/data-cache.h File be/src/runtime/io/data-cache.h: http://gerrit.cloudera.org:8080/#/c/12987/4/be/src/runtime/io/data-cache.h@88 PS4, Line 88: /// - bound number of backing files per partition by consolidating the content of very > I think with our current assumption of 4TB of logical space per file, if yo Oh I didn't realise the files were that large, ok, that makes sense then. -- To view, visit http://gerrit.cloudera.org:8080/12987 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I734803c1c1787c858dc3ffa0a2c0e33e77b12edc Gerrit-Change-Number: 12987 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Ho Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 25 Apr 2019 18:10:57 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8119: document how to set heap size in docker
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13119 ) Change subject: IMPALA-8119: document how to set heap size in docker .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2916/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13119 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie6ddba3c42a698b52d7c4e2ff6a9c73068e198b2 Gerrit-Change-Number: 13119 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Thu, 25 Apr 2019 18:15:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. Patch Set 6: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/2915/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 6 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 25 Apr 2019 18:14:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8341: Data cache for remote reads
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/12987 ) Change subject: IMPALA-8341: Data cache for remote reads .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/12987/4/be/src/runtime/io/data-cache.h File be/src/runtime/io/data-cache.h: http://gerrit.cloudera.org:8080/#/c/12987/4/be/src/runtime/io/data-cache.h@88 PS4, Line 88: /// - bound number of backing files per partition by consolidating the content of very > Yeah this scenario is a bit concerning for me still since it's conceivable I think with our current assumption of 4TB of logical space per file, if you assume you're writing a relatively aggressive estimate of 100MB/sec to the cache in a super heavy workload, then you'll only need to roll to a new file every 11 hours. So, 1000 file descriptors will last you over a year. Given that fd counts can be set into the 100k range without any real issues, I don't think this is going to be too problematic under these assumptions. If we find that we need to support a smaller "rolling" interval than 4TB for some reason, we'll definitely need to address it, but it seems like a lot of complexity to take for now. -- To view, visit http://gerrit.cloudera.org:8080/12987 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I734803c1c1787c858dc3ffa0a2c0e33e77b12edc Gerrit-Change-Number: 12987 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Ho Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 25 Apr 2019 18:05:56 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8119: document how to set heap size in docker
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13119 Change subject: IMPALA-8119: document how to set heap size in docker .. IMPALA-8119: document how to set heap size in docker JAVA_TOOL_OPTIONS is a standard mechanism to pass arguments to a JVM. Let's just document this as the canonical way to pass in the heap size. Change-Id: Ie6ddba3c42a698b52d7c4e2ff6a9c73068e198b2 --- M bin/start-impala-cluster.py M docker/README.md M docker/daemon_entrypoint.sh 3 files changed, 9 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/13119/1 -- To view, visit http://gerrit.cloudera.org:8080/13119 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ie6ddba3c42a698b52d7c4e2ff6a9c73068e198b2 Gerrit-Change-Number: 13119 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] Configure Hive 3's HS2 to execute queries using Tez local mode
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/12931 ) Change subject: Configure Hive 3's HS2 to execute queries using Tez local mode .. Patch Set 5: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/12931 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I76e47fbd1d6ff5103d81a8de430d5465dba284cd Gerrit-Change-Number: 12931 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 25 Apr 2019 17:34:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Vihang Karajgaonkar has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 This change upgrades the hive dependencies of Impala to use Hive 3.1.0 based binaries. Most of the changes in this patch are based off patches provided by Todd (links available in JIRA). Upgrading the dependencies allows us to work with both Hive 3.1.0 and Hive 2.1.0 in the same code line. In order to do this, the patch trims down a lot of unnecessary hive dependencies of the front end code by creating a shaded-deps module. The pom.xml of shaded-deps includes only the files from Hive source which Impala depends for compilation. Additionally, it also uses a custom build of Hive which is based of Hive 3.1.0. This custom build includes patches for HIVE-21596 and HIVE-21586 which are needed by Impala so that it can compile against Hive-3 libraries and be able to talk to HMS-2.x metastore. Once these patches are merged we can get rid of this custom build and rely on more official sources of Hive builds. The patch also changes impala-config.sh so that it always downloads the Hive-3 libraries from the toolchain. The code is always built using Hive-3 jars. However, based on the value of USE_CDP_HIVE, the minicluster is deployed using Hive-3 or Hive-2 binaries. Since Impala implements HiveServer2's TCLIService.thrift interface, it requires us to use the existing mechanism of copying the hive-2/api TCLIService.thrift to hive-3/api. It also adds a few environment variables which point to the metastore's thrift file and the CDH Hive version. Testing: 1. Code compiles and runs against both HMS-3 and HMS-2 2. Ran full-suite of tests using the private jenkins job against HMS-2 3. Running full-tests against HMS-3 will need more work like supporting Tez in the mini-cluster (for dataloading) and HMS transaction support since HMS3 create transactional tables by default. Notes: 1. Patch uses a custom build of Hive to be deployed in mini-cluster. This build has the fixes for HIVE-21596 2. The Hive 3.1 does not include some of the patches like HIVE-21077. Some of the recent code in Impala depends on that patch to exist in hive. We will need to port HIVE-21077 to Hive 3.1.0 in order to compile against Hive 3.1.0 jars. I will continue working on getting this merged along with HIVE-21586 Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 --- M CMakeLists.txt M README.md M bin/bootstrap_toolchain.py M bin/impala-config.sh M bin/set-classpath.sh M common/thrift/.gitignore M common/thrift/CMakeLists.txt M fe/CMakeLists.txt M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java M fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java M fe/src/main/java/org/apache/impala/catalog/TableLoader.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java M fe/src/main/java/org/apache/impala/service/MetadataOp.java A fe/src/main/java/org/apache/impala/util/MetadataFormatUtils.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java M fe/src/test/java/org/apache/impala/testutil/EmbeddedMetastoreClientPool.java M impala-parent/pom.xml A shaded-deps/.gitignore A shaded-deps/CMakeLists.txt A shaded-deps/dependency-reduced-pom.xml A shaded-deps/pom.xml M testdata/bin/run-hive-server.sh M tests/authorization/test_owner_privileges.py M tests/common/sentry_cache_test_suite.py M tests/custom_cluster/test_permanent_udfs.py 35 files changed, 1,275 insertions(+), 292 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/13005/6 -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 6 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/13005 )
Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive
3.1.0
..
Patch Set 6:
(25 comments)
http://gerrit.cloudera.org:8080/#/c/13005/6/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/13005/6/bin/impala-config.sh@196
PS6, Line 196: # When USE_CDP_HIVE is set we use the latest hive version
available to deply in minicluster
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/13005/6/bin/impala-config.sh@200
PS6, Line 200: # TODO(Vihang) we should repackage the tarballs so that the
src and binaries are extracted
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/13005/6/bin/impala-config.sh@209
PS6, Line 209: export
HIVE_HOME="$IMPALA_TOOLCHAIN/cdh_components-${CDH_BUILD_NUMBER}/hive-${MINICLUSTER_HIVE_VERSION}"
line too long (106 > 90)
http://gerrit.cloudera.org:8080/#/c/13005/6/bin/impala-config.sh@546
PS6, Line 546: export
HIVE_METASTORE_THRIFT_DIR=$CDP_COMPONENTS_HOME/apache-hive-${IMPALA_HIVE_VERSION}-src/standalone-metastore/src/main/thrift
line too long (129 > 90)
http://gerrit.cloudera.org:8080/#/c/13005/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/13005/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1073
PS6, Line 1073:
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/13005/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1169
PS6, Line 1169:
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/13005/6/testdata/bin/run-hive-server.sh
File testdata/bin/run-hive-server.sh:
http://gerrit.cloudera.org:8080/#/c/13005/6/testdata/bin/run-hive-server.sh@66
PS6, Line 66: export HIVE_METASTORE_HADOOP_OPTS="-verbose:class -Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=30010"
line too long (121 > 90)
http://gerrit.cloudera.org:8080/#/c/13005/6/tests/authorization/test_owner_privileges.py
File tests/authorization/test_owner_privileges.py:
http://gerrit.cloudera.org:8080/#/c/13005/6/tests/authorization/test_owner_privileges.py@103
PS6, Line 103: )
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/6/tests/authorization/test_owner_privileges.py@179
PS6, Line 179: =
flake8: E501 line too long (93 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/6/tests/authorization/test_owner_privileges.py@182
PS6, Line 182: =
flake8: E501 line too long (93 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/6/tests/authorization/test_owner_privileges.py@186
PS6, Line 186: )
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/6/tests/authorization/test_owner_privileges.py@193
PS6, Line 193: ,
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/6/tests/authorization/test_owner_privileges.py@213
PS6, Line 213: )
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/6/tests/authorization/test_owner_privileges.py@226
PS6, Line 226: )
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/6/tests/authorization/test_owner_privileges.py@236
PS6, Line 236: )
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/6/tests/authorization/test_owner_privileges.py@282
PS6, Line 282: _
flake8: E501 line too long (100 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/6/tests/authorization/test_owner_privileges.py@287
PS6, Line 287: t
flake8: E501 line too long (101 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/6/tests/authorization/test_owner_privileges.py@293
PS6, Line 293: 2
flake8: E501 line too long (92 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/6/tests/authorization/test_owner_privileges.py@355
PS6, Line 355: _
flake8: E501 line too long (100 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/6/tests/authorization/test_owner_privileges.py@360
PS6, Line 360: t
flake8: E501 line too long (101 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/6/tests/authorization/test_owner_privileges.py@364
PS6, Line 364: 2
flake8: E501 line too long (92 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/6/tests/authorization/test_owner_privileges.py@368
PS6, Line 368: )
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/6/tests/authorization/test_owner_privileges.py@412
PS6, Line 412: t
flake8: E501 line too long (101 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/13005/6/tests/authorization/test_owner_privileges.py@421
PS6, Line 421: i
flake8: E501 line too long (102 > 90
[Impala-ASF-CR] Remove hive-benchmark test workload
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13117 ) Change subject: Remove hive-benchmark test workload .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4067/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13117 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I383638ef4acce59792fee287da0016b3d949e693 Gerrit-Change-Number: 13117 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 25 Apr 2019 17:13:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8381: Optimize ParquetPlainEncoder::DecodeBatch() for simple types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12985 ) Change subject: IMPALA-8381: Optimize ParquetPlainEncoder::DecodeBatch() for simple types .. Patch Set 15: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2914/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12985 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I57b7d2573bb6dfd038e581acb3bd8ea1565aa20d Gerrit-Change-Number: 12985 Gerrit-PatchSet: 15 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 25 Apr 2019 17:19:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8381: Optimize ParquetPlainEncoder::DecodeBatch() for simple types
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/12985 ) Change subject: IMPALA-8381: Optimize ParquetPlainEncoder::DecodeBatch() for simple types .. Patch Set 15: Code-Review+1 (2 comments) http://gerrit.cloudera.org:8080/#/c/12985/15//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/12985/15//COMMIT_MSG@32 PS15, Line 32: The overall performance increase is up to 2x for small strides (8 bytes, : INT32) but decreases as the stride increases, and disappears from around : 40 bytes. With bigger strides, there is no performance difference from : the previous implementation. Please mention that the benchmarks are added in another patch/JIRA. http://gerrit.cloudera.org:8080/#/c/12985/14/be/src/exec/parquet/parquet-common.h File be/src/exec/parquet/parquet-common.h: http://gerrit.cloudera.org:8080/#/c/12985/14/be/src/exec/parquet/parquet-common.h@210 PS14, Line 210: s fun > DecodeByParquetType should actually never call it, but it cannot be determi No, I think that we shouldn't check in release, DCHECK's are enough. The compatibility of Parquet and SQL metadata is checked earlier during the initialization of the column readers. -- To view, visit http://gerrit.cloudera.org:8080/12985 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I57b7d2573bb6dfd038e581acb3bd8ea1565aa20d Gerrit-Change-Number: 12985 Gerrit-PatchSet: 15 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 25 Apr 2019 17:18:21 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Remove hive-benchmark test workload
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13117 ) Change subject: Remove hive-benchmark test workload .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2913/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13117 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I383638ef4acce59792fee287da0016b3d949e693 Gerrit-Change-Number: 13117 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 25 Apr 2019 17:11:56 + Gerrit-HasComments: No
[Impala-ASF-CR] Make infra/python compatible with both Python 2 & 3
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13070 ) Change subject: Make infra/python compatible with both Python 2 & 3 .. Patch Set 4: The job I started - https://jenkins.impala.io/job/gerrit-verify-dryrun/4065/ - will automatically merge it if/when the tests pass. It takes a few hours. -- To view, visit http://gerrit.cloudera.org:8080/13070 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4285a021bb581f88425daa52ef8a3f844017d82 Gerrit-Change-Number: 13070 Gerrit-PatchSet: 4 Gerrit-Owner: Akshesh Doshi Gerrit-Reviewer: Akshesh Doshi Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 25 Apr 2019 17:04:27 + Gerrit-HasComments: No
[Impala-ASF-CR] Remove hive-benchmark test workload
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13117 ) Change subject: Remove hive-benchmark test workload .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13117 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I383638ef4acce59792fee287da0016b3d949e693 Gerrit-Change-Number: 13117 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 25 Apr 2019 17:02:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8454 (part 1): Refactor file descriptor loading code
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12950 ) Change subject: IMPALA-8454 (part 1): Refactor file descriptor loading code .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4066/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/12950 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I59edf493b9ba38be5f556b4795a7684d9c9e3a07 Gerrit-Change-Number: 12950 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Thu, 25 Apr 2019 16:58:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8381: Optimize ParquetPlainEncoder::DecodeBatch() for simple types
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/12985 ) Change subject: IMPALA-8381: Optimize ParquetPlainEncoder::DecodeBatch() for simple types .. Patch Set 15: (4 comments) http://gerrit.cloudera.org:8080/#/c/12985/14//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/12985/14//COMMIT_MSG@33 PS14, Line 33: and disappears from around : 40 bytes. > Is there a performance decrease after some point? It's worth to state if th Done. http://gerrit.cloudera.org:8080/#/c/12985/14/be/src/exec/parquet/parquet-common.h File be/src/exec/parquet/parquet-common.h: http://gerrit.cloudera.org:8080/#/c/12985/14/be/src/exec/parquet/parquet-common.h@95 PS14, Line 95: EncodedByteSize(const > nit: maybe 'EncodedByteSize()', or 'EncodedSizeOf()'? Done http://gerrit.cloudera.org:8080/#/c/12985/14/be/src/exec/parquet/parquet-common.h@210 PS14, Line 210: s fun > But if it calls it then we would hit this DCHECK immediately, right? There DecodeByParquetType should actually never call it, but it cannot be determined at compile time. It it templated on `InternamType`, the parquet type is a value parameter, and the switch goes through every parquet type and calls Decode with `InternalType` and the parquet type as template parameters. Therefore the compiler has to generate code for every combination, not only the valid ones. For example , which is never used. The comment could make it more clear. Should we also check it in release mode and return -1? http://gerrit.cloudera.org:8080/#/c/12985/14/be/src/exec/parquet/parquet-plain-test.cc File be/src/exec/parquet/parquet-plain-test.cc: http://gerrit.cloudera.org:8080/#/c/12985/14/be/src/exec/parquet/parquet-plain-test.cc@122 PS14, Line 122: = ParquetPlainEncoder::DecodeBatch( > nit: this line fits to the previous line It would be 93 characters long. -- To view, visit http://gerrit.cloudera.org:8080/12985 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I57b7d2573bb6dfd038e581acb3bd8ea1565aa20d Gerrit-Change-Number: 12985 Gerrit-PatchSet: 15 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 25 Apr 2019 16:32:17 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8381: Optimize ParquetPlainEncoder::DecodeBatch() for simple types
Hello Zoltan Borok-Nagy, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12985 to look at the new patch set (#15). Change subject: IMPALA-8381: Optimize ParquetPlainEncoder::DecodeBatch() for simple types .. IMPALA-8381: Optimize ParquetPlainEncoder::DecodeBatch() for simple types Refactored the ParquetPlainEncoder::Decode() and ParquetPlainEncoder::DecodeBatch() methods to increase performance in batch decoding. The `Decode` and `DecodeBatch` methods retain their behaviour and outward interface, but the internal structure changes. We change how we split up the `Decode` template specialisations. The generic unspecialised template is used for numerical parquet types (INT32, INT64, INT96, FLOAT and DOUBLE) and various specialisations are used for BYTE_ARRAY and FIXED_LEN_BYTE_ARRAY. We add a new method template, DecodeNoCheck, which does the actual decoding without bounds checking. It is called by the generic Decode method template internally. For all parquet types except for BYTE_ARRAY, DecodeBatch performs the bounds check once for the whole batch at the same time and calls DecodeNoCheck, so we save the cost of bounds checking for every decoded value. For BYTE_ARRAY, this cannot be done and we have to perform the checks for every value. In the non-BYTE_ARRAY version of DecodeBatch, we explicitly unroll the loop in batches of 8 to increase performance. The overall performance increase is up to 2x for small strides (8 bytes, INT32) but decreases as the stride increases, and disappears from around 40 bytes. With bigger strides, there is no performance difference from the previous implementation. Testing: Added tests to parquet-plain-test.cc to test the `Decode` and the `DecodeBatch` methods both in single-value decoding and batch decoding. Change-Id: I57b7d2573bb6dfd038e581acb3bd8ea1565aa20d --- M be/src/exec/parquet/parquet-common.h M be/src/exec/parquet/parquet-plain-test.cc A be/src/testutil/random-vector-generators.h 3 files changed, 468 insertions(+), 95 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/12985/15 -- To view, visit http://gerrit.cloudera.org:8080/12985 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I57b7d2573bb6dfd038e581acb3bd8ea1565aa20d Gerrit-Change-Number: 12985 Gerrit-PatchSet: 15 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] Remove hive-benchmark test workload
Hello Tim Armstrong, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/13117 to review the following change. Change subject: Remove hive-benchmark test workload .. Remove hive-benchmark test workload This workload was added in 2012 or so, and hasn't actually been able to run in years: - fails to load because the name has a '-' in it, and we use unquoted database names in the load script - the SQL to load it seems to refer to testdata files which don't exist Even if it did load, it looks like it's only a set of ~10 trivial queries which I doubt have any real benchmark interest in modern days. Change-Id: I383638ef4acce59792fee287da0016b3d949e693 --- M testdata/bin/generate-schema-statements.py M testdata/datasets/README D testdata/datasets/hive-benchmark/hive-benchmark_schema_template.sql M testdata/datasets/tpcds/tpcds_schema_template.sql M testdata/datasets/tpch/tpch_schema_template.sql D testdata/workloads/hive-benchmark/hive-benchmark_core.csv D testdata/workloads/hive-benchmark/hive-benchmark_dimensions.csv D testdata/workloads/hive-benchmark/hive-benchmark_exhaustive.csv D testdata/workloads/hive-benchmark/hive-benchmark_pairwise.csv D testdata/workloads/hive-benchmark/queries/hive-benchmark.test 10 files changed, 34 insertions(+), 215 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/17/13117/1 -- To view, visit http://gerrit.cloudera.org:8080/13117 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I383638ef4acce59792fee287da0016b3d949e693 Gerrit-Change-Number: 13117 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] Make infra/python compatible with both Python 2 & 3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13070 ) Change subject: Make infra/python compatible with both Python 2 & 3 .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2912/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13070 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4285a021bb581f88425daa52ef8a3f844017d82 Gerrit-Change-Number: 13070 Gerrit-PatchSet: 3 Gerrit-Owner: Akshesh Doshi Gerrit-Reviewer: Akshesh Doshi Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 25 Apr 2019 16:08:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8444: Fix performance regression when building privilege name
Csaba Ringhofer has posted comments on this change. (
http://gerrit.cloudera.org:8080/13095 )
Change subject: IMPALA-8444: Fix performance regression when building privilege
name
..
Patch Set 7:
(4 comments)
I am still uncertain about the case sensitivity, once that is clarified, I can
give +2.
http://gerrit.cloudera.org:8080/#/c/13095/7/fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationPolicy.java
File
fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationPolicy.java:
http://gerrit.cloudera.org:8080/#/c/13095/7/fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationPolicy.java@66
PS7, Line 66: for (String privilegeName: role.getPrivilegeNames()) {
: privileges.add(privilegeName);
: }
I think that this could be further simplified to
privileges.addAll(role.getPrivilegeNames()); It is also possible that the union
is a bit faster than inserting one-by-one, especially when adding the first set
to the initial empty set.
http://gerrit.cloudera.org:8080/#/c/13095/7/fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationPolicy.java@84
PS7, Line 84: for (String privilegeName: user.getPrivilegeNames()) {
: privileges.add(privilegeName);
: }
Same as line 66.
http://gerrit.cloudera.org:8080/#/c/13095/2/fe/src/main/java/org/apache/impala/catalog/Principal.java
File fe/src/main/java/org/apache/impala/catalog/Principal.java:
http://gerrit.cloudera.org:8080/#/c/13095/2/fe/src/main/java/org/apache/impala/catalog/Principal.java@63
PS2, Line 63:*/
: public boolean addPrivilege(PrincipalPrivilege privilege) {
: // URIs are case sensitive. Sentry treats everything in the
URI
> > if non-URI parts of the name are always lower case, then it
> doesn't matter if we treat them as case insensitive or not
>
> When calling getPrivilege(String privilegeName) for non URI, we
> need to know the exact case that's stored in the cache. It's
> simpler to special case the URI and make everything else to be case
> insensitive.
>
This is still not clear to me. Keys for non-URIs are lowered twice: once in
buildPrivilegeName() (or action and grant option can contain upper case
latters?), and once again in CatalogObjectCache functions. If the goal is to
find non-URI privileges in getPrivilege() even if there are upper/lower case
differences, then this could be achieved by adding function to
PrincipalPrivilege like string convertToCacheKey(string privilegeName) that
check if the name is an URI, and call toLower() if it is not. I would prefer
this because it would move the case handling issues to a single class (that
contains buildPrivilegeName()).
+ see my comment for getPrivilege()
> > if non-URI parts of the name can be upper case (including
> 'server'), then this add() will also treat the 'server' part of
> URIs in case sensitive way, which seems wrong
>
> Yeah I agree, this is weird. This is actually a bug in Sentry.
> Sentry treats everything in the URI as case sensitive including the
> host name. We can fix it by always lower casing the host name
> before sending it to Sentry. However if a privilege was granted
> outside Impala that contains upper case host name, there's nothing
> much that Impala can do. I'll add a comment in the code.
Ouch, thanks for the explanation.
http://gerrit.cloudera.org:8080/#/c/13095/2/fe/src/main/java/org/apache/impala/catalog/Principal.java@89
PS2, Line 89:
Don't we need to handle URIs case sensitively in this function?
principalPrivileges_.get(privilegeName) will call toLower() on privilegeName,
so it will not match with URI keys if they contain upper case latter.
--
To view, visit http://gerrit.cloudera.org:8080/13095
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I942d9b55f07c8972f69e532567d9b7d80fceb6e5
Gerrit-Change-Number: 13095
Gerrit-PatchSet: 7
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Thu, 25 Apr 2019 15:40:58 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] Make infra/python compatible with both Python 2 & 3
Akshesh Doshi has posted comments on this change. ( http://gerrit.cloudera.org:8080/13070 ) Change subject: Make infra/python compatible with both Python 2 & 3 .. Patch Set 4: Code-Review+1 Thank you so much Tim - appreciate your efforts. What would now be the next step to merge this? -- To view, visit http://gerrit.cloudera.org:8080/13070 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4285a021bb581f88425daa52ef8a3f844017d82 Gerrit-Change-Number: 13070 Gerrit-PatchSet: 4 Gerrit-Owner: Akshesh Doshi Gerrit-Reviewer: Akshesh Doshi Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 25 Apr 2019 15:32:54 + Gerrit-HasComments: No
[Impala-ASF-CR] Make infra/python compatible with both Python 2 & 3
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13070 ) Change subject: Make infra/python compatible with both Python 2 & 3 .. Patch Set 3: That's weird. Anyway I was able to grab the patchset from your github account, so no worries. -- To view, visit http://gerrit.cloudera.org:8080/13070 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4285a021bb581f88425daa52ef8a3f844017d82 Gerrit-Change-Number: 13070 Gerrit-PatchSet: 3 Gerrit-Owner: Akshesh Doshi Gerrit-Reviewer: Akshesh Doshi Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 25 Apr 2019 15:22:49 + Gerrit-HasComments: No
[Impala-ASF-CR] Make infra/python compatible with both Python 2 & 3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13070 ) Change subject: Make infra/python compatible with both Python 2 & 3 .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4065/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13070 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4285a021bb581f88425daa52ef8a3f844017d82 Gerrit-Change-Number: 13070 Gerrit-PatchSet: 4 Gerrit-Owner: Akshesh Doshi Gerrit-Reviewer: Akshesh Doshi Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 25 Apr 2019 15:22:26 + Gerrit-HasComments: No
