[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21160 ) Change subject: IMPALA-12921: Support locally built Ranger .. Patch Set 11: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/21160 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27 Gerrit-Change-Number: 21160 Gerrit-PatchSet: 11 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 08 May 2024 14:52:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21160 ) Change subject: IMPALA-12921: Support locally built Ranger .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/16129/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21160 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27 Gerrit-Change-Number: 21160 Gerrit-PatchSet: 11 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 08 May 2024 09:55:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/21160 ) Change subject: IMPALA-12921: Support locally built Ranger .. Patch Set 11: (1 comment) In patch set 11 I added some steps to create the user 'non_owner_2' on the Ranger server since in Apache Ranger, the Ranger server also checks the existence of the grantee in the GRANT/REVOKE statements. In addition, I plan to add a startup flag to the catalog server so that we could specify whether the resource type of 'storage-type' is supported by the Ranger service. Without this, some basic statements like GRANT ALL ON SERVER could not even be supported if Apache Ranger is used. http://gerrit.cloudera.org:8080/#/c/21160/7//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21160/7//COMMIT_MSG@64 PS7, Line 64:Note that the resource type of 'storage-type' is not supported > Thanks Quanlong! After running some authorization-related tests, e.g., RangerAuditLogTest, I realized that the resource type of 'storage-type' is not supported in Apache Ranger yet. Hence the support for storage handler privileges added in IMPALA-10436 is not supported. This also affects the ALL privilege on SERVER since this the ALL privilege on SERVER also implies the RWSTORAGE privilege on all storage handler URI's. It may be a good idea to add a startup flag to the catalog server so that we could specify whether the resource type of 'storage-type' is supported by the Ranger service so the catalog server won't send the requests to grant/revoke privileges on the storage handler URI's. -- To view, visit http://gerrit.cloudera.org:8080/21160 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27 Gerrit-Change-Number: 21160 Gerrit-PatchSet: 11 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 08 May 2024 09:47:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21160 ) Change subject: IMPALA-12921: Support locally built Ranger .. Patch Set 11: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10618/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/21160 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27 Gerrit-Change-Number: 21160 Gerrit-PatchSet: 11 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 08 May 2024 09:48:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger
Hello Quanlong Huang, Aman Sinha, Joe McDonnell, Impala Public Jenkins, John
Sherman,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21160
to look at the new patch set (#11).
Change subject: IMPALA-12921: Support locally built Ranger
..
IMPALA-12921: Support locally built Ranger
This patch does the following.
1. Add the support for locally built Ranger.
2. Resolve IMPALA-12985 by switching to the new constructor when
instantiating RangerAccessRequestImpl.
The instructions on how to build Ranger locally and point
Impala to the locally built Ranger server are provided as below.
Suppose the Ranger project is under the folder $RANGER_SRC_DIR. We could
execute the following to build Ranger. By default, the compressed
tarball is produced under $RANGER_SRC_DIR/target.
mvn clean compile -B -nsu -DskipCheck=true -Dcheckstyle.skip=true \
package install -DskipITs -DskipTests -Dmaven.javadoc.skip=true
After building Ranger, we need to build Impala's Java code so that
Impala's Java code could consume the locally produced Ranger classes. We
will need to export the following environment variables before building
Impala. This prevents bootstrap_toolchain.py from trying to download the
compressed Ranger tarball. Moreover, we need to apply
IMPALA-12921_pom_patching.diff on Impala if the locally built Ranger
pulls in an incompatible version of hive-storage-api.
1. export RANGER_VERSION_OVERRIDE=\
$(mvn -f $RANGER_SRC_DIR/pom.xml -q help:evaluate \
-Dexpression=project.version -DforceStdout)
2. export RANGER_HOME_OVERRIDE=$RANGER_SRC_DIR/target/\
ranger-${RANGER_VERSION_OVERRIDE}-admin
It then suffices to execute the following to point
Impala to the locally built Ranger server before starting Impala.
1. source $IMPALA_HOME/bin/impala-config.sh
2. tar zxv -f $RANGER_SRC_DIR/target/\
ranger-${IMPALA_RANGER_VERSION}-admin.tar.gz \
-C $RANGER_SRC_DIR/target/
3. $IMPALA_HOME/bin/create-test-configuration.sh
4. $IMPALA_HOME/bin/create-test-configuration.sh \
-create_ranger_policy_db
5. $IMPALA_HOME/testdata/bin/run-ranger.sh
(run-all.sh has to be executed instead if other underlying services
have not been started)
6. $IMPALA_HOME/testdata/bin/setup-ranger.sh
Testing:
- Manually verified that we could point Impala to a locally built
Apache Ranger on the master branch (with tip being RANGER-4745).
- Manually verified that with the Ranger patch in RANGER-4771.diff,
we could manage the policy repository via GRANT/REVOKE statements.
Note that the resource type of 'storage-type' is not supported
in Apache Ranger yet and thus the support for storage handler
privileges added in IMPALA-10436 is not supported. The ALL privilege
on SERVER is also affected in that this privilege on SERVER also
applies to all storage handler URI's.
- Verified that this patch passed the core tests when CDP Ranger is
used.
Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27
---
M README-build.md
M bin/bootstrap_toolchain.py
M bin/create-test-configuration.sh
M bin/impala-config.sh
M bin/rat_exclude_files.txt
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java
M testdata/bin/setup-ranger.sh
A testdata/cluster/ranger/IMPALA-12921_pom_patching.diff
A testdata/cluster/ranger/RANGER-4771.diff
A testdata/cluster/ranger/README
R testdata/cluster/ranger/setup/all_database_policy_revised.json.template
A testdata/cluster/ranger/setup/impala_group_non_owner_2.json
M testdata/cluster/ranger/setup/impala_user_non_owner.json.template
A testdata/cluster/ranger/setup/impala_user_non_owner_2.json.template
M testdata/cluster/ranger/setup/impala_user_owner.json.template
16 files changed, 183 insertions(+), 16 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/21160/11
--
To view, visit http://gerrit.cloudera.org:8080/21160
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27
Gerrit-Change-Number: 21160
Gerrit-PatchSet: 11
Gerrit-Owner: Fang-Yu Rao
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Fang-Yu Rao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: John Sherman
Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21160 ) Change subject: IMPALA-12921: Support locally built Ranger .. Patch Set 10: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/21160 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27 Gerrit-Change-Number: 21160 Gerrit-PatchSet: 10 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 07 May 2024 14:03:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21160 ) Change subject: IMPALA-12921: Support locally built Ranger .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/16112/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21160 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27 Gerrit-Change-Number: 21160 Gerrit-PatchSet: 10 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 07 May 2024 09:08:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21160 ) Change subject: IMPALA-12921: Support locally built Ranger .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/16111/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21160 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27 Gerrit-Change-Number: 21160 Gerrit-PatchSet: 9 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 07 May 2024 09:04:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/21160 ) Change subject: IMPALA-12921: Support locally built Ranger .. Patch Set 10: (2 comments) Hi all, I have revised patch set 7 according to Quanlong's comments. Please ignore patch sets 8 and 9 since I made some mistakes when addressing Quanlong's comments. Let me know if there is any additional suggestion. I plan to collect the list of passed authorization-related tests when Apache Ranger is used with this patch and will update on this when the results are available. http://gerrit.cloudera.org:8080/#/c/21160/7//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21160/7//COMMIT_MSG@64 PS7, Line 64: - Verified that this patch passed the core tests when CDP Ranger is > Do you mean using Apache Ranger? That would be great! Thanks Quanlong! I only verified that this patch passed the core tests with CDP Ranger. I will clarify this in the commit message of the next patch. I could try to collect the list of passed authorization-related tests when the patched Apache Ranger (with RANGER-4771.diff on top of RANGER-4745) is used. http://gerrit.cloudera.org:8080/#/c/21160/7/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java: http://gerrit.cloudera.org:8080/#/c/21160/7/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@500 PS7, Line 500: req.setResource(resource); > It seems using the empty constructor and setting the fields later is a more Thanks Quanlong! I will revise this accordingly in the next patch. -- To view, visit http://gerrit.cloudera.org:8080/21160 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27 Gerrit-Change-Number: 21160 Gerrit-PatchSet: 10 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 07 May 2024 08:49:26 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21160 ) Change subject: IMPALA-12921: Support locally built Ranger .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10612/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/21160 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27 Gerrit-Change-Number: 21160 Gerrit-PatchSet: 10 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 07 May 2024 08:50:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger
Hello Quanlong Huang, Aman Sinha, Joe McDonnell, Impala Public Jenkins, John
Sherman,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21160
to look at the new patch set (#10).
Change subject: IMPALA-12921: Support locally built Ranger
..
IMPALA-12921: Support locally built Ranger
This patch does the following.
1. Add the support for locally built Ranger.
2. Resolve IMPALA-12985 by switching to the new constructor when
instantiating RangerAccessRequestImpl.
The instructions on how to build Ranger locally and point
Impala to the locally built Ranger server are provided as below.
Suppose the Ranger project is under the folder $RANGER_SRC_DIR. We could
execute the following to build Ranger. By default, the compressed
tarball is produced under $RANGER_SRC_DIR/target.
mvn clean compile -B -nsu -DskipCheck=true -Dcheckstyle.skip=true \
package install -DskipITs -DskipTests -Dmaven.javadoc.skip=true
After building Ranger, we need to build Impala's Java code so that
Impala's Java code could consume the locally produced Ranger classes. We
will need to export the following environment variables before building
Impala. This prevents bootstrap_toolchain.py from trying to download the
compressed Ranger tarball. Moreover, we need to apply
IMPALA-12921_pom_patching.diff on Impala if the locally built Ranger
pulls in an incompatible version of hive-storage-api.
1. export RANGER_VERSION_OVERRIDE=\
$(mvn -f $RANGER_SRC_DIR/pom.xml -q help:evaluate \
-Dexpression=project.version -DforceStdout)
2. export RANGER_HOME_OVERRIDE=$RANGER_SRC_DIR/target/\
ranger-${RANGER_VERSION_OVERRIDE}-admin
It then suffices to execute the following to point
Impala to the locally built Ranger server before starting Impala.
1. source $IMPALA_HOME/bin/impala-config.sh
2. tar zxv -f $RANGER_SRC_DIR/target/\
ranger-${IMPALA_RANGER_VERSION}-admin.tar.gz \
-C $RANGER_SRC_DIR/target/
3. $IMPALA_HOME/bin/create-test-configuration.sh
4. $IMPALA_HOME/bin/create-test-configuration.sh \
-create_ranger_policy_db
5. $IMPALA_HOME/testdata/bin/run-ranger.sh
(run-all.sh has to be executed instead if other underlying services
have not been started)
6. $IMPALA_HOME/testdata/bin/setup-ranger.sh
Testing:
- Manually verified that we could point Impala to a locally built
Apache Ranger on the master branch (with tip being RANGER-4745).
- Manually verified that with the Ranger patch in RANGER-4771.diff,
we could manage the policy repository via GRANT/REVOKE statements.
- Verified that this patch passed the core tests when CDP Ranger is
used.
Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27
---
M README-build.md
M bin/bootstrap_toolchain.py
M bin/create-test-configuration.sh
M bin/impala-config.sh
M bin/rat_exclude_files.txt
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java
M testdata/bin/setup-ranger.sh
A testdata/cluster/ranger/IMPALA-12921_pom_patching.diff
A testdata/cluster/ranger/RANGER-4771.diff
A testdata/cluster/ranger/README
R testdata/cluster/ranger/setup/all_database_policy_revised.json.template
M testdata/cluster/ranger/setup/impala_user_non_owner.json.template
M testdata/cluster/ranger/setup/impala_user_owner.json.template
14 files changed, 142 insertions(+), 16 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/21160/10
--
To view, visit http://gerrit.cloudera.org:8080/21160
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27
Gerrit-Change-Number: 21160
Gerrit-PatchSet: 10
Gerrit-Owner: Fang-Yu Rao
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Fang-Yu Rao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: John Sherman
Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger
Hello Quanlong Huang, Aman Sinha, Joe McDonnell, Impala Public Jenkins, John
Sherman,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21160
to look at the new patch set (#9).
Change subject: IMPALA-12921: Support locally built Ranger
..
IMPALA-12921: Support locally built Ranger
This patch does the following.
1. Add the support for locally built Ranger.
2. Resolve IMPALA-12985 by switching to the new constructor when
instantiating RangerAccessRequestImpl.
The instructions on how to build Ranger locally and point
Impala to the locally built Ranger server are provided as below.
Suppose the Ranger project is under the folder $RANGER_SRC_DIR. We could
execute the following to build Ranger. By default, the compressed
tarball is produced under $RANGER_SRC_DIR/target.
mvn clean compile -B -nsu -DskipCheck=true -Dcheckstyle.skip=true \
package install -DskipITs -DskipTests -Dmaven.javadoc.skip=true
After building Ranger, we need to build Impala's Java code so that
Impala's Java code could consume the locally produced Ranger classes. We
will need to export the following environment variables before building
Impala. This prevents bootstrap_toolchain.py from trying to download the
compressed Ranger tarball. Moreover, we need to apply
IMPALA-12921_pom_patching.diff on Impala if the locally built Ranger
pulls in an incompatible version of hive-storage-api.
1. export RANGER_VERSION_OVERRIDE=\
$(mvn -f $RANGER_SRC_DIR/pom.xml -q help:evaluate \
-Dexpression=project.version -DforceStdout)
2. export RANGER_HOME_OVERRIDE=$RANGER_SRC_DIR/target/\
ranger-${RANGER_VERSION_OVERRIDE}-admin
It then suffices to execute the following to point
Impala to the locally built Ranger server before starting Impala.
1. source $IMPALA_HOME/bin/impala-config.sh
2. tar zxv -f $RANGER_SRC_DIR/target/\
ranger-${IMPALA_RANGER_VERSION}-admin.tar.gz \
-C $RANGER_SRC_DIR/target/
3. $IMPALA_HOME/bin/create-test-configuration.sh
4. $IMPALA_HOME/bin/create-test-configuration.sh \
-create_ranger_policy_db
5. $IMPALA_HOME/testdata/bin/run-ranger.sh
(run-all.sh has to be executed instead if other underlying services
have not been started)
6. $IMPALA_HOME/testdata/bin/setup-ranger.sh
Testing:
- Manually verified that we could point Impala to a locally built
Apache Ranger on the master branch (with tip being RANGER-4745).
- Manually verified that with the Ranger patch in RANGER-4771.diff,
we could manage the policy repository via GRANT/REVOKE statements.
- Verified that this patch passed the core tests when CDP Ranger is
used.
Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27
---
M README-build.md
M bin/bootstrap_toolchain.py
M bin/create-test-configuration.sh
M bin/impala-config.sh
M bin/rat_exclude_files.txt
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java
M testdata/bin/setup-ranger.sh
A testdata/cluster/ranger/IMPALA-12921_pom_patching.diff
A testdata/cluster/ranger/RANGER-4771.diff
A testdata/cluster/ranger/README
R testdata/cluster/ranger/setup/all_database_policy_revised.json.template
M testdata/cluster/ranger/setup/impala_user_non_owner.json.template
M testdata/cluster/ranger/setup/impala_user_owner.json.template
14 files changed, 142 insertions(+), 16 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/21160/9
--
To view, visit http://gerrit.cloudera.org:8080/21160
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27
Gerrit-Change-Number: 21160
Gerrit-PatchSet: 9
Gerrit-Owner: Fang-Yu Rao
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Fang-Yu Rao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: John Sherman
Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21160 ) Change subject: IMPALA-12921: Support locally built Ranger .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/16110/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21160 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27 Gerrit-Change-Number: 21160 Gerrit-PatchSet: 8 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 07 May 2024 08:35:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger
Hello Quanlong Huang, Aman Sinha, Joe McDonnell, Impala Public Jenkins, John
Sherman,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21160
to look at the new patch set (#8).
Change subject: IMPALA-12921: Support locally built Ranger
..
IMPALA-12921: Support locally built Ranger
This patch does the following.
1. Add the support for locally built Ranger.
2. Resolve IMPALA-12985 by switching to the new constructor when
instantiating RangerAccessRequestImpl.
The instructions on how to build Ranger locally and point
Impala to the locally built Ranger server are provided as below.
Suppose the Ranger project is under the folder $RANGER_SRC_DIR. We could
execute the following to build Ranger. By default, the compressed
tarball is produced under $RANGER_SRC_DIR/target.
mvn clean compile -B -nsu -DskipCheck=true -Dcheckstyle.skip=true \
package install -DskipITs -DskipTests -Dmaven.javadoc.skip=true
After building Ranger, we need to build Impala's Java code so that
Impala's Java code could consume the locally produced Ranger classes. We
will need to export the following environment variables before building
Impala. This prevents bootstrap_toolchain.py from trying to download the
compressed Ranger tarball. Moreover, we need to apply
IMPALA-12921_pom_patching.diff on Impala if the locally built Ranger
pulls in an incompatible version of hive-storage-api.
1. export RANGER_VERSION_OVERRIDE=\
$(mvn -f $RANGER_SRC_DIR/pom.xml -q help:evaluate \
-Dexpression=project.version -DforceStdout)
2. export RANGER_HOME_OVERRIDE=$RANGER_SRC_DIR/target/\
ranger-${RANGER_VERSION_OVERRIDE}-admin
It then suffices to execute the following to point
Impala to the locally built Ranger server before starting Impala.
1. source $IMPALA_HOME/bin/impala-config.sh
2. tar zxv -f $RANGER_SRC_DIR/target/\
ranger-${IMPALA_RANGER_VERSION}-admin.tar.gz \
-C $RANGER_SRC_DIR/target/
3. $IMPALA_HOME/bin/create-test-configuration.sh
4. $IMPALA_HOME/bin/create-test-configuration.sh \
-create_ranger_policy_db
5. $IMPALA_HOME/testdata/bin/run-ranger.sh
(run-all.sh has to be executed instead if other underlying services
have not been started)
6. $IMPALA_HOME/testdata/bin/setup-ranger.sh
Testing:
- Manually verified that we could point Impala to a locally built
Apache Ranger on the master branch (with tip being RANGER-4745).
- Manually verified that with the Ranger patch in RANGER-4771.diff,
we could manage the policy repository via GRANT/REVOKE statements.
- Verified that this patch passed the core tests when CDP Ranger is
used.
Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27
---
M README-build.md
M bin/bootstrap_toolchain.py
M bin/create-test-configuration.sh
M bin/impala-config.sh
M bin/rat_exclude_files.txt
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java
M testdata/bin/setup-ranger.sh
A testdata/cluster/ranger/IMPALA-12921_pom_patching.diff
A testdata/cluster/ranger/RANGER-4771.diff
A testdata/cluster/ranger/README
R testdata/cluster/ranger/setup/all_database_policy_revised.json.template
M testdata/cluster/ranger/setup/impala_user_non_owner.json.template
M testdata/cluster/ranger/setup/impala_user_owner.json.template
14 files changed, 134 insertions(+), 13 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/21160/8
--
To view, visit http://gerrit.cloudera.org:8080/21160
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27
Gerrit-Change-Number: 21160
Gerrit-PatchSet: 8
Gerrit-Owner: Fang-Yu Rao
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Fang-Yu Rao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: John Sherman
Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21160 ) Change subject: IMPALA-12921: Support locally built Ranger .. Patch Set 7: (2 comments) http://gerrit.cloudera.org:8080/#/c/21160/7//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21160/7//COMMIT_MSG@64 PS7, Line 64: - Verified that this patch passed the core tests. Do you mean using Apache Ranger? That would be great! http://gerrit.cloudera.org:8080/#/c/21160/7/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java: http://gerrit.cloudera.org:8080/#/c/21160/7/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@500 PS7, Line 500: SELECT_ACCESS_TYPE, user.getShortName(), getUserGroups(user), null); It seems using the empty constructor and setting the fields later is a more compatible way. As shown in Yubi's patch here: https://gerrit.cloudera.org/c/18977/17/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java -- To view, visit http://gerrit.cloudera.org:8080/21160 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27 Gerrit-Change-Number: 21160 Gerrit-PatchSet: 7 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Sun, 28 Apr 2024 08:58:06 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/21160 ) Change subject: IMPALA-12921: Support locally built Ranger .. Patch Set 7: Hi all, please let me know if you have any comment on this patch (patch set 7). This patch is similar to https://gerrit.cloudera.org/c/17094/ where we added the support for locally built Hive (IMPALA-9218). The patch set 6 passed the dryrun tests at https://jenkins.impala.io/job/gerrit-verify-dryrun/10516/. In patch set 7, I only removed an exclusion from fe/pom.xml and updated some files that would not affect the functionality or the build of Impala when CDP dependencies are used. Thank you very much for the help! -- To view, visit http://gerrit.cloudera.org:8080/21160 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27 Gerrit-Change-Number: 21160 Gerrit-PatchSet: 7 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 10 Apr 2024 20:28:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21160 ) Change subject: IMPALA-12921: Support locally built Ranger .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15860/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21160 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27 Gerrit-Change-Number: 21160 Gerrit-PatchSet: 7 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 10 Apr 2024 19:22:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21160
to look at the new patch set (#7).
Change subject: IMPALA-12921: Support locally built Ranger
..
IMPALA-12921: Support locally built Ranger
This patch does the following.
1. Add the support for locally built Ranger.
2. Resolve IMPALA-12985 by switching to the new constructor when
instantiating RangerAccessRequestImpl.
The instructions on how to build Ranger locally and point
Impala to the locally built Ranger server are provided as below.
Suppose the Ranger project is under the folder $RANGER_SRC_DIR. We could
execute the following to build Ranger. By default, the compressed
tarball is produced under $RANGER_SRC_DIR/target.
mvn clean compile -B -nsu -DskipCheck=true -Dcheckstyle.skip=true \
package install -DskipITs -DskipTests -Dmaven.javadoc.skip=true
After building Ranger, we need to build Impala's Java code so that
Impala's Java code could consume the locally produced Ranger classes. We
will need to export the following environment variables before building
Impala. This prevents bootstrap_toolchain.py from trying to download the
compressed Ranger tarball. Moreover, we need to apply
IMPALA-12921_pom_patching.diff on Impala if the locally built Ranger
pulls in an incompatible version of hive-storage-api.
1. export RANGER_VERSION_OVERRIDE=\
$(mvn -f $RANGER_SRC_DIR/pom.xml -q help:evaluate \
-Dexpression=project.version -DforceStdout)
2. export RANGER_HOME_OVERRIDE=$RANGER_SRC_DIR/target/\
ranger-${RANGER_VERSION_OVERRIDE}-admin
It then suffices to execute the following to point
Impala to the locally built Ranger server before starting Impala.
1. source $IMPALA_HOME/bin/impala-config.sh
2. tar zxv -f $RANGER_SRC_DIR/target/\
ranger-${IMPALA_RANGER_VERSION}-admin.tar.gz \
-C $RANGER_SRC_DIR/target/
3. $IMPALA_HOME/bin/create-test-configuration.sh
4. $IMPALA_HOME/bin/create-test-configuration.sh \
-create_ranger_policy_db
5. $IMPALA_HOME/testdata/bin/run-ranger.sh
(run-all.sh has to be executed instead if other underlying services
have not been started)
6. $IMPALA_HOME/testdata/bin/setup-ranger.sh
Testing:
- Manually verified that we could point Impala to a locally built
Apache Ranger on the master branch (with tip being RANGER-4745).
- Manually verified that with the Ranger patch in RANGER-4771.diff,
we could manage the policy repository via GRANT/REVOKE statements.
- Verified that this patch passed the core tests.
Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27
---
M README-build.md
M bin/bootstrap_toolchain.py
M bin/create-test-configuration.sh
M bin/impala-config.sh
M bin/rat_exclude_files.txt
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java
M testdata/bin/setup-ranger.sh
A testdata/cluster/ranger/IMPALA-12921_pom_patching.diff
A testdata/cluster/ranger/RANGER-4771.diff
A testdata/cluster/ranger/README
R testdata/cluster/ranger/setup/all_database_policy_revised.json.template
M testdata/cluster/ranger/setup/impala_user_non_owner.json.template
M testdata/cluster/ranger/setup/impala_user_owner.json.template
14 files changed, 126 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/21160/7
--
To view, visit http://gerrit.cloudera.org:8080/21160
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27
Gerrit-Change-Number: 21160
Gerrit-PatchSet: 7
Gerrit-Owner: Fang-Yu Rao
Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21160 ) Change subject: IMPALA-12921: Support locally built Ranger .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/21160 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27 Gerrit-Change-Number: 21160 Gerrit-PatchSet: 6 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 10 Apr 2024 11:25:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21160 ) Change subject: IMPALA-12921: Support locally built Ranger .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10515/ -- To view, visit http://gerrit.cloudera.org:8080/21160 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27 Gerrit-Change-Number: 21160 Gerrit-PatchSet: 5 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 10 Apr 2024 07:01:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21160 ) Change subject: IMPALA-12921: Support locally built Ranger .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15842/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21160 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27 Gerrit-Change-Number: 21160 Gerrit-PatchSet: 6 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 10 Apr 2024 06:32:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21160
to look at the new patch set (#6).
Change subject: IMPALA-12921: Support locally built Ranger
..
IMPALA-12921: Support locally built Ranger
This patch does the following.
1. Add the support for locally built Ranger.
2. Resolve IMPALA-12985 by switching to the new constructor when
instantiating RangerAccessRequestImpl.
The instructions on how to build Ranger locally and point
Impala to the locally built Ranger server are provided as below.
Suppose the Ranger project is under the folder $RANGER_SRC_DIR. We could
execute the following to build Ranger. By default, the compressed
tarball is produced under $RANGER_SRC_DIR/target.
mvn clean compile -B -nsu -DskipCheck=true -Dcheckstyle.skip=true \
package install -DskipITs -DskipTests -Dmaven.javadoc.skip=true
After building Ranger, we need to build Impala's Java code so that
Impala's Java code could consume the locally produced Ranger classes. We
will need to export the following environment variables before building
Impala. This prevents bootstrap_toolchain.py from trying to download the
compressed Ranger tarball.
1. export RANGER_VERSION_OVERRIDE=\
$(mvn -f $RANGER_SRC_DIR/pom.xml -q help:evaluate \
-Dexpression=project.version -DforceStdout)
2. export RANGER_HOME_OVERRIDE=$RANGER_SRC_DIR/target/\
ranger-${RANGER_VERSION_OVERRIDE}-admin
It then suffices to execute the following to point
Impala to the locally built Ranger server before starting Impala.
1. source $IMPALA_HOME/bin/impala-config.sh
2. tar zxv -f $RANGER_SRC_DIR/target/\
ranger-${IMPALA_RANGER_VERSION}-admin.tar.gz \
-C $RANGER_SRC_DIR/target/
3. $IMPALA_HOME/bin/create-test-configuration.sh
4. $IMPALA_HOME/bin/create-test-configuration.sh \
-create_ranger_policy_db
5. $IMPALA_HOME/testdata/bin/run-ranger.sh
(run-all.sh has to be executed instead if other underlying services
have not been started)
6. $IMPALA_HOME/testdata/bin/setup-ranger.sh
Testing:
- Manually verified that we could point Impala to a locally built
Apache Ranger on the master branch (with tip being RANGER-4745).
- Manually verified that with the Ranger patch in RANGER-4771.diff,
we could manage the policy repository via GRANT/REVOKE statements.
- Verified that this patch passed the core tests.
Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27
---
M bin/bootstrap_toolchain.py
M bin/create-test-configuration.sh
M bin/impala-config.sh
M bin/rat_exclude_files.txt
M fe/pom.xml
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java
M testdata/bin/setup-ranger.sh
A testdata/cluster/ranger/RANGER-4771.diff
A testdata/cluster/ranger/README
R testdata/cluster/ranger/setup/all_database_policy_revised.json.template
M testdata/cluster/ranger/setup/impala_user_non_owner.json.template
M testdata/cluster/ranger/setup/impala_user_owner.json.template
13 files changed, 96 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/21160/6
--
To view, visit http://gerrit.cloudera.org:8080/21160
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27
Gerrit-Change-Number: 21160
Gerrit-PatchSet: 6
Gerrit-Owner: Fang-Yu Rao
Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21160 ) Change subject: IMPALA-12921: Support locally built Ranger .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10516/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/21160 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27 Gerrit-Change-Number: 21160 Gerrit-PatchSet: 6 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 10 Apr 2024 06:08:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21160 ) Change subject: IMPALA-12921: Support locally built Ranger .. Patch Set 5: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/15840/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/21160 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27 Gerrit-Change-Number: 21160 Gerrit-PatchSet: 5 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 10 Apr 2024 02:07:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21160 ) Change subject: IMPALA-12921: Support locally built Ranger .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10515/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/21160 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27 Gerrit-Change-Number: 21160 Gerrit-PatchSet: 5 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 10 Apr 2024 01:48:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21160
to look at the new patch set (#5).
Change subject: IMPALA-12921: Support locally built Ranger
..
IMPALA-12921: Support locally built Ranger
This patch does the following.
1. Add the support for locally built Ranger.
2. Resolve IMPALA-12985 by switching to the new constructor when
instantiating RangerAccessRequestImpl.
The instructions on how to build Ranger locally and point
Impala to the locally built Ranger server are provided as below.
Suppose the Ranger project is under the folder $RANGER_SRC_DIR. We could
execute the following to build Ranger. By default, the compressed
tarball is produced under $RANGER_SRC_DIR/target.
mvn clean compile -B -nsu -DskipCheck=true -Dcheckstyle.skip=true \
package install -DskipITs -DskipTests -Dmaven.javadoc.skip=true
After building Ranger, we need to build Impala's Java code so that
Impala's Java code could consume the locally produced Ranger classes. We
will need to export the following environment variables before building
Impala. This prevents bootstrap_toolchain.py from trying to download the
compressed Ranger tarball.
1. export RANGER_VERSION_OVERRIDE=\
$(mvn -f $RANGER_SRC_DIR/pom.xml -q help:evaluate \
-Dexpression=project.version -DforceStdout)
2. export RANGER_HOME_OVERRIDE=$RANGER_SRC_DIR/target/\
ranger-${RANGER_VERSION_OVERRIDE}-admin
It then suffices to execute the following to point
Impala to the locally built Ranger server before starting Impala.
1. source $IMPALA_HOME/bin/impala-config.sh
2. tar zxv -f $RANGER_SRC_DIR/target/\
ranger-${IMPALA_RANGER_VERSION}-admin.tar.gz \
-C $RANGER_SRC_DIR/target/
3. $IMPALA_HOME/bin/create-test-configuration.sh
4. $IMPALA_HOME/bin/create-test-configuration.sh \
-create_ranger_policy_db
5. $IMPALA_HOME/testdata/bin/run-ranger.sh
(run-all.sh has to be executed instead if other underlying services
have not been started)
6. $IMPALA_HOME/testdata/bin/setup-ranger.sh
Testing:
- Manually verified that we could point Impala to a locally built
Apache Ranger on the master branch (with tip being RANGER-4745).
- Manually verified that with the Ranger patch in RANGER-4771.diff,
we could manage the policy repository via GRANT/REVOKE statements.
- Verified that this patch passed the core tests.
Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27
---
M bin/bootstrap_toolchain.py
M bin/create-test-configuration.sh
M bin/impala-config.sh
M fe/pom.xml
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java
M testdata/bin/setup-ranger.sh
A testdata/cluster/ranger/RANGER-4771.diff
A testdata/cluster/ranger/README
R testdata/cluster/ranger/setup/all_database_policy_revised.json.template
M testdata/cluster/ranger/setup/impala_user_non_owner.json.template
M testdata/cluster/ranger/setup/impala_user_owner.json.template
12 files changed, 92 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/21160/5
--
To view, visit http://gerrit.cloudera.org:8080/21160
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27
Gerrit-Change-Number: 21160
Gerrit-PatchSet: 5
Gerrit-Owner: Fang-Yu Rao
Gerrit-Reviewer: Impala Public Jenkins
