adoroszlai commented on code in PR #9901:
URL: https://github.com/apache/ozone/pull/9901#discussion_r2916011637
##########
hadoop-ozone/dist/src/main/compose/ozonesecure-ha/.env:
##########
@@ -24,7 +24,7 @@ OZONE_VOLUME=./data
OZONE_OPTS=
RANGER_DB_IMAGE=postgres
RANGER_DB_IMAGE_VERSION=12
-RANGER_IMAGE=ghcr.io/adoroszlai/ranger-admin
-RANGER_IMAGE_VERSION=0ae34250d3af672776fca6a53047699adf3afce5-${ranger.version}-8
+RANGER_IMAGE=ghcr.io/fmorg-git/ranger-admin
+RANGER_IMAGE_VERSION=5c567041d56bf153a1c7fb320af42b4a9af673b0-${ranger.version}-8
Review Comment:
We have already switched to official Ranger images on `master`, since they
are now available for both amd64 and arm64. (Previously it was arm64-only,
which does not work in CI.)
##########
hadoop-ozone/dist/src/main/compose/ozonesecure-ha/test-ranger.sh:
##########
@@ -31,20 +31,79 @@ export OM_SERVICE_ID="omservice"
export SCM=scm1.org
export SECURITY_ENABLED=true
-if [[ "${SKIP_APACHE_VERIFY_DOWNLOAD}" != "true" ]]; then
+# Check if we are using a snapshot version
+if [[ "${RANGER_VERSION}" =~ [0-9]+\.[0-9]+\.[0-9]+-[0-9]{8}\.[0-9]{6}-[0-9]+
]] || [[ "${RANGER_VERSION}" == *"SNAPSHOT"* ]]; then
+ IS_SNAPSHOT=true
+else
+ IS_SNAPSHOT=false
+fi
+
+if [[ "${SKIP_APACHE_VERIFY_DOWNLOAD}" != "true" ]] && [[ "${IS_SNAPSHOT}" ==
"false" ]]; then
curl -LO https://downloads.apache.org/ranger/KEYS
gpg --import KEYS
fi
-download_and_verify_apache_release
"ranger/${RANGER_VERSION}/apache-ranger-${RANGER_VERSION}.tar.gz"
-tar -C "${DOWNLOAD_DIR}" -x -z -f
"${DOWNLOAD_DIR}/apache-ranger-${RANGER_VERSION}.tar.gz"
-export RANGER_SOURCE_DIR="${DOWNLOAD_DIR}/apache-ranger-${RANGER_VERSION}"
+if [[ "${IS_SNAPSHOT}" == "true" ]]; then
+ # Snapshot download logic
+ RANGER_BASE_VERSION=$(echo "${RANGER_VERSION}" | sed -E
's/-[0-9]{8}\.[0-9]{6}-[0-9]+//')
+ if [[ "${RANGER_BASE_VERSION}" == "${RANGER_VERSION}" ]]; then
+ RANGER_BASE_VERSION="${RANGER_VERSION}"
+ else
+ RANGER_BASE_VERSION="${RANGER_BASE_VERSION}-SNAPSHOT"
+ fi
+
SNAPSHOT_REPO="https://repository.apache.org/content/groups/snapshots/org/apache/ranger/ranger-distro/${RANGER_BASE_VERSION}";
+
+ if [[ "${RANGER_VERSION}" == *"SNAPSHOT"* ]]; then
+ # If RANGER_VERSION is a snapshot (e.g. 2.8.0-SNAPSHOT), resolve it to
the latest timestamped version
+ download_if_not_exists "${SNAPSHOT_REPO}/maven-metadata.xml"
"${DOWNLOAD_DIR}/maven-metadata.xml"
+ TIMESTAMP=$(grep "<timestamp>" "${DOWNLOAD_DIR}/maven-metadata.xml" |
head -1 | sed -e 's/.*<timestamp>\(.*\)<\/timestamp>.*/\1/')
+ BUILDNUM=$(grep "<buildNumber>" "${DOWNLOAD_DIR}/maven-metadata.xml" |
head -1 | sed -e 's/.*<buildNumber>\(.*\)<\/buildNumber>.*/\1/')
+ if [[ -n "${TIMESTAMP}" ]] && [[ -n "${BUILDNUM}" ]]; then
+
RANGER_VERSION="${RANGER_BASE_VERSION%-SNAPSHOT}-${TIMESTAMP}-${BUILDNUM}"
+ echo "Resolved RANGER_VERSION to ${RANGER_VERSION}"
+ fi
+ fi
+
+ SRC_TAR="ranger-distro-${RANGER_VERSION}-src.tar.gz"
+ download_if_not_exists "${SNAPSHOT_REPO}/${SRC_TAR}"
"${DOWNLOAD_DIR}/${SRC_TAR}"
+ tar -C "${DOWNLOAD_DIR}" -x -z -f "${DOWNLOAD_DIR}/${SRC_TAR}"
+
+ # Find the extracted directory name
+ EXTRACTED_DIR=$(tar -tf "${DOWNLOAD_DIR}/${SRC_TAR}" | grep -o '^[^/]*' |
sort | uniq | head -1)
+ export RANGER_SOURCE_DIR="${DOWNLOAD_DIR}/${EXTRACTED_DIR}"
+else
+ # Release download logic
+ download_and_verify_apache_release
"ranger/${RANGER_VERSION}/apache-ranger-${RANGER_VERSION}.tar.gz"
+ tar -C "${DOWNLOAD_DIR}" -x -z -f
"${DOWNLOAD_DIR}/apache-ranger-${RANGER_VERSION}.tar.gz"
+ export RANGER_SOURCE_DIR="${DOWNLOAD_DIR}/apache-ranger-${RANGER_VERSION}"
+fi
+
chmod -R a+rX "${RANGER_SOURCE_DIR}"
-chmod a+x "${RANGER_SOURCE_DIR}"/dev-support/ranger-docker/config/*.sh
-download_and_verify_apache_release
"ranger/${RANGER_VERSION}/plugins/ozone/ranger-${RANGER_VERSION}-ozone-plugin.tar.gz"
-tar -C "${DOWNLOAD_DIR}" -x -z -f
"${DOWNLOAD_DIR}/ranger-${RANGER_VERSION}-ozone-plugin.tar.gz"
-export
RANGER_OZONE_PLUGIN_DIR="${DOWNLOAD_DIR}/ranger-${RANGER_VERSION}-ozone-plugin"
+# Ranger docker support scripts moved between releases (eg: from config/*.sh
to scripts/**).
+# Ensure we don't fail if a glob doesn't match, but still make init scripts
executable when present.
+shopt -s nullglob
+chmod_targets=(
+ "${RANGER_SOURCE_DIR}"/dev-support/ranger-docker/config/*.sh
+ "${RANGER_SOURCE_DIR}"/dev-support/ranger-docker/scripts/rdbms/*.sh
+)
+shopt -u nullglob
+if (( ${#chmod_targets[@]} > 0 )); then
+ chmod a+x "${chmod_targets[@]}"
+fi
Review Comment:
This can be simplified:
```bash
find "${RANGER_SOURCE_DIR}"/dev-support/ranger-docker -name '*.sh' | xargs
--no-run-if-empty chmod a+x
```
##########
hadoop-ozone/dist/src/main/compose/ozonesecure-ha/test-ranger.sh:
##########
@@ -53,6 +112,8 @@ perl -wpl -i \
-e 's@^POLICY_MGR_URL=.*@POLICY_MGR_URL=http://ranger:6080@;' \
-e 's@^REPOSITORY_NAME=.*@REPOSITORY_NAME=dev_ozone@;' \
-e 's@^CUSTOM_USER=ozone@CUSTOM_USER=hadoop@;' \
+ -e 's@^XAAUDIT.LOG4J.ENABLE=.*@XAAUDIT.LOG4J.ENABLE=false@;' \
+ -e
's@^XAAUDIT.LOG4J.DESTINATION.LOG4J=.*@XAAUDIT.LOG4J.DESTINATION.LOG4J=false@;'
\
Review Comment:
This was already added as part of bump to Ranger 2.7.0
(42c7ff52690a8ffa99fd4de60f7d6ed07466af1f).
##########
pom.xml:
##########
@@ -197,7 +197,7 @@
<protobuf-maven-plugin.version>0.6.1</protobuf-maven-plugin.version>
<protobuf2.version>2.5.0</protobuf2.version>
<protobuf3.version>3.25.8</protobuf3.version>
- <ranger.version>2.6.0</ranger.version>
+ <ranger.version>2.8.0</ranger.version>
Review Comment:
Ranger 2.8.0 switched from `hadoop-common` to `hadoop-client-api`, we need
to exclude the new dependencies to avoid [build
failure](https://github.com/fmorg-git/ozone/actions/runs/22932059910/job/66555565363#step:16:17):
```diff
diff --git hadoop-ozone/multitenancy-ranger/pom.xml
hadoop-ozone/multitenancy-ranger/pom.xml
index 526171a750..9d22925d2e 100644
--- hadoop-ozone/multitenancy-ranger/pom.xml
+++ hadoop-ozone/multitenancy-ranger/pom.xml
@@ -73,6 +73,14 @@
<groupId>net.minidev</groupId>
<artifactId>json-smart</artifactId>
</exclusion>
+ <exclusion>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-client-api</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-client-runtime</artifactId>
+ </exclusion>
<exclusion>
<groupId>org.apache.hive</groupId>
<artifactId>hive-storage-api</artifactId>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
