nsivabalan commented on code in PR #8779:
URL: https://github.com/apache/hudi/pull/8779#discussion_r1200947437
##########
packaging/bundle-validation/README.md:
##########
@@ -50,4 +50,20 @@ Note that for each library like Hive and Spark, the download
and extraction happ
only one layer is generated to limit the size of the image. However, this
makes repeated downloads when rebuilding the
image. If you need faster iteration for local build, you may use the
`Dockerfile` under `base-dev/`, which uses `ADD`
instruction for downloads, which provides caching across builds. This
increases the size of the generated image compared
-to `base/` and the image should only be used for development only and not be
pushed to remote.
\ No newline at end of file
+to `base/` and the image should only be used for development only and not be
pushed to remote.
+
+## Running Bundle Validation on a Release Candidate
+
+The bundle validation on a release candidate is specified in the Github Action
job `validate-release-candidate-bundles`
+in `.github/workflows/bot.yml`. By default, this is disabled.
+
+To enable the bundle validation on a particular release candidate, makes the
following changes to the job by fipping the
Review Comment:
we should update our release guide as well rgdn this step
##########
.github/workflows/bot.yml:
##########
@@ -210,3 +210,62 @@ jobs:
run: |
HUDI_VERSION=$(mvn help:evaluate -Dexpression=project.version -q
-DforceStdout)
./packaging/bundle-validation/ci_run.sh $HUDI_VERSION openjdk17
+
+ validate-release-candidate-bundles:
+ if: true
+ runs-on: ubuntu-latest
+ env:
+ HUDI_VERSION: 0.13.1-rc1
Review Comment:
by default can you set some invalid release(instead of 0.13.1 rc1), so that
it will fail rather than succeeding by mistake.
##########
packaging/bundle-validation/ci_run.sh:
##########
@@ -96,14 +99,73 @@ fi
# Copy bundle jars to temp dir for mounting
TMP_JARS_DIR=/tmp/jars/$(date +%s)
mkdir -p $TMP_JARS_DIR
-cp
${GITHUB_WORKSPACE}/packaging/hudi-flink-bundle/target/hudi-*-$HUDI_VERSION.jar
$TMP_JARS_DIR/
-cp
${GITHUB_WORKSPACE}/packaging/hudi-hadoop-mr-bundle/target/hudi-*-$HUDI_VERSION.jar
$TMP_JARS_DIR/
-cp
${GITHUB_WORKSPACE}/packaging/hudi-kafka-connect-bundle/target/hudi-*-$HUDI_VERSION.jar
$TMP_JARS_DIR/
-cp
${GITHUB_WORKSPACE}/packaging/hudi-spark-bundle/target/hudi-*-$HUDI_VERSION.jar
$TMP_JARS_DIR/
-cp
${GITHUB_WORKSPACE}/packaging/hudi-utilities-bundle/target/hudi-*-$HUDI_VERSION.jar
$TMP_JARS_DIR/
-cp
${GITHUB_WORKSPACE}/packaging/hudi-utilities-slim-bundle/target/hudi-*-$HUDI_VERSION.jar
$TMP_JARS_DIR/
-cp
${GITHUB_WORKSPACE}/packaging/hudi-metaserver-server-bundle/target/hudi-*-$HUDI_VERSION.jar
$TMP_JARS_DIR/
-echo 'Validating jars below:'
+
+if [[ "$HUDI_VERSION" == *"SNAPSHOT" ]]; then
+ cp
${GITHUB_WORKSPACE}/packaging/hudi-flink-bundle/target/hudi-*-$HUDI_VERSION.jar
$TMP_JARS_DIR/
+ cp
${GITHUB_WORKSPACE}/packaging/hudi-hadoop-mr-bundle/target/hudi-*-$HUDI_VERSION.jar
$TMP_JARS_DIR/
+ cp
${GITHUB_WORKSPACE}/packaging/hudi-kafka-connect-bundle/target/hudi-*-$HUDI_VERSION.jar
$TMP_JARS_DIR/
+ cp
${GITHUB_WORKSPACE}/packaging/hudi-spark-bundle/target/hudi-*-$HUDI_VERSION.jar
$TMP_JARS_DIR/
+ cp
${GITHUB_WORKSPACE}/packaging/hudi-utilities-bundle/target/hudi-*-$HUDI_VERSION.jar
$TMP_JARS_DIR/
+ cp
${GITHUB_WORKSPACE}/packaging/hudi-utilities-slim-bundle/target/hudi-*-$HUDI_VERSION.jar
$TMP_JARS_DIR/
+ cp
${GITHUB_WORKSPACE}/packaging/hudi-metaserver-server-bundle/target/hudi-*-$HUDI_VERSION.jar
$TMP_JARS_DIR/
+ echo 'Validating jars below:'
+else
+ echo 'Adding environment variables for bundles in the release candidate'
+
+ HUDI_HADOOP_MR_BUNDLE_NAME=hudi-hadoop-mr-bundle
+ HUDI_KAFKA_CONNECT_BUNDLE_NAME=hudi-kafka-connect-bundle
+ HUDI_METASERVER_SERVER_BUNDLE_NAME=hudi-metaserver-server-bundle
+
+ if [[ ${SPARK_PROFILE} == 'spark' ]]; then
+ HUDI_SPARK_BUNDLE_NAME=hudi-spark-bundle_2.11
+ HUDI_UTILITIES_BUNDLE_NAME=hudi-utilities-bundle_2.11
+ HUDI_UTILITIES_SLIM_BUNDLE_NAME=hudi-utilities-slim-bundle_2.11
+ elif [[ ${SPARK_PROFILE} == 'spark2.4' ]]; then
+ HUDI_SPARK_BUNDLE_NAME=hudi-spark2.4-bundle_2.11
+ HUDI_UTILITIES_BUNDLE_NAME=hudi-utilities-bundle_2.11
+ HUDI_UTILITIES_SLIM_BUNDLE_NAME=hudi-utilities-slim-bundle_2.11
+ elif [[ ${SPARK_PROFILE} == 'spark3.1' ]]; then
+ HUDI_SPARK_BUNDLE_NAME=hudi-spark3.1-bundle_2.12
+ HUDI_UTILITIES_BUNDLE_NAME=hudi-utilities-bundle_2.12
+ HUDI_UTILITIES_SLIM_BUNDLE_NAME=hudi-utilities-slim-bundle_2.12
+ elif [[ ${SPARK_PROFILE} == 'spark3.2' ]]; then
+ HUDI_SPARK_BUNDLE_NAME=hudi-spark3.2-bundle_2.12
+ HUDI_UTILITIES_BUNDLE_NAME=hudi-utilities-bundle_2.12
+ HUDI_UTILITIES_SLIM_BUNDLE_NAME=hudi-utilities-slim-bundle_2.12
+ elif [[ ${SPARK_PROFILE} == 'spark3.3' ]]; then
+ HUDI_SPARK_BUNDLE_NAME=hudi-spark3.3-bundle_2.12
+ HUDI_UTILITIES_BUNDLE_NAME=hudi-utilities-bundle_2.12
+ HUDI_UTILITIES_SLIM_BUNDLE_NAME=hudi-utilities-slim-bundle_2.12
+ elif [[ ${SPARK_PROFILE} == 'spark3' ]]; then
+ HUDI_SPARK_BUNDLE_NAME=hudi-spark3-bundle_2.12
+ HUDI_UTILITIES_BUNDLE_NAME=hudi-utilities-bundle_2.12
+ HUDI_UTILITIES_SLIM_BUNDLE_NAME=hudi-utilities-slim-bundle_2.12
+ fi
+
+ if [[ ${FLINK_PROFILE} == 'flink1.13' ]]; then
+ HUDI_FLINK_BUNDLE_NAME=hudi-flink1.13-bundle
+ elif [[ ${FLINK_PROFILE} == 'flink1.14' ]]; then
+ HUDI_FLINK_BUNDLE_NAME=hudi-flink1.14-bundle
+ elif [[ ${FLINK_PROFILE} == 'flink1.15' ]]; then
+ HUDI_FLINK_BUNDLE_NAME=hudi-flink1.15-bundle
+ elif [[ ${FLINK_PROFILE} == 'flink1.16' ]]; then
+ HUDI_FLINK_BUNDLE_NAME=hudi-flink1.16-bundle
+ elif [[ ${FLINK_PROFILE} == 'flink1.17' ]]; then
+ HUDI_FLINK_BUNDLE_NAME=hudi-flink1.17-bundle
+ fi
+
+ echo "Downloading bundle jars from staging repo
orgapachehudi-$STAGING_REPO_NUM ..."
+
REPO_BASE_URL=https://repository.apache.org/content/repositories/orgapachehudi-$STAGING_REPO_NUM/org/apache/hudi
+ wget -q
$REPO_BASE_URL/$HUDI_FLINK_BUNDLE_NAME/$HUDI_VERSION/$HUDI_FLINK_BUNDLE_NAME-$HUDI_VERSION.jar
-P $TMP_JARS_DIR/
+ wget -q
$REPO_BASE_URL/$HUDI_HADOOP_MR_BUNDLE_NAME/$HUDI_VERSION/$HUDI_HADOOP_MR_BUNDLE_NAME-$HUDI_VERSION.jar
-P $TMP_JARS_DIR/
+ wget -q
$REPO_BASE_URL/$HUDI_KAFKA_CONNECT_BUNDLE_NAME/$HUDI_VERSION/$HUDI_KAFKA_CONNECT_BUNDLE_NAME-$HUDI_VERSION.jar
-P $TMP_JARS_DIR/
+ wget -q
$REPO_BASE_URL/$HUDI_SPARK_BUNDLE_NAME/$HUDI_VERSION/$HUDI_SPARK_BUNDLE_NAME-$HUDI_VERSION.jar
-P $TMP_JARS_DIR/
+ wget -q
$REPO_BASE_URL/$HUDI_UTILITIES_BUNDLE_NAME/$HUDI_VERSION/$HUDI_UTILITIES_BUNDLE_NAME-$HUDI_VERSION.jar
-P $TMP_JARS_DIR/
+ wget -q
$REPO_BASE_URL/$HUDI_UTILITIES_SLIM_BUNDLE_NAME/$HUDI_VERSION/$HUDI_UTILITIES_SLIM_BUNDLE_NAME-$HUDI_VERSION.jar
-P $TMP_JARS_DIR/
+ wget -q
$REPO_BASE_URL/$HUDI_METASERVER_SERVER_BUNDLE_NAME/$HUDI_VERSION/$HUDI_METASERVER_SERVER_BUNDLE_NAME-$HUDI_VERSION.jar
-P $TMP_JARS_DIR/
+ echo "Downloaded these jars from $REPO_BASE_URL for validation:"
Review Comment:
do you think we should add a count validation on the output of
```
ls -l $TMP_JARS_DIR
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
