dongjoon-hyun opened a new pull request #24445: [SPARK-27550][TEST][BRANCH-2.4] Fix `test-dependencies.sh` not to use `kafka-0-8`profile for Scala-2.12 URL: https://github.com/apache/spark/pull/24445 ## What changes were proposed in this pull request? Since SPARK-27274 deprecated Scala-2.11 at Spark 2.4.1, we need to test Scala-2.12 more. Kakfa 0.8 doesn't have Scala-2.12 artifacts, e.g., `org.apache.kafka:kafka_2.12:jar:0.8.2.1`. This PR aims to fix `test-dependencies.sh` script to understand Scala binary version. ``` $ dev/change-scala-version.sh 2.12 $ dev/test-dependencies.sh Using `mvn` from path: /usr/local/bin/mvn Using `mvn` from path: /usr/local/bin/mvn Performing Maven install for hadoop-2.6 Using `mvn` from path: /usr/local/bin/mvn [ERROR] Failed to execute goal on project spark-streaming-kafka-0-8_2.12: Could not resolve dependencies for project org.apache.spark:spark-streaming-kafka-0-8_2.12:jar:spark-335572: Could not find artifact org.apache.kafka:kafka_2.12:jar:0.8.2.1 in central (https://repo.maven.apache.org/maven2) -> [Help 1] ``` ## How was this patch tested? Manually do `dev/change-scala-version.sh 2.12` and `dev/test-dependencies.sh`. The script should show `DO NOT MATCH` message instead of Maven `[ERROR]`. ``` $ dev/test-dependencies.sh Using `mvn` from path: /usr/local/bin/mvn ... Generating dependency manifest for hadoop-3.1 Using `mvn` from path: /usr/local/bin/mvn Spark's published dependencies DO NOT MATCH the manifest file (dev/spark-deps). To update the manifest file, run './dev/test-dependencies.sh --replace-manifest'. diff --git a/dev/deps/spark-deps-hadoop-2.6 b/dev/pr-deps/spark-deps-hadoop-2.6 ... ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
