This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 864ff67 [SPARK-35429][CORE] Remove commons-httpclient from Hadoop-3.2 profile due to EOL and CVEs 864ff67 is described below commit 864ff677469172ca24fdef69b7d3a3482c688f47 Author: Sumeet Gajjar <sumeetgajja...@gmail.com> AuthorDate: Tue Jun 15 14:43:30 2021 -0700 [SPARK-35429][CORE] Remove commons-httpclient from Hadoop-3.2 profile due to EOL and CVEs ### What changes were proposed in this pull request? Remove commons-httpclient as a direct dependency for Hadoop-3.2 profile. Hadoop-2.7 profile distribution still has it, hadoop-client has a compile dependency on commons-httpclient, thus we cannot remove it for Hadoop-2.7 profile. ``` [INFO] +- org.apache.hadoop:hadoop-client:jar:2.7.4:compile [INFO] | +- org.apache.hadoop:hadoop-common:jar:2.7.4:compile [INFO] | | +- commons-cli:commons-cli:jar:1.2:compile [INFO] | | +- xmlenc:xmlenc:jar:0.52:compile [INFO] | | +- commons-httpclient:commons-httpclient:jar:3.1:compile ``` ### Why are the changes needed? Spark is pulling in commons-httpclient as a dependency directly. commons-httpclient went EOL years ago and there are most likely CVEs not being reported against it, thus we should remove it. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Existing unittests - Checked the dependency tree before and after introducing the changes Before: ``` ./build/mvn dependency:tree -Phadoop-3.2 | grep -i "commons-httpclient" Using `mvn` from path: /usr/bin/mvn [INFO] +- commons-httpclient:commons-httpclient:jar:3.1:compile [INFO] | +- commons-httpclient:commons-httpclient:jar:3.1:provided ``` After ``` ./build/mvn dependency:tree | grep -i "commons-httpclient" Using `mvn` from path: /Users/sumeet.gajjar/cloudera/upstream-spark/build/apache-maven-3.6.3/bin/mvn ``` P.S. Reopening this since [spark upgraded](https://github.com/apache/spark/commit/463daabd5afd9abfb8027ebcb2e608f169ad1e40) its `hive.version` to `2.3.9` which does not have a dependency on `commons-httpclient`. Closes #32912 from sumeetgajjar/SPARK-35429. Authored-by: Sumeet Gajjar <sumeetgajja...@gmail.com> Signed-off-by: Dongjoon Hyun <dongj...@apache.org> --- dev/deps/spark-deps-hadoop-3.2-hive-2.3 | 1 - pom.xml | 11 ----------- sql/hive/pom.xml | 4 ---- 3 files changed, 16 deletions(-) diff --git a/dev/deps/spark-deps-hadoop-3.2-hive-2.3 b/dev/deps/spark-deps-hadoop-3.2-hive-2.3 index 8b79d7e5..3482dd2 100644 --- a/dev/deps/spark-deps-hadoop-3.2-hive-2.3 +++ b/dev/deps/spark-deps-hadoop-3.2-hive-2.3 @@ -35,7 +35,6 @@ commons-compiler/3.1.4//commons-compiler-3.1.4.jar commons-compress/1.20//commons-compress-1.20.jar commons-crypto/1.1.0//commons-crypto-1.1.0.jar commons-dbcp/1.4//commons-dbcp-1.4.jar -commons-httpclient/3.1//commons-httpclient-3.1.jar commons-io/2.8.0//commons-io-2.8.0.jar commons-lang/2.6//commons-lang-2.6.jar commons-lang3/3.12.0//commons-lang3-3.12.0.jar diff --git a/pom.xml b/pom.xml index 82a047f..ca038b2 100644 --- a/pom.xml +++ b/pom.xml @@ -157,8 +157,6 @@ <!-- org.apache.httpcomponents/httpclient--> <commons.httpclient.version>4.5.13</commons.httpclient.version> <commons.httpcore.version>4.4.14</commons.httpcore.version> - <!-- commons-httpclient/commons-httpclient--> - <httpclient.classic.version>3.1</httpclient.classic.version> <commons.math3.version>3.4.1</commons.math3.version> <!-- managed up from 3.2.1 for SPARK-11652 --> <commons.collections.version>3.2.2</commons.collections.version> @@ -593,11 +591,6 @@ <version>${jsr305.version}</version> </dependency> <dependency> - <groupId>commons-httpclient</groupId> - <artifactId>commons-httpclient</artifactId> - <version>${httpclient.classic.version}</version> - </dependency> - <dependency> <groupId>org.apache.httpcomponents</groupId> <artifactId>httpclient</artifactId> <version>${commons.httpclient.version}</version> @@ -1811,10 +1804,6 @@ <artifactId>commons-codec</artifactId> </exclusion> <exclusion> - <groupId>commons-httpclient</groupId> - <artifactId>commons-httpclient</artifactId> - </exclusion> - <exclusion> <groupId>org.apache.avro</groupId> <artifactId>avro-mapred</artifactId> </exclusion> diff --git a/sql/hive/pom.xml b/sql/hive/pom.xml index 729d3f4..67a9854 100644 --- a/sql/hive/pom.xml +++ b/sql/hive/pom.xml @@ -134,10 +134,6 @@ <artifactId>avro-mapred</artifactId> </dependency> <dependency> - <groupId>commons-httpclient</groupId> - <artifactId>commons-httpclient</artifactId> - </dependency> - <dependency> <groupId>org.apache.httpcomponents</groupId> <artifactId>httpclient</artifactId> </dependency> --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org