zhangxinyao88 opened a new pull request, #57872:
URL: https://github.com/apache/spark/pull/57872
### What changes were proposed in this pull request?
Remove the direct `hadoop-huaweicloud` dependency from `hadoop-cloud`,
together with its
explicit OkHttp and Okio dependencies, version properties, dependency scope
property, and
`huaweicloud-provided` profile.
Keep the `hadoop-huaweicloud` exclusion on `hadoop-cloud-storage` so that
the connector is not
reintroduced transitively. Update `LICENSE-binary` and the Hadoop 3
dependency manifest to
remove the three JARs.
Closes #57794.
### Why are the changes needed?
Spark has no source code, tests, or documentation that use the Huawei OBS
connector. Bundling
it pulls in the obsolete OkHttp 3.12.12 and Okio 1.17.6 dependency tree,
including an OkHttp
version affected by CVE-2021-0341. Removing the unused connector eliminates
that dependency
tree without changing Spark's supported default paths.
### Does this PR introduce _any_ user-facing change?
Yes. Spark distributions built with the Hadoop cloud module no longer bundle
`hadoop-huaweicloud`, OkHttp 3.12.12, or Okio 1.17.6. Spark APIs and
documented/default cloud
storage paths are unchanged.
### How was this patch tested?
No unit tests were added because this patch only removes Maven and
binary-distribution metadata.
It was validated with:
```bash
./build/mvn -Phadoop-3,hadoop-cloud -pl hadoop-cloud help:effective-pom \
-Doutput=/private/tmp/spark-hadoop-cloud-effective-pom.xml
./build/mvn -Phadoop-3,hadoop-cloud,hadoop-provided -pl hadoop-cloud
help:effective-pom \
-Doutput=/private/tmp/spark-hadoop-cloud-provided-effective-pom.xml -q
xmllint --noout pom.xml hadoop-cloud/pom.xml
LC_ALL=C sort -c dev/deps/spark-deps-hadoop-3-hive-2.3
git diff --check upstream/master
```
Both effective POMs contain zero direct `hadoop-huaweicloud`, `okhttp`, or
`okio`
dependencies, retain exactly one `hadoop-huaweicloud` exclusion, and contain
no
`huaweicloud-provided` profile or `huaweicloud.deps.scope` property.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (codex-cli 0.146.0)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
