roczei commented on code in PR #47795:
URL: https://github.com/apache/spark/pull/47795#discussion_r1742198067
##########
hadoop-cloud/pom.xml:
##########
@@ -171,6 +171,41 @@
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-cos</artifactId>
</exclusion>
+ <!--
+ HADOOP-19224 / SPARK-48867:
com.huaweicloud:esdk-obs-java:jar:3.20.4.2 is
+ vulnerable due to okhttp 3.x (CVE-2023-0833, CVE-2021-0341),
+ it has to be upgraded to 3.24.3 which depends on okhttp 4.12.0
+ -->
+ <exclusion>
+ <groupId>com.huaweicloud</groupId>
+ <artifactId>esdk-obs-java</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
Review Comment:
Sorry for the late reply
> will it actually work with this removal?
I do not have access to Huawei Cloud, therefore I could not test it with an
obs bucket. Found this documentation for Spark testing:
https://support.huaweicloud.com/intl/en-us/devg-dli/dli_09_0205.html#section6
> I don't think apache spark have any tests for this, does hadoop have it
I did some research and found only Hadoop unit tests but these are disabled
by default. Related documentation:
https://github.com/apache/hadoop/blob/trunk/hadoop-cloud-storage-project/hadoop-huaweicloud/src/site/markdown/index.md#testing-the-hadoop-huaweicloud-module
Similar configuration has to be created if somebody has such credentials:
```
$ cat src/test/resources/auth-keys.xml
<configuration>
<property>
<name>fs.contract.test.fs.obs</name>
<value>obs://testobscontract</value>
</property>
<property>
<name>fs.obs.access.key</name>
<value>secret</value>
</property>
<property>
<name>fs.obs.secret.key</name>
<value>secret</value>
</property>
</configuration>
$
```
Just to be on the safe side, I agree that we should do what @steveloughran
suggested above:
> if not best to stop trying to restore it and exclude all huaweicloud
support with the release note/spark docs saying "explicitly import it"
@pan3793 / @panbingkun / @dongjoon-hyun / @melin / @bjornjorgensen what
is your opinion about this suggestion? If you agre as well, I would like to
implement these:
- Exclude the whole org.apache.hadoop:hadoop-huaweicloud artifact instead of
com.huaweicloud:esdk-obs-java
- Update the "Does this PR introduce any user-facing change" section and
mention that it includes this user facing change
- Request to add it to the release notes but I do not know what is the
proper way to do it in case of Apache Spark. Please share with me if you know
the official process for this. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
