[
https://issues.apache.org/jira/browse/SPARK-22634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398659#comment-16398659
]
Steve Loughran commented on SPARK-22634:
----------------------------------------
moving to jets3t 0.9.4 breaks the (legacy) s3 & s3n clients (SPARK-23562).
Filed SPARK-23654 to cut jets3t out completely. It's not going to be fixable
for spark to keep bouncy castle current and have the old FS connectors work.
Simplest to cut the JAR and have those filesystems fail fast.
If this patch is backported, the release notes should tell people to switch to
s3a. Which they can only safely do with Hadoop 2.7.x+
or, put differently: this patch stops anyone with the ASF Hadoop-2.6 set of
jars in their packaging from talking to s3.
> Update Bouncy castle dependency
> -------------------------------
>
> Key: SPARK-22634
> URL: https://issues.apache.org/jira/browse/SPARK-22634
> Project: Spark
> Issue Type: Task
> Components: Spark Core, SQL, Structured Streaming
> Affects Versions: 2.2.0
> Reporter: Lior Regev
> Assignee: Sean Owen
> Priority: Minor
> Fix For: 2.3.0
>
>
> Spark's usage of jets3t library as well as Spark's own Flume and Kafka
> streaming uses bouncy castle version 1.51
> This is an outdated version as the latest one is 1.58
> This, in turn renders packages such as
> [spark-hadoopcryptoledger-ds|https://github.com/ZuInnoTe/spark-hadoopcryptoledger-ds]
> unusable since these require 1.58 and spark's distributions come along with
> 1.51
> My own attempt was to run on EMR, and since I automatically get all of
> spark's dependecies (bouncy castle 1.51 being one of them) into the
> classpath, using the library to parse blockchain data failed due to missing
> functionality.
> I have also opened an
> [issue|https://bitbucket.org/jmurty/jets3t/issues/242/bouncycastle-dependency]
> with jets3t to update their dependecy as well, but along with that Spark
> would have to update it's own or at least be packaged with a newer version
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
