[
https://issues.apache.org/jira/browse/SPARK-35954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikita ROUSSEAU updated SPARK-35954:
------------------------------------
Target Version/s: (was: 3.2.0)
> Upgrade Apache Curator Dependency to 4.2.0
> ------------------------------------------
>
> Key: SPARK-35954
> URL: https://issues.apache.org/jira/browse/SPARK-35954
> Project: Spark
> Issue Type: Dependency upgrade
> Components: Deploy
> Affects Versions: 3.1.2
> Environment: * OS: Linux
> * JAVA: 1.8.0_292
> * {color:#FF0000}*hadoop-3.3.1*{color}
>
> Reporter: Nikita ROUSSEAU
> Priority: Major
>
> +Abstract :+ as a Spark Cluster Administrator, I want to connect spark
> masters deployed in HA/mode to Zookeeper over SSL/TLS, so that my network
> traffic is ciphered between y components.
> ([https://spark.apache.org/docs/latest/spark-standalone.html#standby-masters-with-zookeeper]
> )
>
> With the release of Hadoop 3.3.1, ZKFC libraries and their dependencies were
> updated : it is now possible to connect ZKFC to ZooKeeper over TLS.
>
> +Note:+ TLS is possible with Zookeeper Server Version >= 3.5.6
> ([https://docs.confluent.io/platform/current/installation/versions-interoperability.html#zk]
> )
>
> Spark 3.2.0 aims to support Hadoop 3.3.1 ; this Hadoop release bundles the
> following shared libraries :
> * curator-client-4.2.0.jar
> * curator-framework-4.2.0.jar
> * curator-recipes-4.2.0.jar
> * zookeeper-3.5.6.jar
> * zookeeper-jute-3.5.6.jar
>
> Currently, Spark dependency is set to 2.13.0 for the Currator framework
> ([https://github.com/apache/spark/blob/master/pom.xml#L127).|https://github.com/apache/spark/blob/master/pom.xml#L127)]
>
> It would be great to update "curator-*" dependencies to 4.2.0 in order to be
> compatible with shared jars of the hadoop stack.
> Moreover, it will allow administrators to connect Spark Masters to ZooKeeper
> over TLS.
>
> Some patches will be required, such as :
> *
> [https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkCuratorUtil.scala#L51]
>
> I will try to prepare a MR for this.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
