Nikita ROUSSEAU created SPARK-35954:
---------------------------------------
Summary: Upgrade Apache Curator Dependency to 4.2.0
Key: SPARK-35954
URL: https://issues.apache.org/jira/browse/SPARK-35954
Project: Spark
Issue Type: Dependency upgrade
Components: Deploy
Affects Versions: 3.1.2
Environment: * OS: Linux
* JAVA: 1.8.0_292
* {color:#FF0000}*hadoop-3.3.1*{color}
Reporter: Nikita ROUSSEAU
+Abstract :+ as a Spark Cluster Administrator, I want to connect spark masters
deployed in HA/mode to Zookeeper over SSL/TLS, so that my network traffic is
ciphered between y components.
([https://spark.apache.org/docs/latest/spark-standalone.html#standby-masters-with-zookeeper]
)
With the release of Hadoop 3.3.1, ZKFC libraries and their dependencies were
updated : it is now possible to connect ZKFC to ZooKeeper over TLS.
+Note:+ TLS is possible with Zookeeper Server Version >= 3.5.6
([https://docs.confluent.io/platform/current/installation/versions-interoperability.html#zk]
)
Spark 3.2.0 aims to support Hadoop 3.3.1 ; this Hadoop release bundles the
following shared libraries :
* curator-client-4.2.0.jar
* curator-framework-4.2.0.jar
* curator-recipes-4.2.0.jar
* zookeeper-3.5.6.jar
* zookeeper-jute-3.5.6.jar
Currently, Spark dependency is set to 2.13.0 for the Currator framework
([https://github.com/apache/spark/blob/master/pom.xml#L127).|https://github.com/apache/spark/blob/master/pom.xml#L127)]
It would be great to update "curator-*" dependencies to 4.2.0 in order to be
compatible with shared jars of the hadoop stack.
Moreover, it will allow administrators to connect Spark Masters to ZooKeeper
over TLS.
Some patches will be required, such as :
*
[https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkCuratorUtil.scala#L51]
I will try to prepare a MR for this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
