Daniel Lanza García created SPARK-20693:
-------------------------------------------
Summary: Kafka+SSL: path for security related files needs to be
different for driver and executors
Key: SPARK-20693
URL: https://issues.apache.org/jira/browse/SPARK-20693
Project: Spark
Issue Type: Bug
Components: Input/Output
Affects Versions: 2.1.1
Reporter: Daniel Lanza García
Priority: Critical
When consuming/producing from Kafka with security enable (SSL), you need to
refer to security related files (keystore and truststore) in the configuration
of the KafkaDirectStream.
If the scenario is YARN-client mode, you would need to distribute these files,
it can be achieved with --files argument. Now, what is the path to these files?
taking into account that driver and executors interact with Kafka.
When these files are accessed from the driver, you need to provide the local
path to them. When they are accessed from the executors, you need to provide
the name of the file that has been distributed with --files.
The problem is that you can only configure one value for the path to these
files.
Proposed configurations here:
http://www.opencore.com/blog/2017/1/spark-2-0-streaming-from-ssl-kafka-with-hdp-2-4/
works because both paths are the same (./truststore.jks). But if different, I
do not think there is a way to configure Kafka+SSL
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
