Marcelo Vanzin created SPARK-19520:
--------------------------------------
Summary: WAL should not be encrypted
Key: SPARK-19520
URL: https://issues.apache.org/jira/browse/SPARK-19520
Project: Spark
Issue Type: Bug
Components: DStreams
Affects Versions: 2.1.0
Reporter: Marcelo Vanzin
If I/O encryption is enabled in Spark (SPARK-5682), the write ahead logs
written by streaming applications will also be encrypted.
That creates issues because only the driver that wrote those files knows the
key to decrypt them. So basically the WAL is unusuable in that scenario.
Instead, the WAL should not be encrypted by Spark; if the user has a need to
encrypt that data, other means (such as HDFS encryption zones) should be used
instead.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
