[
https://issues.apache.org/jira/browse/SPARK-43179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen resolved SPARK-43179.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 41502
[https://github.com/apache/spark/pull/41502]
> Add option for applications to control saving of metadata in the External
> Shuffle Service LevelDB
> -------------------------------------------------------------------------------------------------
>
> Key: SPARK-43179
> URL: https://issues.apache.org/jira/browse/SPARK-43179
> Project: Spark
> Issue Type: Improvement
> Components: Shuffle
> Affects Versions: 3.4.0
> Reporter: Chandni Singh
> Assignee: Chandni Singh
> Priority: Major
> Fix For: 3.5.0
>
>
> Currently, the External Shuffle Service stores application metadata in
> LevelDB. This is necessary to enable the shuffle server to resume serving
> shuffle data for an application whose executors registered before the
> NodeManager restarts. However, the metadata includes the application secret,
> which is stored in LevelDB without encryption. This is a potential security
> risk, particularly for applications with high security requirements. While
> filesystem access control lists (ACLs) can help protect keys and
> certificates, they may not be sufficient for some use cases. In response, we
> have decided not to store metadata for these high-security applications in
> LevelDB. As a result, these applications may experience more failures in the
> event of a node restart, but we believe this trade-off is acceptable given
> the increased security risk.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
