[
https://issues.apache.org/jira/browse/SPARK-43179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chandni Singh updated SPARK-43179:
----------------------------------
Summary: Add option for applications to control saving of metadata in
External Shuffle Service LevelDB (was: Allow applications to control whether
their metadata gets saved by the shuffle server in the db)
> Add option for applications to control saving of metadata in External Shuffle
> Service LevelDB
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-43179
> URL: https://issues.apache.org/jira/browse/SPARK-43179
> Project: Spark
> Issue Type: Improvement
> Components: Shuffle
> Affects Versions: 3.4.0
> Reporter: Chandni Singh
> Priority: Major
>
> Currently, the External Shuffle Service stores application metadata in
> LevelDB. This is necessary to enable the shuffle server to resume serving
> shuffle data for an application whose executors registered before the
> NodeManager restarts. However, the metadata includes the application secret,
> which is stored in LevelDB without encryption. This is a potential security
> risk, particularly for applications with high security requirements. While
> filesystem access control lists (ACLs) can help protect keys and
> certificates, they may not be sufficient for some use cases. In response, we
> have decided not to store metadata for these high-security applications in
> LevelDB. As a result, these applications may experience more failures in the
> event of a node restart, but we believe this trade-off is acceptable given
> the increased security risk.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
