hasnain-db opened a new pull request, #43238: URL: https://github.com/apache/spark/pull/43238
### What changes were proposed in this pull request? This PR adds the options added in https://github.com/apache/spark/pull/43220 to `SSLOptions` and `SparkTransportConf`. By adding it to the `SSLOptions` we can support inheritance of options, so settings for the UI and RPC SSL settings can be shared as much as possible. The `SparkTransportConf` changes are needed to support propagating these settings. I also make some changes to `SecurityManager` to log when this feature is enabled, and make the existing `spark.authenticate` and `spark.network.crypto` options mutually exclusive with this new settings (it would just involve double encryption then). Lastly, make these flags propagate down to when executors are launch, and allow the passwords to be sent via environment variables (similar to how it's done for an existing secret). This ensures they are not visible in plaintext, but also ensures they are available at executor startup (otherwise it can never talk to the driver/worker) ### Why are the changes needed? The propagation of these options are needed for the RPC functionality to work ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? CI Added some unit tests which I verified passed: ``` build/sbt > project core > testOnly org.apache.spark.SparkConfSuite org.apache.spark.SSLOptionsSuite org.apache.spark.SecurityManagerSuite org.apache.spark.deploy.worker.CommandUtilsSuite ``` The rest of the changes and integration were tested as part of https://github.com/apache/spark/pull/42685 ### Was this patch authored or co-authored using generative AI tooling? No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
