Hi Team, We are deploying spark standalone cluster and using features likes rpc authentication with spark.authenticate.secret and encryption also. We have below queries from our Security teams on this topic and need your help.
1. How do we make sure spark.authenticate.secret is not visible to end users as they can use it to authenticate from other servers as well. Placing in spark-default.conf file allow all users to know the secret. Even if used spark.authenticate.secret.file we have to give read privilege on it which is again a risk. 2. In continuation to above - Can we integrate a vault like hashicorp to store this secret and use it in runtime ? 3. Encryption when enabled for RPC connections, does it use secret mentioned in spark.authenticate.secret itself as encryption key ? Regards, Bharath ________________________________ NOTICE: All information in and attached to the e-mails below may be proprietary, confidential, privileged and otherwise protected from improper or erroneous disclosure. If you are not the sender's intended recipient, you are not authorized to intercept, read, print, retain, copy, forward, or disseminate this message. If you have erroneously received this communication, please notify the sender immediately by phone (704-758-1000) or by e-mail and destroy all copies of this message electronic, paper, or otherwise. By transmitting documents via this email: Users, Customers, Suppliers and Vendors collectively acknowledge and agree the transmittal of information via email is voluntary, is offered as a convenience, and is not a secured method of communication; Not to transmit any payment information E.G. credit card, debit card, checking account, wire transfer information, passwords, or sensitive and personal information E.G. Driver's license, DOB, social security, or any other information the user wishes to remain confidential; To transmit only non-confidential information such as plans, pictures and drawings and to assume all risk and liability for and indemnify Lowe's from any claims, losses or damages that may arise from the transmittal of documents or including non-confidential information in the body of an email transmittal. Thank you.
