Hi All, We are trying to enable encryption between spark-shuffle and local FileSystem. We wanted to clarify our understanding on this. Currently we're working on Spark 2.4
According to our understanding of Spark supporting Local Storage Encryption, that is, "Enabling local disk I/O encryption", it looks like the following properties:- spark.io.encryption.enabled spark.io.encryption.keySizeBits spark.io.encryption.keygen.algorithm spark.io.encryption.commons.config.* needs to be enabled only in spark and not in spark-shuffle's configuration properties. So, on performing spark-submit using external shuffle service, only when we set ioEncryption properties enabled in the configuration used for spark-submit, we can see ioEncryption related messages in the driver log. But when we use ioEncryption properties enabled only in spark-shuffle's configuration we do not see any ioEncryption related messages in shuffle logs. We have followed the below links:- https://spark.apache.org/docs/latest/security.html https://dzone.com/articles/securing-apache-spark-shuffle-using-apache-commons https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.0/configuring-spark/content/configuring_spark_for_wire_encryption.html https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-SerializerManager.html Can you please clarify on this if our understanding that ioEncryption related properties needs to be enabled only in spark is correct? Thanks. Breeta
