Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/4688#discussion_r29286123
--- Diff: docs/security.md ---
@@ -32,6 +32,8 @@ SSL must be configured on each node and configured for
each component involved i
### YARN mode
The key-store can be prepared on the client side and then distributed and
used by the executors as the part of the application. It is possible because
the user is able to deploy files before the application is started in YARN by
using `spark.yarn.dist.files` or `spark.yarn.dist.archives` configuration
settings. The responsibility for encryption of transferring these files is on
YARN side and has nothing to do with Spark.
+For long-running apps like Spark Streaming apps to be able to write to
HDFS, it is possible to pass a principal and keytab to `spark-submit` via the
`--principal` and `--keytab` parameters respectively. The keytab passed in will
be copied over to the machine running the Application Master via the Hadoop
Distributed Cache (securely - if YARN is configured with SSL and HDFS
encryption is enabled). The Kerberos login will be periodically renewed using
this principal and keytab and the delegation tokens required for HDFS will be
generated periodically so the application can continue writing to HDFS. Please
note that the HDFS client configuration file, `hdfs-site.xml` on each executor
node must have the value of `dfs.namenode.delegation.token.renew-interval` be
the same as it is on the HDFS Namenode for this functionality.
+
--- End diff --
remove the reference to the renew-interval if you remove use of it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
