[
https://issues.apache.org/jira/browse/SPARK-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14285266#comment-14285266
]
Hari Shreedharan commented on SPARK-5342:
-----------------------------------------
Looks like we cannot write to the cache in such a way that the container can
see it, once we start the app. So we'd have to use the HTTP server. With
authentication (which Spark does automatically with YARN), and SSL, we should
be good.
> Allow long running Spark apps to run on secure YARN/HDFS
> --------------------------------------------------------
>
> Key: SPARK-5342
> URL: https://issues.apache.org/jira/browse/SPARK-5342
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 1.2.0
> Reporter: Hari Shreedharan
> Attachments: SparkYARN.pdf
>
>
> Currently, Spark apps cannot write to HDFS after the delegation tokens reach
> their expiry, which maxes out at 7 days. We must find a way to ensure that we
> can run applications for longer - for example, spark streaming apps are
> expected to run forever.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
