[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14122064#comment-14122064
]
Thomas Graves commented on SPARK-3129:
--------------------------------------
On yarn, it generates the secret automatically. In cluster mode, it does it in
the applicationMaster. Since it generates it in the applicationmaster, it goes
away when the application master dies. If the secret was generated on the
client side and populated into the credentials in the UGI similar to how we do
tokens then a restart of the AM in cluster mode should be able to pick it back
up.
This won't work for client mode though since the client/spark driver wouldn't
have a way to get ahold of the UGI again.
> Prevent data loss in Spark Streaming
> ------------------------------------
>
> Key: SPARK-3129
> URL: https://issues.apache.org/jira/browse/SPARK-3129
> Project: Spark
> Issue Type: New Feature
> Reporter: Hari Shreedharan
> Assignee: Hari Shreedharan
> Attachments: StreamingPreventDataLoss.pdf
>
>
> Spark Streaming can small amounts of data when the driver goes down - and the
> sending system cannot re-send the data (or the data has already expired on
> the sender side). The document attached has more details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
