[
https://issues.apache.org/jira/browse/SPARK-27742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16851828#comment-16851828
]
Stavros Kontopoulos edited comment on SPARK-27742 at 5/30/19 12:57 PM:
-----------------------------------------------------------------------
[~gsomogyi] what happens with Kafka delegation tokens after max life time.
Streaming jobs in production may run for many days without interruption. From
what I see in the code `expiryTimestamp` is used to calculate the new time for
renewal but no option for setting the max life time at least, it defaults to
maxLifeTimeMs = -1L at
[https://github.com/apache/spark/blob/2f558094257c38d26650049f2ac93be6d65d6d85/external/kafka-0-10-token-provider/src/main/scala/org/apache/spark/kafka010/KafkaTokenUtil.scala#L65]
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka]
Which means pickup whatever the server has configured.
CreateDelegationTokenOptions also allows you to pass a principal.
was (Author: skonto):
[~gsomogyi] what happens with Kafka delegation tokens after max life time.
Streaming jobs in production may run for many days without interruption. From
what I see in the code `expiryTimestamp` is used to calculate the new time for
renewal but no option for setting the max life time at least, it defaults to
maxLifeTimeMs = -1L at
[https://github.com/apache/spark/blob/2f558094257c38d26650049f2ac93be6d65d6d85/external/kafka-0-10-token-provider/src/main/scala/org/apache/spark/kafka010/KafkaTokenUtil.scala#L65]
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka]
CreateDelegationTokenOptions also allows you to pass a principal.
> Security Support in Sources and Sinks for SS and Batch
> ------------------------------------------------------
>
> Key: SPARK-27742
> URL: https://issues.apache.org/jira/browse/SPARK-27742
> Project: Spark
> Issue Type: Brainstorming
> Components: SQL, Structured Streaming
> Affects Versions: 3.0.0
> Reporter: Stavros Kontopoulos
> Priority: Major
>
> As discussed with [~erikerlandson] on the [Big Data on K8s
> UG|https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA]
> it would be good to capture current status and identify work that needs to
> be done for securing Spark when accessing sources and sinks. For example what
> is the status of SSL, Kerberos support in different scenarios. The big
> concern nowadays is how to secure data pipelines end-to-end.
> Note: Not sure if this overlaps with some other ticket.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
