[
https://issues.apache.org/jira/browse/SPARK-23200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16610287#comment-16610287
]
Santiago Saavedra commented on SPARK-23200:
-------------------------------------------
It seems there wasn't much interest in having streaming working on k8s :)
I don't currently have an available k8s cluster in which to test the PR, but if
I get bandwidth to spawn one I'll test this myself in the next days. Otherwise,
I'll build Spark and release the Docker images in Docker Hub for someone else
with the resources to reproduce this.
I recommend you to use the twitter example and have checkpointing configured on
a s3a:// bucket.
Steps: launch with spark-submit once, wait for some checkpoint files to spawn
on the bucket, then remove the driver pod, and then re-send with spark-submit
again. Check in the logs that the driver loaded successfully and was able to
revive the workers and that the names are new (no names from the old instances
are shown in the logs as missing).
> Reset configuration when restarting from checkpoints
> ----------------------------------------------------
>
> Key: SPARK-23200
> URL: https://issues.apache.org/jira/browse/SPARK-23200
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
> Affects Versions: 2.4.0
> Reporter: Anirudh Ramanathan
> Priority: Major
>
> Streaming workloads and restarting from checkpoints may need additional
> changes, i.e. resetting properties - see
> https://github.com/apache-spark-on-k8s/spark/pull/516
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
