[
https://issues.apache.org/jira/browse/SPARK-24036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16489700#comment-16489700
]
Arun Mahadevan commented on SPARK-24036:
----------------------------------------
If I understand correctly, continuous job would have a single stage with tasks
running at the same time shuffling data around (making use of the "TaskInfo" to
figure out the endpoints). This means we cannot re-use the existing shuffle
infra since it makes sense only if there are multiple stages ? Does SPARK-24374
plan to provide the shuffle infra to move data around or is that out of scope ?
> Stateful operators in continuous processing
> -------------------------------------------
>
> Key: SPARK-24036
> URL: https://issues.apache.org/jira/browse/SPARK-24036
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 2.4.0
> Reporter: Jose Torres
> Priority: Major
>
> The first iteration of continuous processing in Spark 2.3 does not work with
> stateful operators.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
