[
https://issues.apache.org/jira/browse/SPARK-31329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17088569#comment-17088569
]
Dale Richardson edited comment on SPARK-31329 at 4/21/20, 11:36 AM:
--------------------------------------------------------------------
Hi Holden,
I've been thinking about this issue as well.
Blocks can't move, but they can be replicated. Any reason we can't just
replicate the blocks out, allow the existing code paths to update the block
locations with the block manager master, then unregister the current blocks?
We should chat and coordinate efforts.
was (Author: tigerquoll):
Hi Holden,
I've been thinking about this issue as well.
Blocks can't move, but they can be replicated. Any reason we can't just
replicate the blocks out, allow the existing code paths to update the block
locations with the block manager master, then unregister the current blocks?
> Modify executor monitor to allow for moving shuffle blocks
> ----------------------------------------------------------
>
> Key: SPARK-31329
> URL: https://issues.apache.org/jira/browse/SPARK-31329
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes, Spark Core
> Affects Versions: 3.1.0
> Reporter: Holden Karau
> Assignee: Holden Karau
> Priority: Minor
>
> To enable Spark-20629 we need to revisit code that assumes shuffle blocks
> don't move. Currently, the executor monitor assumes that shuffle blocks are
> immovable. We should modify this code.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
