mahesh kumar behera created SPARK-47702:
-------------------------------------------
Summary: Shuffle service endpoint is not removed from the
locations list when RDD block is removed form a node.
Key: SPARK-47702
URL: https://issues.apache.org/jira/browse/SPARK-47702
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 3.5.1
Reporter: mahesh kumar behera
If SHUFFLE_SERVICE_FETCH_RDD_ENABLED is set to true, driver stores both
executor end point and the external shuffle end points for a RDD block. When
the RDD is migrated, the location info is updated to add the end point
corresponds to new location and the old end point is removed. But currently,
only the executor end point is removed. The shuffle service end point is not
removed. This cause failure during RDD read if the shuffle service end point is
chosen due to task locality.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
