Ngone51 commented on a change in pull request #31876:
URL: https://github.com/apache/spark/pull/31876#discussion_r618528752
##########
File path: core/src/main/scala/org/apache/spark/TaskEndReason.scala
##########
@@ -81,7 +81,7 @@ case object Resubmitted extends TaskFailedReason {
*/
@DeveloperApi
case class FetchFailed(
- bmAddress: BlockManagerId, // Note that bmAddress can be null
+ bmAddress: Location, // Note that bmAddress can be null
Review comment:
I think we'll have to add interfaces to `Location` in order to support
json serde , e.g.,
```java
public JValue serializeToJson();
public Location deserilaizeFromJson(json: JValue);
```
> How to handle different shuffle impls being in play for the same event
directory.
Adding metadata is good idea, we can have the format like,
```json
"Mapstatus Location": {
"type": "xx.yy.zz", // qualified class name
"content": { // content is generate by `Location.serializeToJson`
"aaa":"bbb"
}
}
```
with the constant format, end-users and SHS are able to parse them as well.
(I had an idea about SHS is to add the location type as the extension of the
event log file. That's the way what compression does now. But I think it
doesn't solve the problem of REST case.)
BTW, I have added https://issues.apache.org/jira/browse/SPARK-35188 for this
support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
