Ngone51 commented on a change in pull request #31876:
URL: https://github.com/apache/spark/pull/31876#discussion_r617519168
##########
File path: core/src/main/scala/org/apache/spark/TaskEndReason.scala
##########
@@ -81,7 +81,7 @@ case object Resubmitted extends TaskFailedReason {
*/
@DeveloperApi
case class FetchFailed(
- bmAddress: BlockManagerId, // Note that bmAddress can be null
+ bmAddress: Location, // Note that bmAddress can be null
Review comment:
> a) change to public api (programmatic, REST api, etc)
Technically, it's really a problem. Although, I can't imagine how users
would use it as an API.
And I have a new idea that we can introduce a new fetch failed class for the
custom location and leave this one unchanged. For example, we can have
`CustomStorageFetchFailed`. Thus, we the location is `BlockManagerId ` then we
use `FetchFailed`, otherwise, uses `CustomStorageFetchFailed`. WDYT?
> b) change to generated/persisted data.
I think we won't change the data of `BlockManagerId` here. If we find the
`Location` is a `BlockManagerId`, we'd still output as `("Block Manager
Address" -> blockManagerAddress)`. So, IIUC, it won't cause problems.
The only problem is the custom location. It's new data, e.g.,
`("XXXLocation" -> XXXLocationJson)`. So it can be a problem if users use the
old version Spark to load event files. Although, I think this's really an
unexpected usage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
