Ngone51 commented on pull request #32114: URL: https://github.com/apache/spark/pull/32114#issuecomment-826114665
So I think the solution now would be: For the heartbeat, using https://github.com/apache/spark/pull/32114#pullrequestreview-634766324. For the`Blokmanager`, adding a `BlockManagerEndpointSharedState` (as mentioned by @sumeetgajjar in https://github.com/apache/spark/pull/32114#discussion_r612814973) for both `BlockManagerMasterEndpoint` and `BlockManagerMasterHeartbeatEndpoint`. It's true that if we only adding a removal state to the `blockManagerInfo`, we have to filter the removed blockmanagers first before traversing it (e.g., we won't expect to return a removed blockmanager in `getPeers`). In `BlockManagerEndpointSharedState`, we'd have both `activeBlockManagerInfo` and the `removedBlockManagerInfo`. We don't have to set up a new cleaner to clear the `removedBlockManagerInfo`. Instead, we can reuse the fix of `HeartbeatReceiver` as whenever there's a sure removal in `HeartbeatReceiver`, we can send a removal request to `BlockManagerEndpointSharedState` as well by following the code path of `!scheduler.executorHeartbeatReceived`(e.g., we could have `scheduler.clearBlockManagerInfo` similarly). WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
