mridulm commented on code in PR #37922:
URL: https://github.com/apache/spark/pull/37922#discussion_r1067551485
##########
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalBlockStoreClient.java:
##########
@@ -256,6 +256,22 @@ public void onFailure(Throwable e) {
}
}
+ @Override
+ public boolean removeShuffleMerge(String host, int port, int shuffleId, int
shuffleMergeId) {
+ checkInit();
+ try {
+ TransportClient client = clientFactory.createClient(host, port);
+ client.send(
+ new RemoveShuffleMerge(appId, comparableAppAttemptId, shuffleId,
shuffleMergeId)
Review Comment:
Failure to remove blocks in `removeBlocks` has an impact on resource
utilization (for example, block could be in memory). Here, it is more of a best
case cleanup - and can fail in context of decommissioned nodes, etc - log
pollution is something I am concerned with here.
Having said that - the exact same problem exists for
`removeShuffle.removeShuffleFromShuffleServicesFutures` unfortunately - and
that is an issue : I would argue for reducing that log message from `warn` to
`info` or even `debug` as well for failure when removing shuffle files from ESS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
