otterc commented on pull request #35325: URL: https://github.com/apache/spark/pull/35325#issuecomment-1023302640
> Thanks @otterc and @mridulm, I verified this patch with 2 rounds of 1TB TPC-DS query, the issue is gone. > > One minor question, can we suppress this warning message? I think it's a little bit noisy for users. > > ``` > 01-27 16:18:35 WARN [shuffle-client-8-5] scheduler.DAGScheduler@94: Exception encountered when trying to finalize shuffle merge on beta-spark2 for shuffle 189 > java.lang.RuntimeException: java.lang.RuntimeException: Shuffle merge finalize request for shuffle 189 with shuffleMergeId 0 is stale shuffle finalize request as shuffle blocks of a higher shuffleMergeId for the shuffle is already being pushed > ``` Thanks @pan3793 for verifying this change. I saw those logs on the shuffle sever as well and the commit `c606185` fixes that already. This will now be logged for a true stale finalize message. Have added a UT for it as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
