mridulm commented on pull request #32733: URL: https://github.com/apache/spark/pull/32733#issuecomment-852688429
I am missing something here, if a block is < `spark.shuffle.accurateBlockThreshold`, it is recorded as a small block - else it is marked as a huge block. The point of `spark.shuffle.accurateBlockThreshold` is that it is not very high, so should not cause OOM - are you configuring it to a very high value ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
