exmy commented on pull request #32733: URL: https://github.com/apache/spark/pull/32733#issuecomment-852849714
> I am missing something here, if a block is < `spark.shuffle.accurateBlockThreshold`, it is recorded as a small block - else it is marked as a huge block. The point of `spark.shuffle.accurateBlockThreshold` is that it is not very high, so should not cause OOM - are you configuring it to a very high value ? > Note that this gets triggered when (typically) you have > 2k partitions - so the benefits of using `HighlyCompressedMapStatus` is to prevent other issues which more accurate tracking ends up with for very large number of partitions. Thanks for review. In our production, `spark.shuffle.accurateBlockThreshold` is set to 10M. We encounted a OOM case, for example, assume that map stage has 3k tasks and shuffle partitions are also 3k, most of output block size of each task is small(like 20K), but some block size(like 2M, but small than `accurateBlockThreshold`) are much bigger than other block, in this situation `HighlyCompressedMapStatus` also get avgrage size for these more bigger blocks, but for the reduce task processing these blocks it will fetch actually 3000 * 2M = 5.8G data which pretty likely cause OOM. Of course we can tune `accurateBlockThreshold` to prevent it, but this approach is specific-application and cannot be used in common case, so I think we should solve it in `HighlyCompressedMapStatus`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
