exmy commented on pull request #32733:
URL: https://github.com/apache/spark/pull/32733#issuecomment-852849714


   > I am missing something here, if a block is < 
`spark.shuffle.accurateBlockThreshold`, it is recorded as a small block - else 
it is marked as a huge block. The point of 
`spark.shuffle.accurateBlockThreshold` is that it is not very high, so should 
not cause OOM - are you configuring it to a very high value ?
   > Note that this gets triggered when (typically) you have > 2k partitions - 
so the benefits of using `HighlyCompressedMapStatus` is to prevent other issues 
which more accurate tracking ends up with for very large number of partitions.
   
   Thanks for review. In our production, `spark.shuffle.accurateBlockThreshold` 
is set to 10M. We encounted a OOM case, for example, assume that map stage has 
3k tasks and shuffle partitions are also 3k, most of output block size of each 
task is small(like 20K), but some block size(like 2M, but small than 
`accurateBlockThreshold`) are much bigger than other block, in this situation 
`HighlyCompressedMapStatus` also get avgrage size for these more bigger blocks, 
but for the reduce task processing these blocks it will fetch actually 3000 * 
2M = 5.8G data which pretty likely cause OOM. Of course we can tune 
`accurateBlockThreshold` to prevent it, but this approach is 
specific-application and cannot be used in common case,  so I think we should 
solve it in `HighlyCompressedMapStatus`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

Reply via email to