uros-b commented on PR #57528: URL: https://github.com/apache/spark/pull/57528#issuecomment-5387897942
One more minor note. Migration guide + config .doc understate worst-case driver memory. For this now-default-ON change, the docs say it retains "up to 100 block sizes per map task" but omit the new cutoff-tie RoaringBitmap, whose scattered-ties worst case is partition-proportional (~62 MiB for a 50000-partition × 10000-task stage). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
