tgravescs commented on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-654838940
thanks, for testing those combinations, that shows the number of jobs doesn't affect the memory usage much, but if I'm not mistaken in all the cases you have 400,000 tasks. My main concern was the memory usage with small number of tasks total - say 400 or 1000? Does it still stay to the same ratio or do we way over estimate. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
