baohe-zhang commented on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-654455672
@tgravescs I measured the memory usage of an app that contains 4 jobs, and each job contains 1 stage, each stage contains 100000 tasks (the max retained tasks per stage by default). The number of entities stored in kvstore are 4 jobs, 4 stages, and 400000 tasks. The log size of this app is 1.1 GB, and the memory usage is 212 MB. I measured another app that contains 1000 jobs (the max retained jobs by default), each job contains 1 stage and each stage contains 400 tasks. The total number of entities for this app are 1000 jobs, 1000 stages, and 400000 tasks, which led to a 1.29 GB log file and 271 MB memory usage. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
