zsxwing commented on issue #25577: [WIP][CORE][SPARK-28867] InMemoryStore checkpoint to speed up replay log file in HistoryServer URL: https://github.com/apache/spark/pull/25577#issuecomment-532869010 > additional (considerable amount of) CPU/memory consumption. Additional memory consumption should be small since the part to copy is not a lot. And the additional CPU consumption is acceptable since I think most of heavy work should be done in executors. I don't expect that the driver needs to heavy CPU work. Otherwise, the cluster resource will be just wasted because the cluster is just idle to wait for driver to do something.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
