squito commented on issue #25229: [SPARK-27900][K8s] Add jvm oom flag URL: https://github.com/apache/spark/pull/25229#issuecomment-519179466 Sorry I am not sure I entirely follow the full discussion here. I think that I tend to agree with dongjoon, though -- can't users just set this themselves? I feel a bit torn here, as I do think we should try to have sensible defaults, and this does seem better. The driver jvm is always tricky, because spark is not entirely responsible for it -- eg. the user might be running a server which is doing all sorts of things, and it happens to use a spark context for servicing some requests. Not an architecture I'd recommend, and I dunno what else it could with an OOM anyway -- but still I hesitate to have spark be changing that. Then again, maybe spark 3.0 is a good chance for us to change this to a more sane default? anyway, I guess I'm not saying anything definitive yet, just thinking "out loud" ... another thing which may be relevant here is https://github.com/apache/spark/pull/24804 / SPARK-23472 -- it at least makes it easy for an admin to set this option across the board.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
