gaogaotiantian commented on PR #53359: URL: https://github.com/apache/spark/pull/53359#issuecomment-3620916287
Hi @dongjoon-hyun , `runner_conf` is hand-picked by daemon to pass to worker in a need-to-know basis. For now, the maximum number is less than 10. 10000 is a very safe upper limit. Also, we have about a few thousand spark configs in total so even if we pass everything, it's still going to be less than 10000. I don't think this should be controllable by an env var because: 1. It's a sanity check, not a run-time validation. We just want to make sure that number at least makes some sense. 2. One of the reasons to introduce `runner_conf` is to avoid passing too many arguments through env var. Hopefully we can put more environment setups in `runner_conf`. Having another env var to control it is kind of against the purpose. 3. We will have more sanity checks in the protocol - which will be super safe but helpful to rule out crazy situations. Having an env var for each of them will explode our env var namespace. If you really hate the magic number, we can only check the positivity of the number. However, that leaves plenty of unreasonable space for sanity check. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
