Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/20669
That's one scenario where an init-container /might/ help. But be aware that
your YARN example only works in a single one scenario - YARN cluster mode, and
maybe even then may need some config customization. And it may not even work in
some cases (e.g. YARN data directory mounted with `noexec`). YARN client, nor
any other supported cluster manager, is able to do what you're saying.
Personally I feel it's perfectly ok to require a custom docker image in
these cases, since they're so uncommon (I've never seen one of our users use
the yarn-cluster feature for this purpose). People can have a "main" Spark
image and a "debug" one that can be easily chosen from when submitting the app.
During this discussion I think someone mentioned that it might be possible
to side-load init containers into Spark without this. I'm not that familiar
with kubernetes, but if that's possible, it's another way you could achieve
this without Spark having its own init container.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
