Github user vanzin commented on the issue:
    That's one scenario where an init-container /might/ help. But be aware that 
your YARN example only works in a single one scenario - YARN cluster mode, and 
maybe even then may need some config customization. And it may not even work in 
some cases (e.g. YARN data directory mounted with `noexec`). YARN client, nor 
any other supported cluster manager, is able to do what you're saying.
    Personally I feel it's perfectly ok to require a custom docker image in 
these cases, since they're so uncommon (I've never seen one of our users use 
the yarn-cluster feature for this purpose). People can have a "main" Spark 
image and a "debug" one that can be easily chosen from when submitting the app.
    During this discussion I think someone mentioned that it might be possible 
to side-load init containers into Spark without this. I'm not that familiar 
with kubernetes, but if that's possible, it's another way you could achieve 
this without Spark having its own init container.


To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to