Hi, I have recently experimented with a few ways to run OS commands from the executors (in a YARN deployment) for a specific use case where we want to interact with an external system of interest for our environment. From that experience I thought that having the possibility to spawn a script at the start of each executors can be quite handy in a few cases and maybe more people are interested. For example I am think of case when interacting with external systems/APIs, or for injecting custom configurations via scripts distributed to the executors and/or for spawning custom monitoring tasks, etc. They are probable all niche cases but the feature seems quite easy to implement. I just wanted to check with the list if something like this has already come up in the past and/or there are thoughts about it or details that I have overlooked. My simple proof of concept for implementing a "startup command" on the executors can be found at: https://github.com/LucaCanali/spark/commit/e294a1f0d55af115f45fa6d2d7dcf81f751955fa I can put all this in a Jira in case people here think it makes sense.
Thanks, Luca --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
