I have a requirement where for every Spark executor threadpool thread, I need to launch an associated external process.
My job will consist of some processing in the Spark executor thread and some processing by its associated external process with the 2 communicating via some IPC mechanism. Is there a hook in Spark where I can put in my code to create / destroy these external processes corresponding to the creation / destruction of executor thread pool threads. Thanks Anand -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-hook-to-create-external-process-tp6526.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
