Hi Matei, Thanks for the reply.
I would like to avoid having to spawn these external processes every time during the processing of the task to reduce task latency. I'd like these to be pre-spawned as much as possible - tying them to lifecycle of corresponding threadpool thread would simplify management for me. Also, during processing some back and forth communication is required between the Spark executer thread and its associated external process. For these 2 reasons, pipe() wouldnt meet my requirement. Is there any hook in the ThreadPoolExecutor created by the Spark Executor to plug in my own ThreadFactory ? Thanks Anand -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-hook-to-create-external-process-tp6526p6552.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
