ifilonenko commented on issue #26093: [SPARK-27812][K8s] Bump client version URL: https://github.com/apache/spark/pull/26093#issuecomment-541161051 @igorcalabria thank you for writing this patch I have ran this internally on my cluster and can confirm that non-daemon threads no longer block the JVM from exiting. Against branch 2.4 and master: ======================= I see the following non-daemon threads exist when running spark-submit against the following pyspark job: ``` from pyspark.sql import SparkSession if __name__ == '__main__': spark = SparkSession.builder.getOrCreate() ``` After the python process exited, the JVM process remained and I see the following using `jstack`: ``` "OkHttp WebSocket https://10.96.0.1/..."; #121 prio=5 os_prio=0 tid=0x00007fb27c005800 nid=0x24b waiting on condition [0x00007fb300847000] "OkHttp WebSocket https://10.96.0.1/..."; #117 prio=5 os_prio=0 tid=0x00007fb28c004000 nid=0x247 waiting on condition [0x00007fb300e4b000] ``` Against this patch: ============== I no longer see the following non-daemon threads and the JVM is able to exit after the completion of the python process. @mccheah @erikerlandson @felixcheung @holdenk we should backport this fix to branch 2.4 as well and cut a new release with this patch.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
