Li Jin created SPARK-24334:
------------------------------
Summary: Race condition in ArrowPythonRunner causes unclean
shutdown of Arrow memory allocator
Key: SPARK-24334
URL: https://issues.apache.org/jira/browse/SPARK-24334
Project: Spark
Issue Type: Sub-task
Components: PySpark
Affects Versions: 2.3.0
Reporter: Li Jin
Currently, ArrowPythonRunner has two thread that frees the Arrow vector schema
root and allocator - The main writer thread and task completion listener
thread.
Having both thread doing the clean up leads to weird case (e.g., negative ref
cnt, NPE, and memory leak exception) when an exceptions are thrown from the
user function.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
