Tolstopyatov Vsevolod created SPARK-22625:
---------------------------------------------
Summary: Properly cleanup inheritable thread-locals
Key: SPARK-22625
URL: https://issues.apache.org/jira/browse/SPARK-22625
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.2.0
Reporter: Tolstopyatov Vsevolod
Memory leak is present due to inherited thread locals, SPARK-20558 didn't fixed
it properly.
Our production application has the following logic: one thread is reading from
HDFS and another one creates spark context, process HDFS files and then closes
it on regular schedule.
Depending on what thread started first, SparkContext thread local may or may
not be inherited by HDFS-daemon (DataStreamer), causing memory leak when
streamer was created after spark context.
Memory consumption increases every time new spark context is created, related
yourkit paths: https://screencast.com/t/tgFBYMEpW
Proper fix: register all cloned properties (in `localProperties#childValue`) in
ConcurrentHashMap and forcefully clear all of them in `SparkContext#close`
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
