[
https://issues.apache.org/jira/browse/SPARK-22575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348603#comment-16348603
]
Marco Gaido commented on SPARK-22575:
-------------------------------------
Then the problem is likely that the executors are killed in dynamicAllocation,
instead of being stopped. This prevents the BlockManager to clear the local
directories and YARN doesn't either since the application is still running. I
am not sure about the reason of killing instead of stopping and I am not expert
enough on this part to say whether we should change it or not.
> Making Spark Thrift Server clean up its cache
> ---------------------------------------------
>
> Key: SPARK-22575
> URL: https://issues.apache.org/jira/browse/SPARK-22575
> Project: Spark
> Issue Type: Improvement
> Components: Block Manager, SQL
> Affects Versions: 2.2.0
> Reporter: Oz Ben-Ami
> Priority: Minor
> Labels: cache, dataproc, thrift, yarn
>
> Currently, Spark Thrift Server accumulates data in its appcache, even for old
> queries. This fills up the disk (using over 100GB per worker node) within
> days, and the only way to clear it is to restart the Thrift Server
> application. Even deleting the files directly isn't a solution, as Spark then
> complains about FileNotFound.
> I asked about this on [Stack
> Overflow|https://stackoverflow.com/questions/46893123/how-can-i-make-spark-thrift-server-clean-up-its-cache]
> a few weeks ago, but it does not seem to be currently doable by
> configuration.
> Am I missing some configuration option, or some other factor here?
> Otherwise, can anyone point me to the code that handles this, so maybe I can
> try my hand at a fix?
> Thanks!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
