[ https://issues.apache.org/jira/browse/SPARK-15619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15305117#comment-15305117 ]
Sean Owen commented on SPARK-15619: ----------------------------------- Interesting, looks like it's related to the lz4 library, and I see a similar issue reported for Cassandra: https://issues.apache.org/jira/browse/CASSANDRA-7712 It does create this temp library: https://github.com/jpountz/lz4-java/blob/b69d5676f74344bf04068594644fa5ecc2bb6a67/src/java/net/jpountz/util/Native.java#L81 but seems to do a pretty comprehensive job of trying to clean it up at shutdown. It might be left around after hard JVM failures / exits, in which case it may unfortunately be a side effect of testing failure conditions. I don't see anything in Spark that tries to manage it, and not sure it could. > spark builds filling up /tmp > ---------------------------- > > Key: SPARK-15619 > URL: https://issues.apache.org/jira/browse/SPARK-15619 > Project: Spark > Issue Type: Bug > Components: Build > Reporter: shane knapp > Priority: Minor > > spark builds aren't cleaning up /tmp after they run... it's hard to pinpoint > EXACTLY what is left there by the spark builds (as other builds are also > guilty of doing this), but a quick perusal of the /tmp directory during some > spark builds show that there are myriad empty directories being created and a > massive pile of shared object libraries being dumped there. > $ for x in $(cat jenkins_workers.txt ); do echo $x; ssh $x "ls -l /tmp/*.so | > wc -l"; done > amp-jenkins-worker-01 > 0 > ls: cannot access /tmp/*.so: No such file or directory > amp-jenkins-worker-02 > 22312 > amp-jenkins-worker-03 > 39673 > amp-jenkins-worker-04 > 39548 > amp-jenkins-worker-05 > 39577 > amp-jenkins-worker-06 > 39299 > amp-jenkins-worker-07 > 39315 > amp-jenkins-worker-08 > 38529 > to help combat this, i set up a cron job on each worker that runs tmpwatch > during system downtime on sundays to clean up files older than a week. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org