I'm observing this as well on 0.9.0, with several 10s of GB accumulating in that directory but never being cleaned up. I think this has gotten more pronounced in 0.9.0 as well with large reducers spilling to disk.
On Wed, Feb 5, 2014 at 3:46 PM, Mingyu Kim <[email protected]> wrote: > After creating a lot of Spark connections, work/app-* folders in Worker > nodes keep getting created without any clean-up being done. This > particularly becomes a problem when the Spark driver programs ship jars or > files. Is there any way to garbage collect these without manually deleting > them? Thanks! > > Mingyu >
