Hello, I'm having trouble with the "inprogress" files generated per application in /tmp with Spark Streaming. They seem to grow continually and never shrink, and end up filling the /tmp partition.
I haven't found too much litterature on those files. I've tried to set spark.cleaner.ttl to a limited value, but it was a wild guess and it doesn't seem to impact those files. What is the use of these files ? How can I limit their size ? Regards, Mathieu Garstecki
