[ https://issues.apache.org/jira/browse/LIVY-763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17140681#comment-17140681 ]
Dustin Koupal commented on LIVY-763: ------------------------------------ Yeah, I ended up going through the same thing. Fwiw, this is the setting that ended up doing the trick for the Livy that is shipped with EMR 5.29 (0.6.0) > FD leak in InteractiveSession and PythonInterpreter > --------------------------------------------------- > > Key: LIVY-763 > URL: https://issues.apache.org/jira/browse/LIVY-763 > Project: Livy > Issue Type: Bug > Components: Interpreter, Server > Affects Versions: 0.6.0, 0.7.0 > Reporter: Dustin Koupal > Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > We recently upgraded from Livy 0.4 to Livy 0.6 and started to run out of FDs > (we fire up thousands of InteractiveSessions a day). Looking at the open FDs, > they were all DIR handles for `/usr/lib/spark/python/lib`. > I believe this change was introduced in > https://github.com/apache/incubator-livy/commit/1bbefe601641aa4db74dbed1f7faf458b9a70a63#diff-7649a51ad4bddc91b6f1038e06479d41R259-R264 > The behavior without this change and the default config is Livy will open 2 > FDs for each InteractiveSession, and they will never be closed (I think > because `DirectoryStream` is > `AutoCloseable` and try-with-resource doesn't work with Scala's `Try`. > As a workaround, in our production clusters, we have set > `livy.rsc.pyspark.archives` to an empty string in `livy.conf` -- This message was sent by Atlassian Jira (v8.3.4#803005)