[
https://issues.apache.org/jira/browse/LIVY-763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Larry McCay reassigned LIVY-763:
--------------------------------
Assignee: Larry McCay (was: Asif Khatri)
> FD leak in InteractiveSession and PythonInterpreter
> ---------------------------------------------------
>
> Key: LIVY-763
> URL: https://issues.apache.org/jira/browse/LIVY-763
> Project: Livy
> Issue Type: Bug
> Components: Interpreter, Server
> Affects Versions: 0.6.0, 0.7.0
> Reporter: Dustin Koupal
> Assignee: Larry McCay
> Priority: Minor
> Fix For: 0.8.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> We recently upgraded from Livy 0.4 to Livy 0.6 and started to run out of FDs
> (we fire up thousands of InteractiveSessions a day). Looking at the open FDs,
> they were all DIR handles for `/usr/lib/spark/python/lib`.
> I believe this change was introduced in
> https://github.com/apache/incubator-livy/commit/1bbefe601641aa4db74dbed1f7faf458b9a70a63#diff-7649a51ad4bddc91b6f1038e06479d41R259-R264
> The behavior without this change and the default config is Livy will open 2
> FDs for each InteractiveSession, and they will never be closed (I think
> because `DirectoryStream` is
> `AutoCloseable` and try-with-resource doesn't work with Scala's `Try`.
> As a workaround, in our production clusters, we have set
> `livy.rsc.pyspark.archives` to an empty string in `livy.conf`
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
