Orhan Kislal created MADLIB-1372:
------------------------------------
Summary: MADlib Keras operations create too many threads
Key: MADLIB-1372
URL: https://issues.apache.org/jira/browse/MADLIB-1372
Project: Apache MADlib
Issue Type: Bug
Components: Deep Learning
Reporter: Orhan Kislal
Fix For: v1.17
We noticed that madlib keras operations that call keras functions create a
number of threads every single time they are called. However, some of these
threads are not cleaned up at the end of the function. If the number of
iterations is very high, the thread count keeps increasing and eventually hits
the system default limit (1024).
Here is the error message from the log
{code}
what(): Resource temporarily
unavailable",,,,,,,,"SysLoggerMain","syslogger.c",618,
{code}
We tried keeping the session at the end of the function (saved the session info
in SD) and reuse it for the next iteration but that didn't help with this issue.
It is possible to increase this limit by editing /etc/security/limits.conf and
the files in /etc/security/limits.d/. This requires a restart of the system and
the database to take effect.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
