[jira] [Comment Edited] (OOZIE-3409) Oozie Server : Possible memory leak in EL evaluation
[
https://issues.apache.org/jira/browse/OOZIE-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732827#comment-16732827
]
Andras Salamon edited comment on OOZIE-3409 at 1/3/19 9:55 AM:
---
The commons.el bug reported in 2005: EL-1
was (Author: asalamon74):
The commons.el bug reported in 2005.
> Oozie Server : Possible memory leak in EL evaluation
>
>
> Key: OOZIE-3409
> URL: https://issues.apache.org/jira/browse/OOZIE-3409
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Praneeth
>Assignee: Andras Salamon
>Priority: Major
>
> I have noticed that there is a potential memory leak in the Oozie server due
> to one of its dependencies - {{commons-el:commons-el:1.0}}. Our oozie servers
> have been running out of memory occasionally. It happens more often when we
> have more activity. Upon analysis of the heap dump, it seems that the culprit
> here is the {{org.apache.commons.el.ExpressionEvaluatorImpl}} from
> {{commons-el}} which is used in the Oozie server through
> {{org.apache.oozie.util.ELEvaluator}} [1] . {{ExpressionEvaluatorImpl}} uses
> a static {{Map}} [2] to cache the EL evaluations. This map never gets
> cleared, eventually taking up all the memory. So, in an environment with
> heavy usage and long running oozie servers, this would lead to an OOM. The
> configurations for workflow actions we have are possibly larger than normal
> which is worsening the situation as the Map has to maintain larger strings.
> #
> [https://github.com/apache/oozie/blob/release-5.1.0/core/src/main/java/org/apache/oozie/util/ELEvaluator.java#L143]
> #
> [http://svn.apache.org/viewvc/commons/dormant/el/trunk/src/java/org/apache/commons/el/ExpressionEvaluatorImpl.java?view=markup#l94]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (OOZIE-3409) Oozie Server : Possible memory leak in EL evaluation
[
https://issues.apache.org/jira/browse/OOZIE-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16726019#comment-16726019
]
Praneeth edited comment on OOZIE-3409 at 12/20/18 5:06 PM:
---
I have just noticed that {{org.apache.commons.el.ExpressionEvaluatorImpl}} is
used only in one place in {{oozie-core}} . As a quick dirty fix, we could
extend the class and override the evaluation method to skip writing to the map
when the flag is disabled.
EDIT: Or just not use the cache. Correct me if I am wrong but I think lack of
that cache won't make much difference in performance considering the way oozie
uses it.
EDIT 2 : Looks like I am not thinking straight this morning. We won't be using
the cache. Ignore my EDIT above and the part about flag in the original comment.
was (Author: praneeth.varma):
I have just noticed that {{org.apache.commons.el.ExpressionEvaluatorImpl}} is
used only in one place in {{oozie-core}} . As a quick dirty fix, we could
extend the class and override the evaluation method to skip writing to the map
when the flag is disabled.
EDIT: Or just not use the cache. Correct me if I am wrong but I think lack of
that cache won't make much difference in performance considering the way oozie
uses it.
> Oozie Server : Possible memory leak in EL evaluation
>
>
> Key: OOZIE-3409
> URL: https://issues.apache.org/jira/browse/OOZIE-3409
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Praneeth
>Priority: Major
>
> I have noticed that there is a potential memory leak in the Oozie server due
> to one of its dependencies - {{commons-el:commons-el:1.0}}. Our oozie servers
> have been running out of memory occasionally. It happens more often when we
> have more activity. Upon analysis of the heap dump, it seems that the culprit
> here is the {{org.apache.commons.el.ExpressionEvaluatorImpl}} from
> {{commons-el}} which is used in the Oozie server through
> {{org.apache.oozie.util.ELEvaluator}} [1] . {{ExpressionEvaluatorImpl}} uses
> a static {{Map}} [2] to cache the EL evaluations. This map never gets
> cleared, eventually taking up all the memory. So, in an environment with
> heavy usage and long running oozie servers, this would lead to an OOM. The
> configurations for workflow actions we have are possibly larger than normal
> which is worsening the situation as the Map has to maintain larger strings.
> #
> [https://github.com/apache/oozie/blob/release-5.1.0/core/src/main/java/org/apache/oozie/util/ELEvaluator.java#L143]
> #
> [http://svn.apache.org/viewvc/commons/dormant/el/trunk/src/java/org/apache/commons/el/ExpressionEvaluatorImpl.java?view=markup#l94]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (OOZIE-3409) Oozie Server : Possible memory leak in EL evaluation
[
https://issues.apache.org/jira/browse/OOZIE-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16726008#comment-16726008
]
Praneeth edited comment on OOZIE-3409 at 12/20/18 4:46 PM:
---
Actually, I don't think it is just the actions that have EL expressions that
sit in the cache, I think the parsing is taking place for every action (or some
part of workflow config), whether or not it has an EL (call to parsing takes
place irrespectively) [1]. If it doesn't have an EL, the cache would store the
same content as both key and value. I was able to verify this by looking at the
contents of the cache.
This is only to point out that it is not just for EL based content. I do not
have a good proposal for a solution given the dependency is dormant. What is
the general practice for fixes to dormant projects ? Is it to try to find an
alternative and if none is found, try to revive the project ?
I agree that it is probably a problem to few deployments like ours which are
long running and relatively large workflows, but this crashes our instances
once in 2-3 months.
# I was checking the usage of
[https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/util/ELEvaluator.java#L201.]
was (Author: praneeth.varma):
Actually, I don't think it is just the actions that have EL expressions that
sit in the cache, I think the parsing is taking place for every action (or some
part of workflow config), whether or not it has an EL (call to parsing takes
place irrespectively) [1]. If it doesn't have an EL, the cache would store the
same content as both key and value. I was able to verify this by looking at the
contents of the cache.
This is only to point out that it is not just for EL based content. I do not
have a good proposal for a solution given the dependency is dormant. What is
the general practice for fixes to dormant projects ? Is it to try to find an
alternative and if none is found, try to revive the project ?
I agree that it is probably a problem to few deployments like ours which are
long running and relatively workflows, but this crashes our instances once in
2-3 months.
# I was checking the usage of
[https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/util/ELEvaluator.java#L201.]
> Oozie Server : Possible memory leak in EL evaluation
>
>
> Key: OOZIE-3409
> URL: https://issues.apache.org/jira/browse/OOZIE-3409
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Praneeth
>Priority: Major
>
> I have noticed that there is a potential memory leak in the Oozie server due
> to one of its dependencies - {{commons-el:commons-el:1.0}}. Our oozie servers
> have been running out of memory occasionally. It happens more often when we
> have more activity. Upon analysis of the heap dump, it seems that the culprit
> here is the {{org.apache.commons.el.ExpressionEvaluatorImpl}} from
> {{commons-el}} which is used in the Oozie server through
> {{org.apache.oozie.util.ELEvaluator}} [1] . {{ExpressionEvaluatorImpl}} uses
> a static {{Map}} [2] to cache the EL evaluations. This map never gets
> cleared, eventually taking up all the memory. So, in an environment with
> heavy usage and long running oozie servers, this would lead to an OOM. The
> configurations for workflow actions we have are possibly larger than normal
> which is worsening the situation as the Map has to maintain larger strings.
> #
> [https://github.com/apache/oozie/blob/release-5.1.0/core/src/main/java/org/apache/oozie/util/ELEvaluator.java#L143]
> #
> [http://svn.apache.org/viewvc/commons/dormant/el/trunk/src/java/org/apache/commons/el/ExpressionEvaluatorImpl.java?view=markup#l94]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (OOZIE-3409) Oozie Server : Possible memory leak in EL evaluation
[
https://issues.apache.org/jira/browse/OOZIE-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16726019#comment-16726019
]
Praneeth edited comment on OOZIE-3409 at 12/20/18 4:51 PM:
---
I have just noticed that {{org.apache.commons.el.ExpressionEvaluatorImpl}} is
used only in one place in {{oozie-core}} . As a quick dirty fix, we could
extend the class and override the evaluation method to skip writing to the map
when the flag is disabled.
EDIT: Or just not use the cache. Correct me if I am wrong but I think lack of
that cache won't make much difference in performance considering the way oozie
uses it.
was (Author: praneeth.varma):
I have just noticed that {{org.apache.commons.el.ExpressionEvaluatorImpl}} is
used only in one place in {{oozie-core}} . As a quick dirty fix, we could
extend the class and override the evaluation method to skip writing to the map
when the flag is disabled.
> Oozie Server : Possible memory leak in EL evaluation
>
>
> Key: OOZIE-3409
> URL: https://issues.apache.org/jira/browse/OOZIE-3409
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Praneeth
>Priority: Major
>
> I have noticed that there is a potential memory leak in the Oozie server due
> to one of its dependencies - {{commons-el:commons-el:1.0}}. Our oozie servers
> have been running out of memory occasionally. It happens more often when we
> have more activity. Upon analysis of the heap dump, it seems that the culprit
> here is the {{org.apache.commons.el.ExpressionEvaluatorImpl}} from
> {{commons-el}} which is used in the Oozie server through
> {{org.apache.oozie.util.ELEvaluator}} [1] . {{ExpressionEvaluatorImpl}} uses
> a static {{Map}} [2] to cache the EL evaluations. This map never gets
> cleared, eventually taking up all the memory. So, in an environment with
> heavy usage and long running oozie servers, this would lead to an OOM. The
> configurations for workflow actions we have are possibly larger than normal
> which is worsening the situation as the Map has to maintain larger strings.
> #
> [https://github.com/apache/oozie/blob/release-5.1.0/core/src/main/java/org/apache/oozie/util/ELEvaluator.java#L143]
> #
> [http://svn.apache.org/viewvc/commons/dormant/el/trunk/src/java/org/apache/commons/el/ExpressionEvaluatorImpl.java?view=markup#l94]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
