[
https://issues.apache.org/jira/browse/PIG-4555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14547071#comment-14547071
]
Remi Catherinot commented on PIG-4555:
--------------------------------------
I've just test the use of -XX:+UseNUMA. I use yarn rather than tez, jdk 1.7 and
running NUMA aware version of linux, hadoop 2.5.2 and pig 0.14. When adding
-XX:+UseNUMA to am command line option, memory estimation for my container just
screw up, i end-up having my containers (the AM one) being killed because they
use too much virtual memory (about 17GB of virtual memory). Maybe i face a bug
in the container resource monitoring stuff when using -XX:+UseNUMA, maybe
that's also why it seems to fix OOM issues in other case. But for sure, in my
case, setting -XX:+UseNUMA do trigger an OOM.
I've re-read your description of the problem. For what i understand, the real
problem is not about -XX!:+UseNUMA. The real problem is more that some option
from the tez configuration are ignored. Can the bug description be changed in
this way if you agree ?
> Add -XX:+UseNUMA for Tez jobs
> -----------------------------
>
> Key: PIG-4555
> URL: https://issues.apache.org/jira/browse/PIG-4555
> Project: Pig
> Issue Type: Improvement
> Reporter: Rohini Palaniswamy
> Assignee: Rohini Palaniswamy
> Fix For: 0.15.0
>
>
> For very big Tez jobs (~50K tasks), AM quickly goes OOM without
> -XX:+UseNUMA. tez.am.launch.cmd-opts default setting has that, but since pig
> gives preference to yarn.app.mapreduce.am.command-opts if present (which
> usually it is), -XX:+UseNUMA is not there. Need to add -XX:+UseNUMA if we
> are picking up mapreduce setting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
