[
https://issues.apache.org/jira/browse/SLIDER-129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14196278#comment-14196278
]
Gour Saha commented on SLIDER-129:
----------------------------------
These are the physical and virtual memory numbers we see in the 2 Slider agent
processes that run for every container (in this example deploying component
HBase RegionServer).
{noformat}
[root@c6401 container_1415001620343_0004_01_000002]# ps -aef | grep slider |
grep -v SliderAppMaster
yarn 17019 17018 0 15:32 ? 00:00:00 /bin/bash -c python
./infra/agent/slider-agent/agent/main.py --label
container_1415001620343_0004_01_000004___HBASE_REGIONSERVER --zk-quorum
c6401.ambari.apache.org:2181,c6402.ambari.apache.org:2181,c6403.ambari.apache.org:2181
--zk-reg-path /registry/users/yarn/services/org-apache-slider/cl2 >
/hadoop/yarn/log/application_1415001620343_0004/container_1415001620343_0004_01_000004/slider-agent.out
2>&1
yarn 17027 17019 0 15:32 ? 00:00:01 python
./infra/agent/slider-agent/agent/main.py --label
container_1415001620343_0004_01_000004___HBASE_REGIONSERVER --zk-quorum
c6401.ambari.apache.org:2181,c6402.ambari.apache.org:2181,c6403.ambari.apache.org:2181
--zk-reg-path /registry/users/yarn/services/org-apache-slider/cl2
{noformat}
The bash process -
{noformat}
[root@c6401 container_1415001620343_0004_01_000002]# pmap 17019 | sed 's/\]//'
| awk '{print $2 " " $NF}' | sed 's/K / /' | sort -n | awk '$1 > 1000'
1576 /lib64/libc-2.12.so
2044 /lib64/libc-2.12.so
2048 /lib64/libdl-2.12.so
2048 /lib64/libtinfo.so.5.7
96836 /usr/lib/locale/locale-archive
106064 106064K
{noformat}
{noformat}
[root@c6401 container_1415001620343_0004_01_000002]# pmap 17027 | sed 's/\]//'
| awk '{print $2 " " $NF}' | sed 's/K / /' | sort -n | awk '$1 > 1000'
1396 /usr/lib64/libpython2.6.so.1.0
1576 /lib64/libc-2.12.so
1764 /usr/lib64/libcrypto.so.1.0.1e
2044 /lib64/libc-2.12.so
2044 /lib64/libcom_err.so.2.1
2044 /lib64/libkeyutils.so.1.3
2044 /lib64/libkrb5.so.3.3
2044 /lib64/libkrb5support.so.0.1
2044 /lib64/libm-2.12.so
2044 /lib64/libselinux.so.1
2044 /lib64/libutil-2.12.so
2044 /lib64/libuuid.so.1.3.0
2044 /lib64/libz.so.1.2.3
2044 /usr/lib64/libcrypto.so.1.0.1e
2044 /usr/lib64/libpython2.6.so.1.0
2044 /usr/lib64/libssl.so.1.0.1e
2044 /usr/lib64/python2.6/lib-dynload/arraymodule.so
2044 /usr/lib64/python2.6/lib-dynload/_bisectmodule.so
2044 /usr/lib64/python2.6/lib-dynload/_collectionsmodule.so
2044 /usr/lib64/python2.6/lib-dynload/cStringIO.so
2044 /usr/lib64/python2.6/lib-dynload/_ctypes.so
2044 /usr/lib64/python2.6/lib-dynload/_hashlib.so
2044 /usr/lib64/python2.6/lib-dynload/_json.so
2044 /usr/lib64/python2.6/lib-dynload/mathmodule.so
2044 /usr/lib64/python2.6/lib-dynload/operator.so
2044 /usr/lib64/python2.6/lib-dynload/_randommodule.so
2044 /usr/lib64/python2.6/lib-dynload/_ssl.so
2044 /usr/lib64/python2.6/lib-dynload/stropmodule.so
2044 /usr/lib64/python2.6/lib-dynload/unicodedata.so
2048 /lib64/libdl-2.12.so
2048 /lib64/libgssapi_krb5.so.2.2
2048 /lib64/libk5crypto.so.3.1
2048 /lib64/libnss_files-2.12.so
2048 /lib64/libpthread-2.12.so
2048 /lib64/libresolv-2.12.so
2048 /usr/lib64/libffi.so.5.0.6
2048 /usr/lib64/python2.6/lib-dynload/binascii.so
2048 /usr/lib64/python2.6/lib-dynload/cPickle.so
2048 /usr/lib64/python2.6/lib-dynload/fcntlmodule.so
2048 /usr/lib64/python2.6/lib-dynload/_functoolsmodule.so
2048 /usr/lib64/python2.6/lib-dynload/_heapq.so
2048 /usr/lib64/python2.6/lib-dynload/itertoolsmodule.so
2048 /usr/lib64/python2.6/lib-dynload/_localemodule.so
2048 /usr/lib64/python2.6/lib-dynload/selectmodule.so
2048 /usr/lib64/python2.6/lib-dynload/_socketmodule.so
2048 /usr/lib64/python2.6/lib-dynload/_struct.so
2048 /usr/lib64/python2.6/lib-dynload/timemodule.so
5792 anon
10240 anon
10240 anon
10240 anon
64856 anon
65404 anon
65404 anon
96836 /usr/lib/locale/locale-archive
431716 431716K
{noformat}
Physical memory snapshot -
{noformat}
top - 15:51:57 up 20:06, 2 users, load average: 0.00, 0.02, 0.00
Tasks: 2 total, 0 running, 2 sleeping, 0 stopped, 0 zombie
Cpu0 : 1.0%us, 0.3%sy, 0.0%ni, 98.3%id, 0.0%wa, 0.3%hi, 0.0%si, 0.0%st
Mem: 1922680k total, 1840076k used, 82604k free, 11568k buffers
Swap: 2621432k total, 129620k used, 2491812k free, 298056k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17027 yarn 20 0 421m 13m 4332 S 0.3 0.7 0:07.63 python
17019 yarn 20 0 103m 1344 1128 S 0.0 0.1 0:00.00 bash
{noformat}
Summary:
Slider agent processes are consuming in total about 524mb virtual memory and
15mb of physical memory. Physical memory consumed is fairly on the acceptable
side. We need to target to reduce the virtual memory consumption. The locale
library is consuming ~200mb across the 2 processes and seems to be an easy
target.
> Evaluate footprint for the agent process and scale it to multiple instances
> ---------------------------------------------------------------------------
>
> Key: SLIDER-129
> URL: https://issues.apache.org/jira/browse/SLIDER-129
> Project: Slider
> Issue Type: Bug
> Components: agent
> Affects Versions: Slider 0.40
> Reporter: Sumit Mohanty
> Assignee: Gour Saha
> Priority: Critical
> Fix For: Slider 0.60
>
>
> Slider create one instance of the agent per container. The foot print of the
> agent instance need to be analyzed against the possibility of a number of
> such agents on a single host.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
