[
https://issues.apache.org/jira/browse/IMPALA-7239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16530610#comment-16530610
]
Tim Armstrong edited comment on IMPALA-7239 at 7/6/18 4:32 PM:
---------------------------------------------------------------
Here's an example on a system that's running a heavy workload of concurrent
queries, which results in 20k+ VM maps in the Impala
On a RHEL7 system with kernel 3.10.0-327.36.3.el7.x86_64
{noformat}
$ time sudo bash -c 'cat /proc/$(pgrep impalad)/smaps | grep 'Size:' | wc -l'
55752
real 0m13.216s
user 0m0.082s
sys 0m12.415s
{noformat}
It takes 12s of system time just to walk the maps, which mean with the default
memory maintenance time that thread will running most of the time iterating
over smaps, burning CPU and potentially interfering with other memory
operations.
was (Author: tarmstrong):
Here's an example on a system that's running a heavy workload of concurrent
queries, which results in 20k+ VM maps in the Impala
On a RHEL7 system with kernel 3.10.0-327.36.3.el7.x86_64
{noformat}
$ time sudo bash -c 'cat /proc/$(pgrep impalad)/smaps | grep 'Size:' | wc -l'
55752
real 0m13.216s
user 0m0.082s
sys 0m12.415s
{noformat}
It takes 12s of system time just to walk the maps, which mean with the default
memory maintenance time that thread will be running non-stop iterating over
smaps, burning CPU and potentially interfering with other memory operations.
> Mitigate ParseSmaps() overhead
> ------------------------------
>
> Key: IMPALA-7239
> URL: https://issues.apache.org/jira/browse/IMPALA-7239
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 2.10.0, Impala 2.11.0, Impala 3.0, Impala 2.12.0
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Critical
> Labels: perf, resource-management
>
> I've heard anecdotes of high system time spent in functions related this the
> smap parsing. It appears that this can be expensive on systems once the
> impalad virtual memory gets fragmented and there are 10s of thousands of maps.
> We can try to mitigate by reducing frequency of the parsing or disabling it
> entirely. I'm not sure if there are cheaper ways to get all of the same
> metrics.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
