[
https://issues.apache.org/jira/browse/IMPALA-14499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yida Wu updated IMPALA-14499:
-----------------------------
Description:
The process memory tracker in the admission service doesn't seem to accurately
account for the process's true memory usage. During active workloads, its
reported consumption remains near zero, while the underlying tcmalloc
statistics show significant memory allocation, creating a large discrepancy.
This makes the memory tracker an unreliable for memory management and OOM
prevention within admissiond.
I added a logging to compare TcmallocMetric::BYTES_IN_USE against
AdmissiondEnv::GetInstance()->process_mem_tracker()>consumption() in
admissiond. During a workload, the logs show tcmalloc reporting over 100MB in
use while the process tracker's current values close to zero:
{code:java}
// Log comparison
I20251014 09:25:45.045312 10639 admission-controller.cc:1655] Here tcmalloc
100965952 process mem tracker 0
I20251014 09:25:46.813424 10638 admission-controller.cc:1655] Here tcmalloc
101201000 process mem tracker 0
I20251014 09:25:47.481946 10642 admission-controller.cc:1655] Here tcmalloc
101604760 process mem tracker 9424
I20251014 09:25:48.281759 10641 admission-controller.cc:1655] Here tcmalloc
101604792 process mem tracker 0
I20251014 09:25:49.306492 10641 admission-controller.cc:1655] Here tcmalloc
101503064 process mem tracker 0 {code}
The same log comparison in {{impalad}} shows that the two values are similar.
It needs further investigation.
was:
The process memory tracker in the admission service doesn't seem to accurately
account for the process's true memory usage. During active workloads, its
reported consumption remains near zero, while the underlying tcmalloc
statistics show significant memory allocation, creating a large discrepancy.
This makes the memory tracker an unreliable for memory management and OOM
prevention within admissiond.
I added a logging to compare TcmallocMetric::BYTES_IN_USE against
AdmissiondEnv::GetInstance()->process_mem_tracker()>consumption() in
admissiond. During a workload, the logs show tcmalloc reporting over 100MB in
use while the process tracker's current values close to zero:
{code:java}
// Log comparison
I20251014 09:25:45.045312 10639 admission-controller.cc:1655] Here tcmalloc
100965952 process mem tracker 0
I20251014 09:25:46.813424 10638 admission-controller.cc:1655] Here tcmalloc
101201000 process mem tracker 0
I20251014 09:25:47.481946 10642 admission-controller.cc:1655] Here tcmalloc
101604760 process mem tracker 9424
I20251014 09:25:48.281759 10641 admission-controller.cc:1655] Here tcmalloc
101604792 process mem tracker 0
I20251014 09:25:49.306492 10641 admission-controller.cc:1655] Here tcmalloc
101503064 process mem tracker 0 {code}
{*}{{*}}The same log comparison in {{impalad}} shows that the two values are
similar. It needs further investigation.
> Admissiond process memory tracker is inaccurate and underreports memory usage
> -----------------------------------------------------------------------------
>
> Key: IMPALA-14499
> URL: https://issues.apache.org/jira/browse/IMPALA-14499
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Reporter: Yida Wu
> Priority: Major
>
> The process memory tracker in the admission service doesn't seem to
> accurately account for the process's true memory usage. During active
> workloads, its reported consumption remains near zero, while the underlying
> tcmalloc statistics show significant memory allocation, creating a large
> discrepancy. This makes the memory tracker an unreliable for memory
> management and OOM prevention within admissiond.
> I added a logging to compare TcmallocMetric::BYTES_IN_USE against
> AdmissiondEnv::GetInstance()->process_mem_tracker()>consumption() in
> admissiond. During a workload, the logs show tcmalloc reporting over 100MB in
> use while the process tracker's current values close to zero:
> {code:java}
> // Log comparison
> I20251014 09:25:45.045312 10639 admission-controller.cc:1655] Here tcmalloc
> 100965952 process mem tracker 0
> I20251014 09:25:46.813424 10638 admission-controller.cc:1655] Here tcmalloc
> 101201000 process mem tracker 0
> I20251014 09:25:47.481946 10642 admission-controller.cc:1655] Here tcmalloc
> 101604760 process mem tracker 9424
> I20251014 09:25:48.281759 10641 admission-controller.cc:1655] Here tcmalloc
> 101604792 process mem tracker 0
> I20251014 09:25:49.306492 10641 admission-controller.cc:1655] Here tcmalloc
> 101503064 process mem tracker 0 {code}
>
> The same log comparison in {{impalad}} shows that the two values are similar.
> It needs further investigation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
