[jira] [Created] (FLINK-29134) fetch metrics may cause oom(ThreadPool task pile up)
Sitan Pang created FLINK-29134:
--
Summary: fetch metrics may cause oom(ThreadPool task pile up)
Key: FLINK-29134
URL: https://issues.apache.org/jira/browse/FLINK-29134
Project: Flink
Issue Type: Improvement
Components: Runtime / Metrics
Affects Versions: 1.11.0
Reporter: Sitan Pang
Attachments: dump-queueTask.png, dump-threadPool.png
When we queryMetrics we use thread pool to process the data which are returned
by TMs.
{code:java}
private void queryMetrics(final MetricQueryServiceGateway queryServiceGateway) {
LOG.debug("Query metrics for {}.", queryServiceGateway.getAddress());
queryServiceGateway
.queryMetrics(timeout)
.whenCompleteAsync(
(MetricDumpSerialization.MetricSerializationResult result,
Throwable t) -> {
if (t != null) {
LOG.debug("Fetching metrics failed.", t);
} else {
metrics.addAll(deserializer.deserialize(result));
}
},
executor);
} {code}
The only condition we will fetch metrics is update time is larger than
updateInterval
{code:java}
public void update() {
synchronized (this) {
long currentTime = System.currentTimeMillis();
if (currentTime - lastUpdateTime > updateInterval) {
lastUpdateTime = currentTime;
fetchMetrics();
}
}
} {code}
Therefore, if we could not process the data in update-interval-time, metrics
data will accumulate.
Besides, Rest handler and metrics share thread pool. When we open ui, it maybe
even worse.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: Fetch Metrics
Hi Suvimal, You may use REST API connecting to Job Manager to retrieve the stored metrics. Have a look at this link https://ci.apache.org/ projects/flink/flink-docs-release-1.4/monitoring/metrics.html#rest-api- integration -- Thanks, Amit On Mon, Feb 5, 2018 at 2:47 PM, Suvimal Yashraj < suvimal.yash...@ericsson.com> wrote: > Hello, > I'm new to apache flink and I want to extract the already present metrics > in flink which is stored on the server of the currently running jobs like > bytes transferred, cpu usage, duration, etc. > I'm finding it difficult so as to how to connect my java program to the > server and fetch these details. > If there's any solution to this problem please provide. > > Thank You. > > Regards, > Suvimal > >
Fetch Metrics
Hello, I'm new to apache flink and I want to extract the already present metrics in flink which is stored on the server of the currently running jobs like bytes transferred, cpu usage, duration, etc. I'm finding it difficult so as to how to connect my java program to the server and fetch these details. If there's any solution to this problem please provide. Thank You. Regards, Suvimal
