Hi,
I have a few Flink jobs running on Kubernetes using the Flink Kubernetes
Operator. By following the documentation [1] I was able to set up
monitoring for the Operator itself. As for the jobs themselves, I'm a bit
confused about how to properly set it up. Here's my FlinkDeployment
Hi,
请发送任意内容的邮件到 user-zh-unsubscr...@flink.apache.org 地址来取消订阅来自
user-zh@flink.apache.org 邮件组的邮件
Best,
Zhanghao Chen
发件人: im huzi
发送时间: 2023年9月15日 18:14
收件人: user-zh@flink.apache.org
主题: Re: flink-metrics如何获取applicationid
退订
On Wed, Aug 30, 2023 at 19:14
退订
On Wed, Aug 30, 2023 at 19:14 allanqinjy wrote:
> hi，
>请教大家一个问题，就是在上报指标到prometheus时候，jobname会随机生成一个后缀，看源码也是new Abstract
> ID（），有方法在这里获取本次上报的作业applicationid吗？
请问好使吗，怎么使用的
| |
吴先生
|
|
15951914...@163.com
|
回复的原邮件
| 发件人 | allanqinjy |
| 发送日期 | 2023年8月30日 20:02 |
| 收件人 | user-zh@flink.apache.org |
| 主题 | 回复：flink-metrics如何获取applicationid |
多谢了，明天改一下代码试试
回复的原邮件
| 发件人 | Feng Jin |
| 发送日期 | 2023年08月30日 19:42 |
| 收件人 | user-zh |
| 主题 | Re
多谢了，明天改一下代码试试
回复的原邮件
| 发件人 | Feng Jin |
| 发送日期 | 2023年08月30日 19:42 |
| 收件人 | user-zh |
| 主题 | Re: flink-metrics如何获取applicationid |
hi,
可以尝试获取下 _APP_ID 这个 JVM 环境变量.
System.getenv(YarnConfigKeys.ENV_APP_ID);
https://github.com/apache/flink/blob/6c9bb3716a3a92f3b5326558c6238432c669556d
多谢，明天修改一下代码试试
回复的原邮件
| 发件人 | Feng Jin |
| 发送日期 | 2023年08月30日 19:42 |
| 收件人 | user-zh |
| 主题 | Re: flink-metrics如何获取applicationid |
hi,
可以尝试获取下 _APP_ID 这个 JVM 环境变量.
System.getenv(YarnConfigKeys.ENV_APP_ID);
https://github.com/apache/flink/blob/6c9bb3716a3a92f3b5326558c6238432c669556d
hi,
可以尝试获取下 _APP_ID 这个 JVM 环境变量.
System.getenv(YarnConfigKeys.ENV_APP_ID);
https://github.com/apache/flink/blob/6c9bb3716a3a92f3b5326558c6238432c669556d/flink-yarn/src/main/java/org/apache/flink/yarn/YarnConfigKeys.java#L28
Best,
Feng
On Wed, Aug 30, 2023 at 7:14 PM allanqinjy wrote:
> hi，
hi，
请教大家一个问题，就是在上报指标到prometheus时候，jobname会随机生成一个后缀，看源码也是new Abstract
ID（），有方法在这里获取本次上报的作业applicationid吗？
Hi Neha,
For the jobs you care about, you can attach additional labels using
`scope-variables-additional` [1]. The example located in the same page
showcases how you can configure KV pairs in its map configuration. Be sure
to replace the reporter name with the name of your prometheus reporter!
Hello,
I have a question about the Prometheus metrics. I am able to fetch the
metrics from the following expression.
sum(flink_jobmanager_job_numRestarts{job_name="$job_name"}) by (job_name)
Now I am interested in only a few jobs and I want to give them a label. How
to achieve this? How to give an
Hi Team,
Is it possible to remove a few labels from Flink operator scope metrics as
we are noticing that sometimes those labels are too large and hence causing
unnecessary load at our monitoring platform. One such label is
operator_name.
Thanks and Regards ,
Surendra Lalwani
--
IMPORTANT
hi
你的图片挂了 可以用图床上传一下图片 在这里贴个链接 另外自定义 reportor 把需要的metric 过滤出来上报就行了
回复的原邮件
| 发件人 | 陈佳豪 |
| 发送日期 | 2022年11月28日 00:54 |
| 收件人 | user-zh |
| 主题 | 请问flink metrics如何获取任务状态？ |
自定义了一个kafka Metric Reporters #请问如何使用上述指标呢？
我想通过上报获取任务状态。除了上述指标外如果有其他方案也可以，当前flink 版本是15.2 还望大神指教一番。
hi
不好意思刚刚图好像又挂了
不知道这个能否查看。
在 2022-11-28 13:50:37，"m17610775726_1" 写道：
hi
你的图片挂了 可以用图床上传一下图片 在这里贴个链接 另外自定义 reportor 把需要的metric 过滤出来上报就行了
回复的原邮件
| 发件人 | 陈佳豪 |
| 发送日期 | 2022年11月28日 00:54 |
| 收件人 | user-zh |
| 主题 | 请问flink metrics如何获取任务状态？ |
自定义了一个kafka Metric Rep
这个metrics 我获取不到。 不知道要怎么配置才可以获取到。
在 2022-11-28 13:50:37，"m17610775726_1" 写道：
hi
你的图片挂了 可以用图床上传一下图片 在这里贴个链接 另外自定义 reportor 把需要的metric 过滤出来上报就行了
回复的原邮件
| 发件人 | 陈佳豪 |
| 发送日期 | 2022年11月28日 00:54 |
| 收件人 | user-zh |
| 主题 | 请问flink metrics如何获取任务状态？ |
自定义了一个kafka Metric
有大佬告诉下吗？ 这个指标的值获取不到。
| |
陈佳豪
邮箱：jagec...@yeah.net
|
回复的原邮件
| 发件人 | 陈佳豪 |
| 发送日期 | 2022年11月28日 00:54 |
| 收件人 | user-zh |
| 主题 | 请问flink metrics如何获取任务状态？ |
自定义了一个kafka Metric Reporters #请问如何使用上述指标呢？
我想通过上报获取任务状态。除了上述指标外如果有其他方案也可以，当前flink 版本是15.2 还望大神指教一番。
自定义了一个kafka Metric Reporters #请问如何使用上述指标呢？
我想通过上报获取任务状态。除了上述指标外如果有其他方案也可以，当前flink 版本是15.2 还望大神指教一番。
Hi Flink Community
We are using Flink version 1.13.5 for our application and every time the
job restarts, Flink Job metrics are flattened following the restart.
For e.g. we are using lastCheckpointDuration and on 05/05 our job restarted
and at the same time the checkpoint duration metric
感觉像是 二次改造的问题。可以关注几个点：
1. 指标 tag 里的 job_id 是怎么带上的，是否可能多个作业相互覆盖或者只有第一个生效的场景
2. 可以在自动修改的代码里增加更多的日志，进一步定位, 例如：在 notifyOfAddedMetric() 时打印注册了哪些 metric
> 2022年4月14日 下午6:41，QiZhu Chan 写道：
>
>
>
>
>
>
Hi，
在做Flink Metrics监控的工作过程中，有发现一个问题，Flink on yarn下，使用yarn session模式提交多个Flink
Job，只有首次提交的Job，才能正常上报Metrics；后续提交的Job，Metrics均不上报，请问是什么原因？
需要说明的是，1、InfluxdbReporter是经过二次改造的，改造后所有指标的tag均会带上job_id，方便以通过job_id查找到所有指标。2、在per-job场景下，没有这个问题，因为per-job作业拥有各自的Jobmanager。
Flink版本：1.13.3
-monitoring.html
On Wed, Feb 23, 2022 at 8:42 AM Sigalit Eliazov wrote:
> Hello. I am looking for a way to expose flink metrics via opentelemerty to
> the gcp could monitoring dashboard.
> Does anyone has experience with that?
>
> If it is not directly possible we thought about u
Hello. I am looking for a way to expose flink metrics via opentelemerty to
the gcp could monitoring dashboard.
Does anyone has experience with that?
If it is not directly possible we thought about using permethous as a
middlewere. If you have experience with that i would appreciate any
guidance
Your best bet is to create a custom reporter that does this calculation.
You could either wrap the reporter, subclass is, or fork it.
In any case,
https://github.com/apache/flink/tree/master/flink-metrics/flink-metrics-datadog
should be a good starting point.
On 01/02/2022 13:26, Jessy Ping
Hi Team,
We are using datadog and its http reporter( packaged in flink image) for
sending metrics from flink application. We do have a requirement for
setting tags with values calculated at runtime for the custom metrics
emitted from Flink. Currently, it is impossible to assign tags at runtime.
Some more background on MetricGroups:
Internally there (mostly) 3 types of metric groups:
On the one hand we have the ComponentMetricGroups (like
TaskManagerMetricGroup) that describe a high-level Flink entity, which
just add a constant expression to the logical scope(like taskmanager,
task
Upon further inspection, it seems like the user scope is not universal (i.e.
comes through the connectors and not UDFs (like rich map function)), but the
question still stands if the process makes sense.
> On Jun 1, 2021, at 10:38 AM, Mason Chen wrote:
>
> Makes sense. We are primarily
Makes sense. We are primarily concerned with removing the metric labels from
the names as the user metrics get too long. i.e. the groups from `addGroup` are
concatenated in the metric name.
Do you think there would be any issues with removing the group information in
the metric name and
The uniqueness of metrics and the naming of the Prometheus reporter are
somewhat related but also somewhat orthogonal.
Prometheus works similar to JMX in that the metric name (e.g.,
taskmanager.job.task.operator.numRecordsIn) is more or less a _class_ of
metrics, with tags/labels allowing you
Hi Mason,
The idea is that a metric is not uniquely identified by its name alone but
instead by its path. The groups in which it is defined specify this path
(similar to directories). That's why it is valid to specify two metrics
with the same name if they reside in different groups.
I think
Can anyone give insight as to why Flink allows 2 metrics with the same “name”?
For example,
getRuntimeContext.addGroup(“group”, “group1”).counter(“myMetricName”);
And
getRuntimeContext.addGroup(“other_group”,
“other_group1”).counter(“myMetricName”);
Are totally valid.
It seems that it has
This is currently not possible. See also FLINK-8358
On 4/9/2021 4:47 AM, Claude M wrote:
Hello,
I've setup Flink as an Application Cluster in Kubernetes. Now I'm
looking into monitoring the Flink cluster in Datadog. This is what is
configured in the flink-conf.yaml to emit metrics:
Hello,
I've setup Flink as an Application Cluster in Kubernetes. Now I'm looking
into monitoring the Flink cluster in Datadog. This is what is configured
in the flink-conf.yaml to emit metrics:
metrics.scope.jm: flink.jobmanager
metrics.scope.jm.job: flink.jobmanager.job
metrics.scope.tm:
Hi,
1)
Do you want to output those metrics as Flink metrics? Or output those
"metrics"/counters as values to some external system (like Kafka)? The
problem discussed in [1], was that the metrics (Counters) were not fitting
in memory, so David suggested to hold them on Flink's state
Hi flinksters,
Scenario: We have cdc messages from our rdbms(various tables) flowing to
Kafka. Our flink job reads the CDC messages and creates events based on
certain rules.
I am using Prometheus and grafana.
Following are there metrics that i need to calculate
A) Number of CDC messages wrt
hmm...in a roundabout way this could be possible I suppose.
For a given job, search through your metrics for some job metric (like
numRestarts on the JM, or any task metric for TMs), and from that you
should be able to infer the JM/TM that belongs to that (based on the TM
ID / host
Is there a way I can look into say for a specific job what’s the cpu usage
or memory usage of the yarn containers when multiple jobs are running on
the same cluster.
Also, the issue am trying to resolve is I’m seeing high memory usage for
one of the containers I want isolate the issue with one job
No, Job-/TaskManager metrics cannot be tagged with the job name.
The reason is that this only makes sense for application clusters
(opposed to session clusters), but we don't differentiate between the
two when it comes to metrics.
On 2/19/2021 3:59 AM, bat man wrote:
I meant the Flink
I meant the Flink jobname. I’m using the below reporter -
metrics.reporter.prom.class:
org.apache.flink.metrics.prometheus.PrometheusReporter
Is there any way to tag job names to the task and job manager metrics.
Thanks,
Hemant
On Fri, 19 Feb 2021 at 12:40 AM, Chesnay Schepler
wrote:
> When
When you mean "job_name", are you referring to the Prometheus concept of
jobs, of the one of Flink?
Which of Flink prometheus reporters are you using?
On 2/17/2021 7:37 PM, bat man wrote:
Hello there,
I am using prometheus to push metrics to prometheus and then use
grafana for
Hello there,
I am using prometheus to push metrics to prometheus and then use grafana
for visualization. There are metrics like
- flink_taskmanager_Status_JVM_CPU_Load,
flink_taskmanager_Status_JVM_CPU_Load,
flink_taskmanager_Status_JVM_CPU_Time
etc which do not gives job_name. It is tied to an
va:353)
>>>>> at
>>>>> org.apache.flink.streaming.runtime.tasks.OperatorChain.(OperatorChain.java:144)
>>>>> at
>>>>> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:433)
>>>>> at
streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:461)
>>>> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707)
>>>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)
>>>> at java.lang.Thread.run(Thread.java:748)
>&
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)
>>> at java.lang.Thread.run(Thread.java:748)
>>> Regards,
>>> Vijay
>>>
>>>
>>> On Wed, Aug 26, 2020 at 7:53 AM Chesnay Schepler
>>> wrote:
>>>
>&
snay Schepler
>> wrote:
>>
>>> metrics.reporter.grph.class:
>>> org.apache.flink.metrics.graphite.GraphiteReporter
>>>
>>>
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/metrics.html#graphite-orgapacheflinkmetricsgraphitegraphitere
gt;> https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/metrics.html#graphite-orgapacheflinkmetricsgraphitegraphitereporter
>>
>> On 26/08/2020 16:40, Vijayendra Yadav wrote:
>>
>> Hi Dawid,
>>
>> I have 1.10.0 version of flink. What is alt
ion of flink. What is alternative for this version ?
>
> Regards,
> Vijay
>
>
> On Aug 25, 2020, at 11:44 PM, Dawid Wysakowicz
> wrote:
>
>
>
> Hi Vijay,
>
> I think the problem might be that you are using a wrong version of the
> reporter.
&
version of flink. What is alternative for this version ?
Regards,
Vijay
On Aug 25, 2020, at 11:44 PM, Dawid Wysakowicz
wrote:
Hi Vijay,
I think the problem might be that you are using a wrong version of
the reporter.
You say you used flink-metrics-graphite-1.10.0.jar from 1.10
t;>> 81268126statsd admin
>>> <https://github.com/etsy/statsd/blob/master/docs/admin_interface.md>
>>>
>>> 2) WebUI:
>>>
>>>
>>>
>>>
>>>
>>> 3) Run Flink example Job.
>>> ./bin/flink run
>&g
11-1.11-SNAPSHOT-SocketWindowWordCount.jar
>> --port
>>
>> with conf/flink-conf.yaml set as:
>>
>> metrics.reporter.grph.factory.class:
>> org.apache.flink.metrics.graphite.GraphiteReporterFactory
>> metrics.reporter.grph.host: localhost
>> metrics.r
Hi Vijay,
I think the problem might be that you are using a wrong version of the
reporter.
You say you used flink-metrics-graphite-1.10.0.jar from 1.10 as a
plugin, but it was migrated to plugins in 1.11 only[1].
I'd recommend trying it out with the same 1.11 version of Flink and
Graphite
tory
metrics.reporter.grph.host: localhost
metrics.reporter.grph.port: 2003
metrics.reporter.grph.protocol: TCP
metrics.reporter.grph.interval: 1 SECONDS
and graphite jar:
plugins/flink-metrics-graphite/flink-metrics-graphite-1.10.0.jar
4) Can't see any activity in webui graphite.
Could you review and
low are correct?
>>
>> 1) Updated flink-conf.yaml
>>
>> metrics.reporter.grph.factory.class:
>> org.apache.flink.metrics.graphite.GraphiteReporterFactory
>> metrics.reporter.grph.host: port
>> metrics.reporter.grph.port: 9109
>> metrics.reporter.grph.protocol: TCP
>> me
metrics.reporter.grph.interval: 30 SECONDS
>
> 2) Added Graphite jar in plugin folder :
>
> ll */usr/lib/flink/plugins/metric/*
> *flink-metrics-graphite-1.10.0.jar*
>
> 3) Looking metrics in graphite server:
>
> http://port:8080/metrics <http://10.108.58.63:8080/metrics>
>
> Note: No code change is done.
>
> Regards,
> Vijay
>
>
>
:
org.apache.flink.metrics.graphite.GraphiteReporterFactory
metrics.reporter.grph.host: port
metrics.reporter.grph.port: 9109
metrics.reporter.grph.protocol: TCP
metrics.reporter.grph.interval: 30 SECONDS
2) Added Graphite jar in plugin folder :
ll */usr/lib/flink/plugins/metric/*
*flink-metrics-graphite-1.10.0.jar*
3) Looking metrics in graphite
scraping Flink metrics data with
application running in kubernetes cluster.
If taskmanager is running on multiple nodes, and prometheus requests
for the metrics data, then is that request directed to one of the
nodes(based on some strategy, like round-robin) or is data aggregated
from all the nodes
Hi,
I have a query regarding prometheus scraping Flink metrics data with
application running in kubernetes cluster.
If taskmanager is running on multiple nodes, and prometheus requests for
the metrics data, then is that request directed to one of the nodes(based
on some strategy, like round
t;>> On 06/07/2020 16:35, Chesnay Schepler wrote:
>>>>>
>>>>> Please enable debug logging and search for warnings from the metric
>>>>> groups/registry/reporter.
>>>>>
>>>>> If you cannot find anything sus
On Mon, Jul 6, 2020 at 5:24 PM Chesnay
Schepler mailto:ches...@apache.org>> wrote:
Have you looked at the SLF4J reporter?
https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/metrics.html#slf4j-orgapacheflinkmetricsslf4jslf4
;>
>>>> On 06/07/2020 16:29, Manish G wrote:
>>>>
>>>> Job is an infinite streaming one, so it keeps going. Flink
>>>> configuration is as:
>>>>
>>>> metrics.reporter.slf4j.class:
>>>> org.apache.flink.metrics.slf4j.Slf4jReporter
>&
link-docs-release-1.10/monitoring/metrics.html#slf4j-orgapacheflinkmetricsslf4jslf4jreporter
On 06/07/2020 13:49, Manish G wrote:
> Hi,
>
> Is it possible to log Flink metrics in
application logs apart from
> publishing it to Prometheus?
>
> With regards
hepler
>>> wrote:
>>>
>>>> How long did the job run for, and what is the configured interval?
>>>>
>>>>
>>>> On 06/07/2020 15:51, Manish G wrote:
>>>>
>>>> Hi,
>>>>
>>>> Thanks for this.
>>>>
>>>> I did the configuration as mentioned at the link(changes in
>>>> flink-conf.yml, copying the jar in lib directory), and registered the Meter
>>>> with metrics group and invoked markEvent() method in the target code. But I
>>>> don't see any related logs.
>>>> I am doing this all on my local computer.
>>>>
>>>> Anything else I need to do?
>>>>
>>>> With regards
>>>> Manish
>>>>
>>>> On Mon, Jul 6, 2020 at 5:24 PM Chesnay Schepler
>>>> wrote:
>>>>
>>>>> Have you looked at the SLF4J reporter?
>>>>>
>>>>>
>>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/metrics.html#slf4j-orgapacheflinkmetricsslf4jslf4jreporter
>>>>>
>>>>> On 06/07/2020 13:49, Manish G wrote:
>>>>> > Hi,
>>>>> >
>>>>> > Is it possible to log Flink metrics in application logs apart from
>>>>> > publishing it to Prometheus?
>>>>> >
>>>>> > With regards
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
On Mon, Jul 6, 2020 at 5:24 PM Chesnay Schepler
mailto:ches...@apache.org>> wrote:
Have you looked at the SLF4J reporter?
https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/metrics.html#slf4j-orgapacheflinkmetricsslf4jslf4
gt; with metrics group and invoked markEvent() method in the target code. But I
>>> don't see any related logs.
>>> I am doing this all on my local computer.
>>>
>>> Anything else I need to do?
>>>
>>> With regards
>>> Manish
>>>
>>> On Mon, Jul 6, 2020 at 5:24 PM Chesnay Schepler
>>> wrote:
>>>
>>>> Have you looked at the SLF4J reporter?
>>>>
>>>>
>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/metrics.html#slf4j-orgapacheflinkmetricsslf4jslf4jreporter
>>>>
>>>> On 06/07/2020 13:49, Manish G wrote:
>>>> > Hi,
>>>> >
>>>> > Is it possible to log Flink metrics in application logs apart from
>>>> > publishing it to Prometheus?
>>>> >
>>>> > With regards
>>>>
>>>>
>>>>
>>>
>>
>>
>
https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/metrics.html#slf4j-orgapacheflinkmetricsslf4jslf4jreporter
On 06/07/2020 13:49, Manish G wrote:
> Hi,
>
> Is it possible to log Flink metrics in applicati
PM Chesnay Schepler
>> wrote:
>>
>>> Have you looked at the SLF4J reporter?
>>>
>>>
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/metrics.html#slf4j-orgapacheflinkmetricsslf4jslf4jreporter
>>>
>>> On 06/07/2020 13:49, Manish G wrote:
>>> > Hi,
>>> >
>>> > Is it possible to log Flink metrics in application logs apart from
>>> > publishing it to Prometheus?
>>> >
>>> > With regards
>>>
>>>
>>>
>>
>
>
n 06/07/2020 13:49, Manish G wrote:
> Hi,
>
> Is it possible to log Flink metrics in application logs
apart from
> publishing it to Prometheus?
>
> With regards
//ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/metrics.html#slf4j-orgapacheflinkmetricsslf4jslf4jreporter
On 06/07/2020 13:49, Manish G wrote:
> Hi,
>
> Is it possible to log Flink metrics in application logs
apart from
>
e you looked at the SLF4J reporter?
>>
>>
>> https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/metrics.html#slf4j-orgapacheflinkmetricsslf4jslf4jreporter
>>
>> On 06/07/2020 13:49, Manish G wrote:
>> > Hi,
>> >
>> > Is it possible to log Flink metrics in application logs apart from
>> > publishing it to Prometheus?
>> >
>> > With regards
>>
>>
>>
>
reporter?
https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/metrics.html#slf4j-orgapacheflinkmetricsslf4jslf4jreporter
On 06/07/2020 13:49, Manish G wrote:
> Hi,
>
> Is it possible to log Flink metrics in application logs apart from
> publis
lf4jreporter
>
> On 06/07/2020 13:49, Manish G wrote:
> > Hi,
> >
> > Is it possible to log Flink metrics in application logs apart from
> > publishing it to Prometheus?
> >
> > With regards
>
>
>
Have you looked at the SLF4J reporter?
https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/metrics.html#slf4j-orgapacheflinkmetricsslf4jslf4jreporter
On 06/07/2020 13:49, Manish G wrote:
Hi,
Is it possible to log Flink metrics in application logs apart from
publishing
Hi,
Is it possible to log Flink metrics in application logs apart from
publishing it to Prometheus?
With regards
corresponding Meter according to tableName in the `invoke` method.
>
>
> Thanks,
> Lei
> --
> wangl...@geekplus.com.cn
>
>
> *Sender:* wangl...@geekplus.com.cn
> *Send Time:* 2020-07-03 14:27
> *Receiver:* Xintong Song
> *cc:* user
> *S
...@geekplus.com.cn
Send Time: 2020-07-03 14:27
Receiver: Xintong Song
cc: user
Subject: Re: Re: How to dynamically initialize flink metrics in invoke method
and then reuse it？
Hi Xintong,
Yes, initializing the metric in the `open` method works, but it doesn't solve
my problem.
I want to initialize
info. I want to monitor the qps by tableName. The
tableName is different for every record.
Thanks,
Lei
wangl...@geekplus.com.cn
Sender: Xintong Song
Send Time: 2020-07-03 13:14
Receiver: wangl...@geekplus.com.cn
cc: user
Subject: Re: How to dynamically initialize flink metrics in invoke
.cn> wrote:
>
> In one flink operator, i want to initialize multiple flink metrics
> according to message content.
> As the code below.
>
> public void invoke(ObjectNode node, Context context) throws Exception {
>
> String tableName = node.get("metadata").
In one flink operator, i want to initialize multiple flink metrics according to
message content.
As the code below.
public void invoke(ObjectNode node, Context context) throws Exception {
String tableName = node.get("metadata").get("topic").asText();
Meter meter
Hi Gary,
Sorry for the false alarm. It's caused by a bug in my deployment - no
metrics were added into the registry.
Sorry for wasting your time.
Thanks and best regards,
Averell
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Hi Gary,
Thanks for the help.
Here below is the output from jstack. It seems not being blocked.
In my JobManager log, there's this WARN, I am not sure whether it's relevant
at all.
Attached is the full jstack dump k8xDump.txt
NewRelic (using a custom ScheduledDropwizardReporter).
>
> From the logs, I could see that an instance of ScheduledDropwizardReporter
> has already been created successfully (the overridden getReporter() method
> <https://github.com/apache/flink/blob/e346215edcf2252cc60c5cef507e
hod
<https://github.com/apache/flink/blob/e346215edcf2252cc60c5cef507ea77ce2ac9aca/flink-metrics/flink-metrics-dropwizard/src/main/java/org/apache/flink/dropwizard/ScheduledDropwizardReporter.java#L234>
was called).
An instance of MetricRegistryImpl
<https://github.com/apach
式GET和POST的区别。
>
> 然后去flink-metrics-prometheus包下面找到org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter#report这个方法，将它推送方式修改一下，重新打包，就可以了。很高兴能帮到你。
> 详细排查过程，参考我的文章：
> https://daijiguo.blog.csdn.net/article/details/105453643
>
>
>
>
>
>
> --原始
BUG??
metrics.reporter.promgateway.randomJobNameSuffixpushgateway??GET??POST
??flink-metrics-prometheus
您好！
我在使用prometheus监控flink时发现一个问题不知是不是bug,反映如下
版本信息
Flink 1.9.1
Prometheus 2.18
pushgateway 1.2.0
问题：
配置
metrics.reporter.promgateway.randomJobNameSuffix为false后，部分metrics不能正确的push到pushgateway里。具体表现是，部分metrics（主要是jobmanager相关，如
flink_jobmanager_Status_JVM_CPU_Load
]
From: Chesnay Schepler
Sent: Wednesday, January 22, 2020 6:07 PM
To: Sidney Feiner ; flink-u...@apache.org
Subject: Re: Flink Metrics - PrometheusReporter
Metrics are exposed via reporters by each process separately, whereas the WebUI
aggregates metrics.
As such you have to configure
in
locally on my computer, I would access the port I configured and see
all the metrics I've created.
In production, we use High Availability mode and when I try to access
the JobManager's metrics in the port I've configured on the
PrometheusReporter, I see some very basic metrics - default Flink
on the PrometheusReporter, I
see some very basic metrics - default Flink metrics, but I can't see any of my
custom metrics.
Weird thing is I can see those metrics through Flink's UI in the Metrics tab:
[cid:dc6050e2-a947-4856-8339-5daea66b6a77]
Does anybody have a clue why my custom metrics
What about using an accumulator? Does it work for you needs?
Il Sab 18 Gen 2020, 10:03 Soheil Pourbafrani ha
scritto:
> Hi,
>
> I'm using Flink to insert some processed records into the database. I need
> to have some aggregated information about records inserted into the
> database so far. For
Hi,
I'm using Flink to insert some processed records into the database. I need
to have some aggregated information about records inserted into the
database so far. For example, for a specific column value, I need to know
how many records have been inserted. Can I use the Flink Matrics to provide
etrics数据无法插入，不影响整体使用。
>
>祝好
>唐云
>
>
>From: 张江
>Sent: Saturday, January 4, 2020 19:14
>To: user-zh ; myas...@live.com
>Subject: 回复：使用influxdb作为flink metrics reporter
>
>
>你好，
>
>
>我看我这里报错的问题是invalid boolean，并不是NaN/infinity value造成的，不知道是什么原因？
>
>
>而且我用的fl
; myas...@live.com
Subject: 回复：使用influxdb作为flink metrics reporter
你好，
我看我这里报错的问题是invalid boolean，并不是NaN/infinity value造成的，不知道是什么原因？
而且我用的flink是1.9.1版本，influxdb是1.7.9版本。
祝好，
<https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1=%E5%BC%A0%E6%B1%9F=zjkingdom2010%40163.
metrics reporter
大家好，
我按照官网所介绍的flink metrics reporter设置，选用了influxdb，进行了如下设置：
metrics.reporter.influxdb.class:org.apache.flink.metrics.influxdb.InfluxdbReportermetrics.reporter.influxdb.host:localhostmetrics.reporter.influxdb.port
Sent: Friday, January 3, 2020 21:22
To: user-zh@flink.apache.org
Subject: 使用influxdb作为flink metrics reporter
大家好，
我按照官网所介绍的flink metrics reporter设置，选用了influxdb，进行了如下设置
大家好，
我按照官网所介绍的flink metrics reporter设置，选用了influxdb，进行了如下设置：
metrics.reporter.influxdb.class:org.apache.flink.metrics.influxdb.InfluxdbReportermetrics.reporter.influxdb.host:localhostmetrics.reporter.influxdb.port:8086metrics.reporter.influxdb.db:flinkmetrics.reporter.influxdb.username:flink
ecommended for a streaming job？
Best,Vino
M Singh 于2019年12月24日周二 下午4:02写道：
Hi:
I wanted to find out what's the best way of collecting Flink metrics using
Prometheus in a streaming application on EMR/Hadoop.
Since the Flink streaming jobs could be running on any node - is there any
Prome
aming job？
>
> Best,
> Vino
>
> M Singh 于2019年12月24日周二 下午4:02写道：
>
>> Hi:
>>
>> I wanted to find out what's the best way of collecting Flink metrics
>> using Prometheus in a streaming application on EMR/Hadoop.
>>
>> Since the Flink streami
treaming job？
Best,
Vino
M Singh 于2019年12月24日周二 下午4:02写道：
> Hi:
>
> I wanted to find out what's the best way of collecting Flink metrics using
> Prometheus in a streaming application on EMR/Hadoop.
>
> Since the Flink streaming jobs could be running on any node - is there
Hi:
I wanted to find out what's the best way of collecting Flink metrics using
Prometheus in a streaming application on EMR/Hadoop.
Since the Flink streaming jobs could be running on any node - is there any
Prometheus configuration or service discovery option available that will
dynamically
Thanks Vino and Biao for your help. Mans
On Thursday, December 19, 2019, 02:25:40 AM EST, Biao Liu
wrote:
Hi Mans,
That's indeed a problem. We have a plan to fix it. I think it could be included
in 1.11. You could follow this issue [1] to check the progress.
[1]
Hi Mans,
That's indeed a problem. We have a plan to fix it. I think it could be
included in 1.11. You could follow this issue [1] to check the progress.
[1] https://issues.apache.org/jira/browse/FLINK-9543
Thanks,
Biao /'bɪ.aʊ/
On Thu, 19 Dec 2019 at 14:51, vino yang wrote:
> Hi Mans,
>
>
Hi Mans,
IMO, one job manager represents one Flink cluster and one Flink cluster has
a suite of Flink configuration e.g. metrics reporter.
Some metrics reporters support tag feature, you can specify it to
distinguish different Flink cluster.[1]
[1]:
1 - 100 of 171 matches
Mail list logo
