[
https://issues.apache.org/jira/browse/FLINK-33681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17800234#comment-17800234
]
Zhanghao Chen commented on FLINK-33681:
---------------------------------------
[~renqs] Hi, I'm reaching out to you to have some inputs. From the comments of
[FLINK-11576|https://issues.apache.org/jira/browse/FLINK-11576] &
[FLIP-33|https://cwiki.apache.org/confluence/display/FLINK/FLIP-33%3A+Standardize+Connector+Metrics],
it seems that at least in some previous versions of Flink, the I/O metrics of
the source/sink can be seen from the web dashboard. However, I verified that
this is not the case in Flink 1.17 & 1.18 (see the screenshot attached). As I
do not have a working environment of ealier versions of Flink, not sure if this
is a bug fix or just an improvement.
After investigation, the current version of Flink records task I/O metrics as
follows:
#
numBytesIn: collected in InputGateWithMetrics, which only accounts for Flink
internal traffic. Before
[FLINK-29567|https://issues.apache.org/jira/browse/FLINK-29567], the
operator-level numBytesIn & numBytesOut counter reuses the task-level I/O
counters, so Flink is able to collect source input bytes metrics as well.
#
numBytesOut: collected in ResultPartition, which only accounts for Flink
internal traffic as well.Before
[FLINK-29567|https://issues.apache.org/jira/browse/FLINK-29567], the
operator-level numBytesIn & numBytesOut counter reuses the task-level I/O
counters, so Flink is able to collect sink output bytes metrics as well.
#
numRecordsIn: reuse the numRecordsIn metric of the head operators SingleInput
MultitpleInput. I didn't find evidence on how Flink was able to collect source
numberRecordsIn metrics in previous versions.
#
numRecordsOut: reuse the numRecordsOut metric of the tail operator in the
opeartor chain. Specifically, the reuse will skip if the opeartor is a
SinkWriterOperator, which means sink output metrics won't be exposed to the UI
as well.
We can see that the I/O metrics of the source/sink are not integrated into the
task-level I/O metrics now, and hence cannot be seen on the UI.
Fix proposal: just reuse opeartor I/O records/bytes metrics for task I/O
metrics for SourceOperator & SinkWriterOperator.
Looking forward to your suggestions on it.
> Display source/sink numRecordsIn/Out & numBytesIn/Out on UI
> -----------------------------------------------------------
>
> Key: FLINK-33681
> URL: https://issues.apache.org/jira/browse/FLINK-33681
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Metrics
> Affects Versions: 1.18.0, 1.17.2
> Reporter: Zhanghao Chen
> Priority: Major
> Attachments: screenshot-20231224-195605-1.png
>
>
> Currently, the numRecordsIn & numBytesIn metrics for sources and the
> numRecordsOut & numBytesOut metrics for sinks are always 0 on the Flink web
> dashboard.
> FLINK-11576 brings us these metrics on the opeartor level, but it does not
> integrate them on the task level. On the other hand, the summay metrics on
> the job overview page is based on the task level I/O metrics. As a result,
> even though new connectors supporting FLIP-33 metrics will report
> operator-level I/O metrics, we still cannot see the metrics on dashboard.
> This ticket serves as an umbrella issue to integrate standard source/sink I/O
> metrics with the corresponding task I/O metrics.
> !screenshot-20231224-195605-1.png|width=608,height=333!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
