[
https://issues.apache.org/jira/browse/FLINK-4812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270693#comment-16270693
]
ASF GitHub Bot commented on FLINK-4812:
---------------------------------------
GitHub user zentol opened a pull request:
https://github.com/apache/flink/pull/5100
[FLINK-4812][metrics] Expose currentLowWatermark for all operators
This is an alternative version of #5092.
## What is the purpose of the change
This PR makes all operators expose the current input/output watermark
through the metric system.
Generally, watermarks are measured separately; input watermarks are
measured in `AbstractStreamOperator#processWatermark(Watermark)` and output are
measured in the `Output` of the respective operator.
Sub-classes may specify to expose the same watermark as input/output by
overriding `AbstractStreamOperator#useSeparateWatermarkGauges`, which is used
for sources, sink and watermark assigners.
## Brief change log
* remove watermark metric logic from `Stream[Two]InputProcessor`
* this implies that this metric is no longer measured at the task level
at all
* introduce `WatermarkGauge` class to decouple metric from local state of
operator classes (i.e. some currentLowWatermark field)
* setup `WatermarkGauge`s in `AbstractStreamOperator#setup`
* introduce `AbstractStreamOperator#useSeparateWatermarkGauges` to
control re-use behavior
* measure the input watermark received by operators in
* `AbstractStreamOperator#processWatermark`
* `AsyncWaitOperator#processWatermark`
* `ContinuousFileReaderOperator#processWatermark`
* measure the output watermark emitted by operators in the `Output`
* rename `CountingOutput` to `OutputWithMetrics`
* update metrics reference in the documentation
## Verifying this change
This change modified the following tests:
*
`TimestampsAndPunctuatedWatermarksOperatorTest#testTimestampsAndPeriodicWatermarksOperator`
*
`TimestampsAndPeriodicWatermarksOperatorTest#testTimestampsAndPeriodicWatermarksOperator`
* `AsyncWaitOperatorTest#testEventTime`
This change added the following tests:
* `WatermarkGaugeTest`
* `StreamOperatorWatermarkMetricsTest`
* `StreamSourceOperatorTest#testWatermarkMetrics`
* `StreamSinkOperatorTest#testWatermarkMetrics`
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (**yes**)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zentol/flink 4812b
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5100.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5100
----
commit 6bbc92d248fe3b7f50025c49dc38f0e8cb2b7883
Author: zentol <[email protected]>
Date: 2017-11-29T11:46:59Z
[FLINK-4812][metrics] Expose currentLowWatermark for all operators
----
> Report Watermark metrics in all operators
> -----------------------------------------
>
> Key: FLINK-4812
> URL: https://issues.apache.org/jira/browse/FLINK-4812
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
> Reporter: Robert Metzger
> Assignee: Chesnay Schepler
> Priority: Critical
> Fix For: 1.5.0
>
>
> As reported by a user, Flink does currently not export the current low
> watermark for sources
> (http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/currentLowWatermark-metric-not-reported-for-all-tasks-td13770.html).
> This JIRA is for adding such a metric for the sources as well.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
