[jira] [Commented] (FLINK-6464) Metric name is not stable
[
https://issues.apache.org/jira/browse/FLINK-6464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345169#comment-16345169
]
ASF GitHub Bot commented on FLINK-6464:
---
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/5332
> Metric name is not stable
> -
>
> Key: FLINK-6464
> URL: https://issues.apache.org/jira/browse/FLINK-6464
> Project: Flink
> Issue Type: Bug
> Components: DataStream API, Metrics
>Affects Versions: 1.2.0
>Reporter: Andrey
>Assignee: Chesnay Schepler
>Priority: Critical
> Fix For: 1.5.0
>
>
> Currently according to the documentation
> (https://ci.apache.org/projects/flink/flink-docs-release-1.2/monitoring/metrics.html)
> operator metrics constructed using the following pattern:
> ,
> For some operators, "operator_name" could contain default implementation of
> toString method. For example:
> {code}
> TriggerWindow(TumblingProcessingTimeWindows(3000),
> ListStateDescriptor{serializer=org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer@c65792d4},
> xxx.Trigger@665fe457, WindowedStream.apply(WindowedStream.java:521)) ->
> Sink: Unnamed
> {code}
> The part "@c65792d4" will be changed every time job is restarted/cancelled.
> As a consequence it's not possible to store metrics for a long time.
> Expected:
> * ensure all operators return human readable, non-default names OR
> * change the way TriggerWindow generates it's name.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-6464) Metric name is not stable
[
https://issues.apache.org/jira/browse/FLINK-6464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343220#comment-16343220
]
ASF GitHub Bot commented on FLINK-6464:
---
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/5332
merging.
> Metric name is not stable
> -
>
> Key: FLINK-6464
> URL: https://issues.apache.org/jira/browse/FLINK-6464
> Project: Flink
> Issue Type: Bug
> Components: DataStream API, Metrics
>Affects Versions: 1.2.0
>Reporter: Andrey
>Assignee: Chesnay Schepler
>Priority: Critical
> Fix For: 1.5.0, 1.4.1
>
>
> Currently according to the documentation
> (https://ci.apache.org/projects/flink/flink-docs-release-1.2/monitoring/metrics.html)
> operator metrics constructed using the following pattern:
> ,
> For some operators, "operator_name" could contain default implementation of
> toString method. For example:
> {code}
> TriggerWindow(TumblingProcessingTimeWindows(3000),
> ListStateDescriptor{serializer=org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer@c65792d4},
> xxx.Trigger@665fe457, WindowedStream.apply(WindowedStream.java:521)) ->
> Sink: Unnamed
> {code}
> The part "@c65792d4" will be changed every time job is restarted/cancelled.
> As a consequence it's not possible to store metrics for a long time.
> Expected:
> * ensure all operators return human readable, non-default names OR
> * change the way TriggerWindow generates it's name.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-6464) Metric name is not stable
[
https://issues.apache.org/jira/browse/FLINK-6464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339295#comment-16339295
]
ASF GitHub Bot commented on FLINK-6464:
---
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/5332
Looks good
> Metric name is not stable
> -
>
> Key: FLINK-6464
> URL: https://issues.apache.org/jira/browse/FLINK-6464
> Project: Flink
> Issue Type: Bug
> Components: DataStream API, Metrics
>Affects Versions: 1.2.0
>Reporter: Andrey
>Assignee: Chesnay Schepler
>Priority: Critical
> Fix For: 1.5.0, 1.4.1
>
>
> Currently according to the documentation
> (https://ci.apache.org/projects/flink/flink-docs-release-1.2/monitoring/metrics.html)
> operator metrics constructed using the following pattern:
> ,
> For some operators, "operator_name" could contain default implementation of
> toString method. For example:
> {code}
> TriggerWindow(TumblingProcessingTimeWindows(3000),
> ListStateDescriptor{serializer=org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer@c65792d4},
> xxx.Trigger@665fe457, WindowedStream.apply(WindowedStream.java:521)) ->
> Sink: Unnamed
> {code}
> The part "@c65792d4" will be changed every time job is restarted/cancelled.
> As a consequence it's not possible to store metrics for a long time.
> Expected:
> * ensure all operators return human readable, non-default names OR
> * change the way TriggerWindow generates it's name.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-6464) Metric name is not stable
[
https://issues.apache.org/jira/browse/FLINK-6464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16337608#comment-16337608
]
ASF GitHub Bot commented on FLINK-6464:
---
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/5332
@aljoscha Done, we now include the function name instead. Is a bit more
complicated that I had hoped though...
> Metric name is not stable
> -
>
> Key: FLINK-6464
> URL: https://issues.apache.org/jira/browse/FLINK-6464
> Project: Flink
> Issue Type: Bug
> Components: DataStream API, Metrics
>Affects Versions: 1.2.0
>Reporter: Andrey
>Assignee: Chesnay Schepler
>Priority: Critical
> Fix For: 1.5.0, 1.4.1
>
>
> Currently according to the documentation
> (https://ci.apache.org/projects/flink/flink-docs-release-1.2/monitoring/metrics.html)
> operator metrics constructed using the following pattern:
> ,
> For some operators, "operator_name" could contain default implementation of
> toString method. For example:
> {code}
> TriggerWindow(TumblingProcessingTimeWindows(3000),
> ListStateDescriptor{serializer=org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer@c65792d4},
> xxx.Trigger@665fe457, WindowedStream.apply(WindowedStream.java:521)) ->
> Sink: Unnamed
> {code}
> The part "@c65792d4" will be changed every time job is restarted/cancelled.
> As a consequence it's not possible to store metrics for a long time.
> Expected:
> * ensure all operators return human readable, non-default names OR
> * change the way TriggerWindow generates it's name.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-6464) Metric name is not stable
[
https://issues.apache.org/jira/browse/FLINK-6464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16337421#comment-16337421
]
ASF GitHub Bot commented on FLINK-6464:
---
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/5332
> Metric name is not stable
> -
>
> Key: FLINK-6464
> URL: https://issues.apache.org/jira/browse/FLINK-6464
> Project: Flink
> Issue Type: Bug
> Components: DataStream API, Metrics
>Affects Versions: 1.2.0
>Reporter: Andrey
>Assignee: Chesnay Schepler
>Priority: Critical
> Fix For: 1.5.0, 1.4.1
>
>
> Currently according to the documentation
> (https://ci.apache.org/projects/flink/flink-docs-release-1.2/monitoring/metrics.html)
> operator metrics constructed using the following pattern:
> ,
> For some operators, "operator_name" could contain default implementation of
> toString method. For example:
> {code}
> TriggerWindow(TumblingProcessingTimeWindows(3000),
> ListStateDescriptor{serializer=org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer@c65792d4},
> xxx.Trigger@665fe457, WindowedStream.apply(WindowedStream.java:521)) ->
> Sink: Unnamed
> {code}
> The part "@c65792d4" will be changed every time job is restarted/cancelled.
> As a consequence it's not possible to store metrics for a long time.
> Expected:
> * ensure all operators return human readable, non-default names OR
> * change the way TriggerWindow generates it's name.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-6464) Metric name is not stable
[
https://issues.apache.org/jira/browse/FLINK-6464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334741#comment-16334741
]
ASF GitHub Bot commented on FLINK-6464:
---
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/5332
How about we replace the line-number with the UDF class name?
> Metric name is not stable
> -
>
> Key: FLINK-6464
> URL: https://issues.apache.org/jira/browse/FLINK-6464
> Project: Flink
> Issue Type: Bug
> Components: DataStream API, Metrics
>Affects Versions: 1.2.0
>Reporter: Andrey
>Assignee: Chesnay Schepler
>Priority: Critical
> Fix For: 1.5.0, 1.4.1
>
>
> Currently according to the documentation
> (https://ci.apache.org/projects/flink/flink-docs-release-1.2/monitoring/metrics.html)
> operator metrics constructed using the following pattern:
> ,
> For some operators, "operator_name" could contain default implementation of
> toString method. For example:
> {code}
> TriggerWindow(TumblingProcessingTimeWindows(3000),
> ListStateDescriptor{serializer=org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer@c65792d4},
> xxx.Trigger@665fe457, WindowedStream.apply(WindowedStream.java:521)) ->
> Sink: Unnamed
> {code}
> The part "@c65792d4" will be changed every time job is restarted/cancelled.
> As a consequence it's not possible to store metrics for a long time.
> Expected:
> * ensure all operators return human readable, non-default names OR
> * change the way TriggerWindow generates it's name.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-6464) Metric name is not stable
[
https://issues.apache.org/jira/browse/FLINK-6464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334556#comment-16334556
]
ASF GitHub Bot commented on FLINK-6464:
---
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/5332
Changes look good!
It might make sense to also remove the line number from the string because
this might also change in the future. It's not a must, though.
> Metric name is not stable
> -
>
> Key: FLINK-6464
> URL: https://issues.apache.org/jira/browse/FLINK-6464
> Project: Flink
> Issue Type: Bug
> Components: DataStream API, Metrics
>Affects Versions: 1.2.0
>Reporter: Andrey
>Assignee: Chesnay Schepler
>Priority: Critical
> Fix For: 1.5.0, 1.4.1
>
>
> Currently according to the documentation
> (https://ci.apache.org/projects/flink/flink-docs-release-1.2/monitoring/metrics.html)
> operator metrics constructed using the following pattern:
> ,
> For some operators, "operator_name" could contain default implementation of
> toString method. For example:
> {code}
> TriggerWindow(TumblingProcessingTimeWindows(3000),
> ListStateDescriptor{serializer=org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer@c65792d4},
> xxx.Trigger@665fe457, WindowedStream.apply(WindowedStream.java:521)) ->
> Sink: Unnamed
> {code}
> The part "@c65792d4" will be changed every time job is restarted/cancelled.
> As a consequence it's not possible to store metrics for a long time.
> Expected:
> * ensure all operators return human readable, non-default names OR
> * change the way TriggerWindow generates it's name.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-6464) Metric name is not stable
[
https://issues.apache.org/jira/browse/FLINK-6464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334297#comment-16334297
]
ASF GitHub Bot commented on FLINK-6464:
---
GitHub user zentol opened a pull request:
https://github.com/apache/flink/pull/5332
[FLINK-6464][streaming] Stabilize default window operator names
## What is the purpose of the change
This PR modifies the generation of default operator names for window
operators to be more concise, and more importantly, stable across job
submissions.
Example: SocketTextWordCount
Before:
```
TriggerWindow(TumblingProcessingTimeWindows(5000),
ReducingStateDescriptor{serializer=org.apache.flink.api.java.typeutils.runtime.PojoSerializer@591ae253,
reduceFunction=org.apache.flink.streaming.examples.socket.SocketWindowWordCount$1@48974e45},
ProcessingTimeTrigger(), WindowedStream.reduce(WindowedStream.java:241))
```
After:
```
Window(TumblingProcessingTimeWindows(5000), ProcessingTimeTrigger,
WindowedStream.reduce(WindowedStream.java:243))
```
## Brief change log
* create shared static utility method to generate names
* remove state-descriptor from operator name
* replace `TriggerWindow` with `Window`
* trigger/assigner are now included with their simple class name instead of
fully qualified class name + object reference
## Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zentol/flink 6464
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5332.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5332
commit 065e5420d9382b15fc32c2738c2ede33aee69d38
Author: zentol
Date: 2018-01-22T13:46:02Z
[FLINK-6464][streaming] Stabilize default window operator names
> Metric name is not stable
> -
>
> Key: FLINK-6464
> URL: https://issues.apache.org/jira/browse/FLINK-6464
> Project: Flink
> Issue Type: Bug
> Components: DataStream API, Metrics
>Affects Versions: 1.2.0
>Reporter: Andrey
>Assignee: Chesnay Schepler
>Priority: Critical
> Fix For: 1.5.0, 1.4.1
>
>
> Currently according to the documentation
> (https://ci.apache.org/projects/flink/flink-docs-release-1.2/monitoring/metrics.html)
> operator metrics constructed using the following pattern:
> ,
> For some operators, "operator_name" could contain default implementation of
> toString method. For example:
> {code}
> TriggerWindow(TumblingProcessingTimeWindows(3000),
> ListStateDescriptor{serializer=org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer@c65792d4},
> xxx.Trigger@665fe457, WindowedStream.apply(WindowedStream.java:521)) ->
> Sink: Unnamed
> {code}
> The part "@c65792d4" will be changed every time job is restarted/cancelled.
> As a consequence it's not possible to store metrics for a long time.
> Expected:
> * ensure all operators return human readable, non-default names OR
> * change the way TriggerWindow generates it's name.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-6464) Metric name is not stable
[
https://issues.apache.org/jira/browse/FLINK-6464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064482#comment-16064482
]
Chesnay Schepler commented on FLINK-6464:
-
This is something that we should solve in general, and not rely on every single
reporter to deal with it. We should also decouple this issue with the length of
the names, as the latter is generally a reporter specific issue.
I'm not quite sure where the names are generated, but probably its either
various DataStream classes or StreamjobGraphGenerator.
For the example above, IMO the only info we actually need is
{{TumblingProcessingTimeWindow(3000, ListStateDescriptor, )}}.
> Metric name is not stable
> -
>
> Key: FLINK-6464
> URL: https://issues.apache.org/jira/browse/FLINK-6464
> Project: Flink
> Issue Type: Bug
> Components: DataStream API, Metrics
>Affects Versions: 1.2.0
>Reporter: Andrey
>
> Currently according to the documentation
> (https://ci.apache.org/projects/flink/flink-docs-release-1.2/monitoring/metrics.html)
> operator metrics constructed using the following pattern:
> ,
> For some operators, "operator_name" could contain default implementation of
> toString method. For example:
> {code}
> TriggerWindow(TumblingProcessingTimeWindows(3000),
> ListStateDescriptor{serializer=org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer@c65792d4},
> xxx.Trigger@665fe457, WindowedStream.apply(WindowedStream.java:521)) ->
> Sink: Unnamed
> {code}
> The part "@c65792d4" will be changed every time job is restarted/cancelled.
> As a consequence it's not possible to store metrics for a long time.
> Expected:
> * ensure all operators return human readable, non-default names OR
> * change the way TriggerWindow generates it's name.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-6464) Metric name is not stable
[
https://issues.apache.org/jira/browse/FLINK-6464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064097#comment-16064097
]
David Brinegar commented on FLINK-6464:
---
nice find! FLINK-7009 tries to address this by removing the instance ids, then
using a hash of the remaining stable part of the string as a compressed metric
name. So the above would convert into something like "TriggerWin_abcdef12"
which is at least the same every time you run the job, and short so metric
systems can handle it without truncation or conversion problems, but in the end
only a shorter more stable default name, not particularly readable in itself.
Thoughts?
> Metric name is not stable
> -
>
> Key: FLINK-6464
> URL: https://issues.apache.org/jira/browse/FLINK-6464
> Project: Flink
> Issue Type: Bug
> Components: DataStream API, Metrics
>Affects Versions: 1.2.0
>Reporter: Andrey
>
> Currently according to the documentation
> (https://ci.apache.org/projects/flink/flink-docs-release-1.2/monitoring/metrics.html)
> operator metrics constructed using the following pattern:
> ,
> For some operators, "operator_name" could contain default implementation of
> toString method. For example:
> {code}
> TriggerWindow(TumblingProcessingTimeWindows(3000),
> ListStateDescriptor{serializer=org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer@c65792d4},
> xxx.Trigger@665fe457, WindowedStream.apply(WindowedStream.java:521)) ->
> Sink: Unnamed
> {code}
> The part "@c65792d4" will be changed every time job is restarted/cancelled.
> As a consequence it's not possible to store metrics for a long time.
> Expected:
> * ensure all operators return human readable, non-default names OR
> * change the way TriggerWindow generates it's name.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
