[jira] [Commented] (FLINK-10452) Expose Additional Metrics to Reason about Statesize
[ https://issues.apache.org/jira/browse/FLINK-10452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17336571#comment-17336571 ] Flink Jira Bot commented on FLINK-10452: This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Expose Additional Metrics to Reason about Statesize > --- > > Key: FLINK-10452 > URL: https://issues.apache.org/jira/browse/FLINK-10452 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics, Runtime / State Backends >Reporter: Konstantin Knauf >Priority: Major > Labels: stale-major > > For monitoring purposes it would be helpful, if Flink could expose metrics > about the number of keys/windows for each registered keyed state. > Open Questions: > * One Metric per Registered State? One Metric per KeyedOperator? > * Performance Impact (should this be default behavior?) > * Possible to know the number of windows during runtime? > * RocksDB only gives you an estimate of the number keys. Would be nice if we > could derive the exact number inside Flink. This would also help in sizing > the RocksDB instances and estimated their memory footprint. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-10452) Expose Additional Metrics to Reason about Statesize
[ https://issues.apache.org/jira/browse/FLINK-10452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17328549#comment-17328549 ] Flink Jira Bot commented on FLINK-10452: This major issue is unassigned and itself and all of its Sub-Tasks have not been updated for 30 days. So, it has been labeled "stale-major". If this ticket is indeed "major", please either assign yourself or give an update. Afterwards, please remove the label. In 7 days the issue will be deprioritized. > Expose Additional Metrics to Reason about Statesize > --- > > Key: FLINK-10452 > URL: https://issues.apache.org/jira/browse/FLINK-10452 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics, Runtime / State Backends >Reporter: Konstantin Knauf >Priority: Major > Labels: stale-major > > For monitoring purposes it would be helpful, if Flink could expose metrics > about the number of keys/windows for each registered keyed state. > Open Questions: > * One Metric per Registered State? One Metric per KeyedOperator? > * Performance Impact (should this be default behavior?) > * Possible to know the number of windows during runtime? > * RocksDB only gives you an estimate of the number keys. Would be nice if we > could derive the exact number inside Flink. This would also help in sizing > the RocksDB instances and estimated their memory footprint. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-10452) Expose Additional Metrics to Reason about Statesize
[ https://issues.apache.org/jira/browse/FLINK-10452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711185#comment-16711185 ] vinoyang commented on FLINK-10452: -- I will agree with [~stevenz3wu] to count the total number of keys, which will make a lot of sense for monitoring. It is not easy to count the state size in the keyed state backend, and the overhead is relatively large. What do you think? [~Zentol] [~srichter] > Expose Additional Metrics to Reason about Statesize > --- > > Key: FLINK-10452 > URL: https://issues.apache.org/jira/browse/FLINK-10452 > Project: Flink > Issue Type: Improvement > Components: Metrics >Reporter: Konstantin Knauf >Assignee: vinoyang >Priority: Major > > For monitoring purposes it would be helpful, if Flink could expose metrics > about the number of keys/windows for each registered keyed state. > Open Questions: > * One Metric per Registered State? One Metric per KeyedOperator? > * Performance Impact (should this be default behavior?) > * Possible to know the number of windows during runtime? > * RocksDB only gives you an estimate of the number keys. Would be nice if we > could derive the exact number inside Flink. This would also help in sizing > the RocksDB instances and estimated their memory footprint. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10452) Expose Additional Metrics to Reason about Statesize
[ https://issues.apache.org/jira/browse/FLINK-10452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696156#comment-16696156 ] Steven Zhen Wu commented on FLINK-10452: I saw two tickets that I filed are marked as duplicate of this one. let me capture the asks from the two closed tickets 1) total number of keys. it might make sense to provide two versions: exact count and approximate count. For some state backend (likes rocksDB), it may be much cheaper to get approximate count. Additionally, it will also be useful to get the total count of timers, which are also stored in state backend. Stefan mentioned timers are just a different namespace in state backend (e.g. column family in rocksDB). So it will be very useful if the metrics have _namespace_ tag. 2) With incremental checkpoint, checkpoint size only captures the delta size. It will be very useful if there is another metric that captures total state size. even an approximate number would be super useful. > Expose Additional Metrics to Reason about Statesize > --- > > Key: FLINK-10452 > URL: https://issues.apache.org/jira/browse/FLINK-10452 > Project: Flink > Issue Type: Improvement > Components: Metrics >Reporter: Konstantin Knauf >Assignee: vinoyang >Priority: Major > > For monitoring purposes it would be helpful, if Flink could expose metrics > about the number of keys/windows for each registered keyed state. > Open Questions: > * One Metric per Registered State? One Metric per KeyedOperator? > * Performance Impact (should this be default behavior?) > * Possible to know the number of windows during runtime? > * RocksDB only gives you an estimate of the number keys. Would be nice if we > could derive the exact number inside Flink. This would also help in sizing > the RocksDB instances and estimated their memory footprint. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
