Hi, Thank you for your description. What I tried to understand is what the counter value is at that moment of spikes. Grafana would take the average out of a continuous data values before rendering result to UI. That is, if the metrics value is not transmitted continuously, where at some data point appears to be zeros, then the average value over time would be lower than the snapshot value. I would suggest to first check what the value is by zooming into the minimum scale in term of data retention policy set in Graphite. (per minute, or second, depending on settings)
I actually do not have concrete answer for that counter in Flink. Perhaps someone knows better on the semantics of this metrics would. However, there is a possibility which we have observed similarly in other Java application. This usually happens to a fast-growing counter, when its next proceeding value exceeds its positive upper bound. Normally, metrics library does not reset its value to 0. If I remember correctly, Long.MAX_VALUE + 1 = Long.MIN_VALUE, take long data type for example. Therefore, taking NonNegativeDerivative( delta ) results in a very high peak in graph. Hope this helps. On Sun, Sep 17, 2017 at 11:02 PM, Philipp Bussche <philipp.buss...@gmail.com > wrote: > Hi, thank you for your answer. So for September 11th which is shown on the > screenshot I had the counter sitting at 26.91k where when the drop happened > it was going down to 26.01k. This happened 3 times during that day and it > was always going back to the same value. > Philipp > > > > -- > Sent from: http://apache-flink-user-mailing-list-archive.2336050. > n4.nabble.com/ >
