Nikolay, Could you please explain deeper what structure will be of PME histogram?
чт, 25 июл. 2019 г. в 11:56, Nikolay Izhikov <nizhi...@apache.org>: > Hello, Nikita. > > I think > > > 1. The totalCacheOperationsBlockedDuration metric that will accumulate > > all blocking durations that happen after node starts. > > No, we don't need it. > > > 2. Blocking duration histogram. Based on the HistogramMetric class. > > Yes, we need it. > > В Чт, 25/07/2019 в 11:50 +0300, Nikita Amelchev пишет: > > Igniters, > > > > All want to see the сacheOperationsBlockedDuration metric that will > > show current blocking duration or 0 if there is no blocking right now. > > > > Do we need the following metrics? It seems one of them will be > superfluous. > > 1. The totalCacheOperationsBlockedDuration metric that will accumulate > > all blocking durations that happen after node starts. > > 2. Blocking duration histogram. Based on the HistogramMetric class. > > User will be able to configure bounds. > > > > ср, 24 июл. 2019 г. в 18:26, Nikolay Izhikov <nizhi...@apache.org>: > > > > > > Guys. > > > > > > I think we should go with the 2 metrics > > > > > > * current PME duration (resets on finish) > > > > > > This metric required for alerting(or automatic > actions) on long PME. > > > > > > * PME duration histogram (value added to metrics on PME finish) > > > This metric required for an: > > > * Quick PME trend analysis > > > * Quick PME history analysis > > > > > > > > > В Ср, 24/07/2019 в 15:01 +0300, Ivan Rakov пишет: > > > > Nikita and Maxim, > > > > > > > > > What if we just update current metric getCurrentPmeDuration > behaviour > > > > > to show durations only for blocking PMEs? > > > > > Remain it as a long value and rename it to > getCacheOperationsBlockedDuration. > > > > > > > > > > No other changes will require. > > > > > > > > > > WDYT? > > > > > > > > I agree with these two metrics. I also think that current > > > > getCurrentPmeDuration will become redundant. > > > > > > > > Anton, > > > > > > > > > It looks like we're trying to implement "extended debug" instead of > > > > > "monitoring". > > > > > It should not be interesting for real admin what phase of PME is in > > > > > progress and so on. > > > > > > > > PME is mission critical cluster process. I agree that there's a fine > > > > line between monitoring and debug here. However, it's not good to add > > > > monitoring capabilities only for scenario when everything is alright. > > > > If PME will really hang, *real admin* will be extremely interested > how > > > > to return cluster back to working state. Metrics about stages > completion > > > > time may really help here: e.g. if one specific node hasn't completed > > > > stage X while rest of the cluster has, it can be a signal that this > node > > > > should be killed. > > > > > > > > Of course, it's possible to build monitoring system that extract this > > > > information from logs, but: > > > > - It's more resource intensive as it requires parsing logs for all > the time > > > > - It's less reliable as log messages may change > > > > > > > > Best Regards, > > > > Ivan Rakov > > > > > > > > On 24.07.2019 14:57, Maxim Muzafarov wrote: > > > > > Folks, > > > > > > > > > > +1 with Anton post. > > > > > > > > > > What if we just update current metric getCurrentPmeDuration > behaviour > > > > > to show durations only for blocking PMEs? > > > > > Remain it as a long value and rename it to > getCacheOperationsBlockedDuration. > > > > > > > > > > No other changes will require. > > > > > > > > > > WDYT? > > > > > > > > > > On Wed, 24 Jul 2019 at 14:02, Nikita Amelchev < > nsamelc...@gmail.com> wrote: > > > > > > Nikolay, > > > > > > > > > > > > The сacheOperationsBlockedDuration metric will show current > blocking > > > > > > duration or 0 if there is no blocking right now. > > > > > > > > > > > > The totalCacheOperationsBlockedDuration metric will accumulate > all > > > > > > blocking durations that happen after node starts. > > > > > > > > > > > > ср, 24 июл. 2019 г. в 13:35, Nikolay Izhikov < > nizhi...@apache.org>: > > > > > > > Nikita > > > > > > > > > > > > > > What is the difference between those two metrics? > > > > > > > > > > > > > > ср, 24 июля 2019 г., 12:45 Nikita Amelchev < > nsamelc...@gmail.com>: > > > > > > > > > > > > > > > Igniters, thanks for comments. > > > > > > > > > > > > > > > > From the discussion it can be seen that we need only two > metrics for now: > > > > > > > > - сacheOperationsBlockedDuration (long) > > > > > > > > - totalCacheOperationsBlockedDuration (long) > > > > > > > > > > > > > > > > I will prepare PR at the nearest time. > > > > > > > > > > > > > > > > ср, 24 июл. 2019 г. в 09:11, Zhenya Stanilovsky > <arzamas...@mail.ru.invalid > > > > > > > > > : > > > > > > > > > > > > > > > > > > +1 with Anton decisions. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Среда, 24 июля 2019, 8:44 +03:00 от Anton Vinogradov < > a...@apache.org>: > > > > > > > > > > > > > > > > > > > > Folks, > > > > > > > > > > > > > > > > > > > > It looks like we're trying to implement "extended debug" > instead of > > > > > > > > > > "monitoring". > > > > > > > > > > It should not be interesting for real admin what phase > of PME is in > > > > > > > > > > progress and so on. > > > > > > > > > > Interested metrics are > > > > > > > > > > - total blocked time (will be used for real SLA counting) > > > > > > > > > > - are we blocked right now (shows we have an SLA > degradation right now) > > > > > > > > > > Duration of the current blocking period can be easily > presented using > > > > > > > > > > > > > > > > any > > > > > > > > > > modern monitoring tool by regular checks. > > > > > > > > > > Initial true will means "period start", precision will > be a result of > > > > > > > > > > checks frequency. > > > > > > > > > > Anyway, I'm ok to have current metric presented with > long, where long > > > > > > > > > > > > > > > > is a > > > > > > > > > > duration, see no reason, but ok :) > > > > > > > > > > > > > > > > > > > > All other features you mentioned are useful for code or > > > > > > > > > > deployment improving and can (should) be taken from logs > at the analysis > > > > > > > > > > phase. > > > > > > > > > > > > > > > > > > > > On Tue, Jul 23, 2019 at 7:22 PM Ivan Rakov < > ivan.glu...@gmail.com > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > Folks, let me step in. > > > > > > > > > > > > > > > > > > > > > > Nikita, thanks for your suggestions! > > > > > > > > > > > > > > > > > > > > > > > 1. initialVersion. Topology version that initiates > the exchange. > > > > > > > > > > > > 2. initTime. Time PME was started. > > > > > > > > > > > > 3. initEvent. Event that triggered PME. > > > > > > > > > > > > 4. partitionReleaseTime. Time when a node has > finished waiting for > > > > > > > > > > > > > > > > all > > > > > > > > > > > > updates and translations on a previous topology. > > > > > > > > > > > > 5. sendSingleMessageTime. Time when a node sent a > single message. > > > > > > > > > > > > 6. recieveFullMessageTime. Time when a node received > a full message. > > > > > > > > > > > > 7. finishTime. Time PME was ended. > > > > > > > > > > > > > > > > > > > > > > > > When new PME started all these metrics resets. > > > > > > > > > > > > > > > > > > > > > > Every metric from Nikita's list looks useful and > simple to implement. > > > > > > > > > > > I think that it would be better to change format of > metrics 4, 5, 6 > > > > > > > > > > > > > > > > and > > > > > > > > > > > 7 a bit: we can keep only difference between time of > previous event > > > > > > > > > > > > > > > > and > > > > > > > > > > > time of corresponding event. Such metrics would be > easier to perceive: > > > > > > > > > > > they answer to specific questions "how much time did > partition release > > > > > > > > > > > take?" or "how much time did awaiting of distributed > phase end take?". > > > > > > > > > > > Also, if results of 4, 5, 6, 7 will be exported to > monitoring system, > > > > > > > > > > > graphs will show how different stages times change > from one PME to > > > > > > > > > > > > > > > > another. > > > > > > > > > > > > When PME cause no blocking, it's a good PME and I > see no reason to > > > > > > > > > > > > > > > > have > > > > > > > > > > > > monitoring related to it > > > > > > > > > > > > > > > > > > > > > > Agree with Anton here. These metrics should be > measured only for true > > > > > > > > > > > distributed exchange. Saving results for client > leave/join PMEs will > > > > > > > > > > > just complicate monitoring. > > > > > > > > > > > > > > > > > > > > > > > I agree with total blocking duration metric but > > > > > > > > > > > > I still don't understand why instant value > indicating that > > > > > > > > > > > > > > > > operations are > > > > > > > > > > > > blocked should be boolean. > > > > > > > > > > > > Duration time since blocking has started looks more > appropriate and > > > > > > > > > > > > > > > > > > > > > > useful. > > > > > > > > > > > > It gives more information while semantic is left the > same. > > > > > > > > > > > > > > > > > > > > > > Totally agree with Pavel here. Both "accumulated block > time" and > > > > > > > > > > > "current PME block time" metrics are useful. Growth of > accumulated > > > > > > > > > > > metric for specific period of time (should be easy to > check via > > > > > > > > > > > monitoring system graph) will show for how much > business operations > > > > > > > > > > > > > > > > were > > > > > > > > > > > blocked in total, and non-zero current metric will > show that we are > > > > > > > > > > > experiencing issues right now. Boolean metric "are we > blocked right > > > > > > > > > > > > > > > > now" > > > > > > > > > > > is not needed as it's obviously can be inferred from > "current PME > > > > > > > > > > > > > > > > block > > > > > > > > > > > time". > > > > > > > > > > > > > > > > > > > > > > Best Regards, > > > > > > > > > > > Ivan Rakov > > > > > > > > > > > > > > > > > > > > > > On 23.07.2019 16:02, Pavel Kovalenko wrote: > > > > > > > > > > > > Nikita, > > > > > > > > > > > > > > > > > > > > > > > > I agree with total blocking duration metric but > > > > > > > > > > > > I still don't understand why instant value > indicating that > > > > > > > > > > > > > > > > operations are > > > > > > > > > > > > blocked should be boolean. > > > > > > > > > > > > Duration time since blocking has started looks more > appropriate and > > > > > > > > > > > > > > > > > > > > > > useful. > > > > > > > > > > > > It gives more information while semantic is left the > same. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 23 июл. 2019 г. в 11:42, Nikita Amelchev < > nsamelc...@gmail.com > > > > > > > > > > > > > > > > > > : > > > > > > > > > > > > > Folks, > > > > > > > > > > > > > > > > > > > > > > > > > > All previous suggestions have some disadvantages. > It can be several > > > > > > > > > > > > > exchanges between two metric updates and fast > exchange can rewrite > > > > > > > > > > > > > previous long exchange. > > > > > > > > > > > > > > > > > > > > > > > > > > We can introduce a metric of total blocking > duration that will > > > > > > > > > > > > > accumulate at the end of the exchange. So, users > will get actual > > > > > > > > > > > > > information about how long operations were > blocked. Cluster metric > > > > > > > > > > > > > will be a maximum of local nodes metrics. And we > need a boolean > > > > > > > > > > > > > > > > metric > > > > > > > > > > > > > that will indicate realtime status. It needs > because of duration > > > > > > > > > > > > > metric updates at the end of the exchange. > > > > > > > > > > > > > > > > > > > > > > > > > > So I propose to change the current metric that not > released to the > > > > > > > > > > > > > totalCacheOperationsBlockingDuration metric and to > add the > > > > > > > > > > > > > isCacheOperationsBlocked metric. > > > > > > > > > > > > > > > > > > > > > > > > > > WDYT? > > > > > > > > > > > > > > > > > > > > > > > > > > пн, 22 июл. 2019 г. в 09:27, Anton Vinogradov < > a...@apache.org >: > > > > > > > > > > > > > > Nikolay, > > > > > > > > > > > > > > > > > > > > > > > > > > > > Still see no reason to replace boolean with long. > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Jul 22, 2019 at 9:19 AM Nikolay Izhikov < > > > > > > > > > > > > > > > > nizhi...@apache.org > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Anton. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Value exported based on SPI settings, not > in the moment it > > > > > > > > > > > > > > > > changed. > > > > > > > > > > > > > > > 2. Clock synchronisation - if we export start > time, we should > > > > > > > > > > > > > > > > also > > > > > > > > > > > > > export > > > > > > > > > > > > > > > node local timestamp. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пн, 22 июля 2019 г., 8:33 Anton Vinogradov < > a...@apache.org >: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Folks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > What's the reason for duration counting? > > > > > > > > > > > > > > > > AFAIU, it's a monitoring system feature to > count the durations. > > > > > > > > > > > > > > > > Sine monitoring system checks metrics > periodically it will know > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > duration by its own log. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 19, 2019 at 7:32 PM Pavel > Kovalenko < > > > > > > > > > > > > > > > > jokse...@gmail.com > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Nikita, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yes, I mean duration not timestamp. For > the metric name, I > > > > > > > > > > > > > > > > suggest > > > > > > > > > > > > > > > > > "cacheOperationsBlockingDuration", I think > it cleaner > > > > > > > > > > > > > > > > represents > > > > > > > > > > > > > what > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > blocked during PME. > > > > > > > > > > > > > > > > > We can also combine both timestamp > > > > > > > > > > > > > > > > > > > > > > > > > > "cacheOperationsBlockingStartTs" and > > > > > > > > > > > > > > > > > duration to have better correlation when > cache operations were > > > > > > > > > > > > > > > > > > > > > > > > > > blocked > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > how much time it's taken. > > > > > > > > > > > > > > > > > For instant view (like in JMX bean) a > calculated value as you > > > > > > > > > > > > > > > > > > > > > > > > > > mentioned > > > > > > > > > > > > > > > > > can be used. > > > > > > > > > > > > > > > > > For metrics are exported to some backend > (IEP-35) a counter > > > > > > > > > > > > > > > > can be > > > > > > > > > > > > > > > used. > > > > > > > > > > > > > > > > > The counter is incremented by blocking > time after blocking has > > > > > > > > > > > > > > > > > > > > > > > > > > ended. > > > > > > > > > > > > > > > > > пт, 19 июл. 2019 г. в 19:10, Nikita > Amelchev < > > > > > > > > > > > > > > > > nsamelc...@gmail.com > > > > > > > > > > > > > > : > > > > > > > > > > > > > > > > > > Pavel, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The main purpose of this metric is > > > > > > > > > > > > > > > > > > > > how much time we wait for resuming > cache operations > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Seems I misunderstood you. Do you mean > timestamp or duration > > > > > > > > > > > > > > > > here? > > > > > > > > > > > > > > > > > > > > What do you think if we change the > boolean value of metric > > > > > > > > > > > > > > > > to a > > > > > > > > > > > > > > > long > > > > > > > > > > > > > > > > > > value that represents time in > milliseconds when operations > > > > > > > > > > > > > > > > were > > > > > > > > > > > > > > > blocked? > > > > > > > > > > > > > > > > > > This time can be calculated as > (currentTime - > > > > > > > > > > > > > > > > > > timeSinceOperationsBlocked) in case of > timestamp. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Duration will be more understandable. > It'll be something like > > > > > > > > > > > > > > > > > > getCurrentBlockingPmeDuration. But I > haven't come up with a > > > > > > > > > > > > > > > > better > > > > > > > > > > > > > > > > > > name yet. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пт, 19 июл. 2019 г. в 18:30, Pavel > Kovalenko < > > > > > > > > > > > > > > > > jokse...@gmail.com > > > > > > > > > > > > > > : > > > > > > > > > > > > > > > > > > > Nikita, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think getCurrentPmeDuration doesn't > show useful > > > > > > > > > > > > > > > > information. > > > > > > > > > > > > > The > > > > > > > > > > > > > > > > main > > > > > > > > > > > > > > > > > > PME side effect for end-users is > blocking cache operations. > > > > > > > > > > > > > > > > Not > > > > > > > > > > > > > all > > > > > > > > > > > > > > > PME > > > > > > > > > > > > > > > > > > time blocks it. > > > > > > > > > > > > > > > > > > > What information gives to an end-user > timestamp of > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > "timeSinceOperationsBlocked"? For what > analysis it can be > > > > > > > > > > > > > > > > used and > > > > > > > > > > > > > > > how? > > > > > > > > > > > > > > > > > > > пт, 19 июл. 2019 г. в 17:48, Nikita > Amelchev < > > > > > > > > > > > > > > > > > > > > > > > > > > nsamelc...@gmail.com > > > > > > > > > > > > > > > > : > > > > > > > > > > > > > > > > > > > > Hi Pavel, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This time already can be obtained > from the > > > > > > > > > > > > > > > > > > > > > > > > > > getCurrentPmeDuration > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > new isOperationsBlockedByPme metrics. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > As an alternative solution, I can > rework recently added > > > > > > > > > > > > > > > > > > > > getCurrentPmeDuration metric (not > released yet). Seems for > > > > > > > > > > > > > > > > > > > > > > > > > > users it > > > > > > > > > > > > > > > > > > > > useless in case of non-blocking PME. > > > > > > > > > > > > > > > > > > > > Lets name it > timeSinceOperationsBlocked. It'll be timestamp > > > > > > > > > > > > > > > > > > > > > > > > > > when > > > > > > > > > > > > > > > > > > > > blocking started (minimal value of > cluster nodes) and 0 if > > > > > > > > > > > > > > > > > > > > > > > > > > blocking > > > > > > > > > > > > > > > > > > > > ends (there is no running PME). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > WDYT? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > пт, 19 июл. 2019 г. в 15:56, Pavel > Kovalenko < > > > > > > > > > > > > > > > > > > > > > > > > > > jokse...@gmail.com >: > > > > > > > > > > > > > > > > > > > > > Hi Nikita, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you for working on this. > What do you think if we > > > > > > > > > > > > > > > > > > > > > > > > > > change the > > > > > > > > > > > > > > > > > > boolean > > > > > > > > > > > > > > > > > > > > > value of metric to a long value > that represents time in > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > milliseconds > > > > > > > > > > > > > > > > > > when > > > > > > > > > > > > > > > > > > > > > operations were blocked? > > > > > > > > > > > > > > > > > > > > > Since we have not only JMX and now > metrics are periodically > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > exported > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > some backend it can give a more > clear picture of how much > > > > > > > > > > > > > > > > > > > > > > > > > > time we > > > > > > > > > > > > > > > > > > wait for > > > > > > > > > > > > > > > > > > > > > resuming cache operations instead > of instant boolean > > > > > > > > > > > > > > > > > > > > > > > > > > indicator. > > > > > > > > > > > > > > > > > > > > > пт, 19 июл. 2019 г. в 14:41, > Nikita Amelchev < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > nsamelc...@gmail.com > > > > > > > > > > > > > > > > > : > > > > > > > > > > > > > > > > > > > > > > Anton, Nikolay, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the support. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > For now, we have the > getCurrentPmeDuration() metric that > > > > > > > > > > > > > > > > > > > > > > > > > > does > > > > > > > > > > > > > > > not > > > > > > > > > > > > > > > > > > show > > > > > > > > > > > > > > > > > > > > > > influence on the cluster > correctly. PME can be without > > > > > > > > > > > > > > > > > > > > > > > > > > blocking > > > > > > > > > > > > > > > > > > > > > > operations. For example, client > node join/leave events. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I suggest add new metric - > isOperationsBlockedByPme(). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Together, > > > > > > > > > > > > > > > > > > these > > > > > > > > > > > > > > > > > > > > > > metrics will show influence of > the PME on cluster and user > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > operations. > > > > > > > > > > > > > > > > > > > > > > I have prepared PR for this (Bot > visa is green). [1] Can > > > > > > > > > > > > > > > > > > > > > > > > > > anyone > > > > > > > > > > > > > > > > > > take a > > > > > > > > > > > > > > > > > > > > > > look? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > https://issues.apache.org/jira/browse/IGNITE-11961 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 16 июл. 2019 г. в 14:58, > Nikolay Izhikov < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > nizhi...@apache.org > > > > > > > > > > > > > > > > > > > : > > > > > > > > > > > > > > > > > > > > > > > I think administator of Ignite > cluster should be able to > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > monitor > > > > > > > > > > > > > > > > > > all > > > > > > > > > > > > > > > > > > > > > > Ignite process, including non > blocking PME. > > > > > > > > > > > > > > > > > > > > > > > В Вт, 16/07/2019 в 14:57 > +0300, Anton Vinogradov пишет: > > > > > > > > > > > > > > > > > > > > > > > > BTW, > > > > > > > > > > > > > > > > > > > > > > > > Found PME metric - > getCurrentPmeDuration(). > > > > > > > > > > > > > > > > > > > > > > > > Seems, it shows exactly PME > time and not so useful > > > > > > > > > > > > > > > > > > > > > > > > > > because > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > this. > > > > > > > > > > > > > > > > > > > > > > > > The goal it so show exactly > blocking period. > > > > > > > > > > > > > > > > > > > > > > > > When PME cause no blocking, > it's a good PME and I see > > > > > > > > > > > > > > > > > > > > > > > > > > no > > > > > > > > > > > > > > > > > > reason to have > > > > > > > > > > > > > > > > > > > > > > > > monitoring related to it :) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jul 16, 2019 at 2:50 > PM Nikolay Izhikov < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > nizhi...@apache.org > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Anton. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Why do we need to postpone > implementation of this > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > metrics? > > > > > > > > > > > > > > > > > > > > > > > > > For now, implementation of > new metric is very simple. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think we can implement > this metrics as a single > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > contribution. > > > > > > > > > > > > > > > > > > > > > > > > > В Вт, 16/07/2019 в 13:47 > +0300, Anton Vinogradov > > > > > > > > > > > > > > > > > > > > > > > > > > пишет: > > > > > > > > > > > > > > > > > > > > > > > > > > Nikita, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Looks like all we need > now is a 1 simple metric: > > > > > > > > > > > > > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > > operations > > > > > > > > > > > > > > > > > > > > > > blocked? > > > > > > > > > > > > > > > > > > > > > > > > > > Just a true or false. > > > > > > > > > > > > > > > > > > > > > > > > > > Lest start from this. > > > > > > > > > > > > > > > > > > > > > > > > > > All other metrics can be > extracted from logs now > > > > > > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > > > implemented > > > > > > > > > > > > > > > > > > > > > > > > > > later. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jul 16, 2019 at > 12:46 PM Nikolay Izhikov < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > nizhi...@apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > +1. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Nikita, please, go > ahead. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > вт, 16 июля 2019 г., > 11:45 Nikita Amelchev < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > nsamelc...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > : > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hello, Igniters. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I suggest to add > some useful metrics about the > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > partition map > > > > > > > > > > > > > > > > > > > > > > exchange > > > > > > > > > > > > > > > > > > > > > > > > > > > > (PME). For now, the > duration of PME stages > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > available > > > > > > > > > > > > > > > > > > only in > > > > > > > > > > > > > > > > > > > > > > log > > > > > > > > > > > > > > > > > > > > > > > > > files > > > > > > > > > > > > > > > > > > > > > > > > > > > > and cannot be > obtained using JMX or other > > > > > > > > > > > > > > > > > > > > > > > > > > external > > > > > > > > > > > > > > > > > > tools. [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > I made the list of > local node metrics that > > > > > > > > > > > > > > > > > > > > > > > > > > help to > > > > > > > > > > > > > > > > > > understand > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > > > actual status of > current PME: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. initialVersion. > Topology version that > > > > > > > > > > > > > > > > > > > > > > > > > > initiates > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > exchange. > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. initTime. Time > PME was started. > > > > > > > > > > > > > > > > > > > > > > > > > > > > 3. initEvent. Event > that triggered PME. > > > > > > > > > > > > > > > > > > > > > > > > > > > > 4. > partitionReleaseTime. Time when a node has > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > finished > > > > > > > > > > > > > > > > > > waiting > > > > > > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > > > > > > all > > > > > > > > > > > > > > > > > > > > > > > > > > > > updates and > translations on a previous > > > > > > > > > > > > > > > > > > > > > > > > > > topology. > > > > > > > > > > > > > > > > > > > > > > > > > > > > 5. > sendSingleMessageTime. Time when a node > > > > > > > > > > > > > > > > > > > > > > > > > > sent a > > > > > > > > > > > > > > > > > > single > > > > > > > > > > > > > > > > > > > > > > message. > > > > > > > > > > > > > > > > > > > > > > > > > > > > 6. > recieveFullMessageTime. Time when a node > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > received > > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > full > > > > > > > > > > > > > > > > > > > > > > message. > > > > > > > > > > > > > > > > > > > > > > > > > > > > 7. finishTime. Time > PME was ended. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > When new PME started > all these metrics resets. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > These metrics help > to understand: > > > > > > > > > > > > > > > > > > > > > > > > > > > > - how long PME was > (current or previous). > > > > > > > > > > > > > > > > > > > > > > > > > > > > - how long awaited > for all updates was > > > > > > > > > > > > > > > > > > > > > > > > > > completed. > > > > > > > > > > > > > > > > > > > > > > > > > > > > - what node blocks > PME (didn't send a single > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > message) > > > > > > > > > > > > > > > > > > > > > > > > > > > > - what triggered PME. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thoughts? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-11961 > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best wishes, > > > > > > > > > > > > > > > > > > > > > > > > > > > > Amelchev Nikita > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > Best wishes, > > > > > > > > > > > > > > > > > > > > > > Amelchev Nikita > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > Best wishes, > > > > > > > > > > > > > > > > > > > > Amelchev Nikita > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > Best wishes, > > > > > > > > > > > > > > > > > > Amelchev Nikita > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > Best wishes, > > > > > > > > > > > > > Amelchev Nikita > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Zhenya Stanilovsky > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Best wishes, > > > > > > > > Amelchev Nikita > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Best wishes, > > > > > > Amelchev Nikita > > > > > > >
