[jira] [Created] (IGNITE-11941) [IEP-35] Rewrite GridLocalMetrics using new framework
Nikolay Izhikov created IGNITE-11941: Summary: [IEP-35] Rewrite GridLocalMetrics using new framework Key: IGNITE-11941 URL: https://issues.apache.org/jira/browse/IGNITE-11941 Project: Ignite Issue Type: Improvement Reporter: Nikolay Izhikov 1. GridLocalMetrics should be moved to GridMetricManager 2. Standard JVM JMX beans should be registered as metrics on startup. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [IEP-35] GridJobProcessorMetrics migration
Hello, Alex.
Based on our private discussion I've additionally migrated `totalExecutionTime`
and `totalWaitingTime` counters.
Can you review the PR [1]?
[1] https://github.com/apache/ignite/pull/6622
В Пн, 24/06/2019 в 15:14 +0300, Nikolay Izhikov пишет:
> Hello, Alex.
>
> Thanks for the answer.
>
> 1. I, actually, don't understand your proposal :)
> Can you write it down?
> What numbers should be additionally migrated in this PR?
> Or it's OK for now?
>
> > I think "idle time" is a useful metric
>
> I think "usefulness" or "uselessness" of specific metrics depends on the
> questions we can answer with it.
> What questions we can ask about Ignite instance and answer with "idle time"
> metric?
>
> > About execution and waiting time , it's not the right way to calculate it
> > using a jobs list.
>
> Same question here.
>
> What questions we can answer with current numbers?
>
> > Will jobs list contain only active jobs?
>
> All jobs that are scheduled for execution on the node(active + waiting)
> should be in the list.
> I try to put more details here, to expose my way of thinking about metrics
> and lists:
>
> If you have some issues with the jobs on the node it can be 2 kinds of
> issues:
> 1. You are waiting for the results of some job and want to know why it
> doesn't execute.
>
> In this case, you should query "jobs list" from Ignite.
> You can get an answer on:
> * What jobs currently executes?
> * How many time your job waiting to be executed?
>
> You can also check "activeJobs", "waitingJobs" metrics graphics
> to know changes in the jobs queue during the time.
> Seems, you can predict the start of your job from these
> numbers.
>
> 2. You want to understand the lifecycle of some finished(failed job).
>
> In this case, you should analyze the log of the node.
> It should contain information about time:
> * node recieve job information
> * job added to the queue
> * job started execution
> * job finished(failed) execution.
>
> I don't see questions we can't ask from these sources.
> Do we have such?
> How numbers from current GridJobMetrics can help with these questions?
>
>
> > But, what if a user doesn't use any
> > external monitoring system and wants to know the health of Ignite instance?
>
> It depends on how we define "health".
> And it's not trivial question :)
>
> > Do we have any plans to implement some simple aggregator and ship it with
> > Ignite?
>
> I think NO.
> We shouldn't do it.
>
> > Do we have plans to provide some presets for Ignite monitoring for
> > popular monitoring systems?
>
> I think we shouldn't do it.
> Because monitoring presets heavily depends on the usage scenario.
> And it can heavily vary for the Ignite.
>
>
> В Пн, 24/06/2019 в 12:46 +0300, Alex Plehanov пишет:
> > Hi Nikolay,
> >
> > I think "idle time" is a useful metric, but it can be calculated outside of
> > Ignite using external monitoring system.
> >
> > About execution and waiting time, it's not the right way to calculate it
> > using a jobs list. Will jobs list contain only active jobs? In this case,
> > you can't calculate these metrics at all, since you don't know the time of
> > finished jobs. If the list will contain all jobs (will it be unlimited?),
> > iterating over this list will be resource consuming. In any way, it's much
> > simpler (and sometimes only possible) for an external monitoring system to
> > just get some scalar metric than iterate over a list with some condition.
> >
> > About aggregation, yes, in an ideal world aggregation should be done with
> > the external monitoring system. But, what if a user doesn't use any
> > external monitoring system and wants to know the health of Ignite instance?
> > Do we have any plans to implement some simple aggregator and ship it with
> > Ignite? Do we have plans to provide some presets for Ignite monitoring for
> > popular monitoring systems? (These questions not related to this PR, but
> > related to IEP at all)
> >
> > Also, some aggregation metrics ("max" for example) can't be effectively
> > calculated using the external system (you should iterate over a jobs list
> > again and still precision of such calculation will be no more than the time
> > between probes).
signature.asc
Description: This is a digitally signed message part
Re: Apache Ignite contribute
Hello guys, My name is Kirill Tkalenko and I want to contribute. Need access to Ignite Jira. My username is ktkalenko. Thanks in advance. Best regards, Kirill On Mon, Jun 24, 2019 at 5:13 PM Kirill Tkalenko wrote: > Hello guys, > > My name is Kirill Tkalenko and I want to contribute. > > Best regards, > Kirill > > -- > Kirill Tkalenko > Software Engineer, Moscow > +79099793376 > https://www.gridgain.com > Powered by Apache® Ignite™ > -- Kirill Tkalenko Software Engineer, Moscow +79099793376 https://www.gridgain.com Powered by Apache® Ignite™
Apache Ignite contribute
Hello guys, My name is Kirill Tkalenko and I want to contribute. Best regards, Kirill -- Kirill Tkalenko Software Engineer, Moscow +79099793376 https://www.gridgain.com Powered by Apache® Ignite™
Re: [IEP-35] GridJobProcessorMetrics migration
Hello, Ivan.
> Ignite is a cluster which almost every
> time assumes an external monitoring for a production use.
+1.
> 1. Are we going to preserve a compatibility with metrics present
> before? Or are we going to keep only those making sense today?
1. Backward compatibility preserved.
2. Deprecated metrics(and metric APIs) will be removed in Ignite 3.
3. We should make a decision what numbers are "make sense" and what don't.
> 2. Can we configure which supported metrics are calculated/exposed? Or
> do we calculate/expose everything every time?
1. You can configure filter for the exposed metrics. Only required subset of
the metric will be exported.
2. For now, all metrics(not lists!) will be calculated. Please, note, that
every metrics is the simple long(double) counter.
В Пн, 24/06/2019 в 14:43 +0300, Павлухин Иван пишет:
> Hi Nikolay, Alex,
>
> A couple of my humble comments
> > Aggregation should be done with the metric collect system(Prometheus,
> > Graphite, etc.).
>
> I like that statement very much!
>
> > But, what if a user doesn't use any external monitoring system and wants to
> > know the health of Ignite instance?
>
> I think that we can add more capabilities if a real user demand
> appears in future. Generally, Ignite is a cluster which almost every
> time assumes an external monitoring for a production use.
>
> And a couple of general questions regarding monitoring. If they are
> answered in IEP you can simply redirect me there.
> 1. Are we going to preserve a compatibility with metrics present
> before? Or are we going to keep only those making sense today?
> 2. Can we configure which supported metrics are calculated/exposed? Or
> do we calculate/expose everything every time?
>
> пн, 24 июн. 2019 г. в 12:46, Alex Plehanov :
> >
> > Hi Nikolay,
> >
> > I think "idle time" is a useful metric, but it can be calculated outside of
> > Ignite using external monitoring system.
> >
> > About execution and waiting time, it's not the right way to calculate it
> > using a jobs list. Will jobs list contain only active jobs? In this case,
> > you can't calculate these metrics at all, since you don't know the time of
> > finished jobs. If the list will contain all jobs (will it be unlimited?),
> > iterating over this list will be resource consuming. In any way, it's much
> > simpler (and sometimes only possible) for an external monitoring system to
> > just get some scalar metric than iterate over a list with some condition.
> >
> > About aggregation, yes, in an ideal world aggregation should be done with
> > the external monitoring system. But, what if a user doesn't use any
> > external monitoring system and wants to know the health of Ignite instance?
> > Do we have any plans to implement some simple aggregator and ship it with
> > Ignite? Do we have plans to provide some presets for Ignite monitoring for
> > popular monitoring systems? (These questions not related to this PR, but
> > related to IEP at all)
> >
> > Also, some aggregation metrics ("max" for example) can't be effectively
> > calculated using the external system (you should iterate over a jobs list
> > again and still precision of such calculation will be no more than the time
> > between probes).
>
>
>
signature.asc
Description: This is a digitally signed message part
Re: [IEP-35] GridJobProcessorMetrics migration
Hello, Alex.
Thanks for the answer.
1. I, actually, don't understand your proposal :)
Can you write it down?
What numbers should be additionally migrated in this PR?
Or it's OK for now?
> I think "idle time" is a useful metric
I think "usefulness" or "uselessness" of specific metrics depends on the
questions we can answer with it.
What questions we can ask about Ignite instance and answer with "idle time"
metric?
> About execution and waiting time , it's not the right way to calculate it
> using a jobs list.
Same question here.
What questions we can answer with current numbers?
> Will jobs list contain only active jobs?
All jobs that are scheduled for execution on the node(active + waiting) should
be in the list.
I try to put more details here, to expose my way of thinking about metrics and
lists:
If you have some issues with the jobs on the node it can be 2 kinds of issues:
1. You are waiting for the results of some job and want to know why it
doesn't execute.
In this case, you should query "jobs list" from Ignite.
You can get an answer on:
* What jobs currently executes?
* How many time your job waiting to be executed?
You can also check "activeJobs", "waitingJobs" metrics graphics
to know changes in the jobs queue during the time.
Seems, you can predict the start of your job from these
numbers.
2. You want to understand the lifecycle of some finished(failed job).
In this case, you should analyze the log of the node.
It should contain information about time:
* node recieve job information
* job added to the queue
* job started execution
* job finished(failed) execution.
I don't see questions we can't ask from these sources.
Do we have such?
How numbers from current GridJobMetrics can help with these questions?
> But, what if a user doesn't use any
> external monitoring system and wants to know the health of Ignite instance?
It depends on how we define "health".
And it's not trivial question :)
> Do we have any plans to implement some simple aggregator and ship it with
> Ignite?
I think NO.
We shouldn't do it.
> Do we have plans to provide some presets for Ignite monitoring for
> popular monitoring systems?
I think we shouldn't do it.
Because monitoring presets heavily depends on the usage scenario.
And it can heavily vary for the Ignite.
В Пн, 24/06/2019 в 12:46 +0300, Alex Plehanov пишет:
> Hi Nikolay,
>
> I think "idle time" is a useful metric, but it can be calculated outside of
> Ignite using external monitoring system.
>
> About execution and waiting time, it's not the right way to calculate it
> using a jobs list. Will jobs list contain only active jobs? In this case,
> you can't calculate these metrics at all, since you don't know the time of
> finished jobs. If the list will contain all jobs (will it be unlimited?),
> iterating over this list will be resource consuming. In any way, it's much
> simpler (and sometimes only possible) for an external monitoring system to
> just get some scalar metric than iterate over a list with some condition.
>
> About aggregation, yes, in an ideal world aggregation should be done with
> the external monitoring system. But, what if a user doesn't use any
> external monitoring system and wants to know the health of Ignite instance?
> Do we have any plans to implement some simple aggregator and ship it with
> Ignite? Do we have plans to provide some presets for Ignite monitoring for
> popular monitoring systems? (These questions not related to this PR, but
> related to IEP at all)
>
> Also, some aggregation metrics ("max" for example) can't be effectively
> calculated using the external system (you should iterate over a jobs list
> again and still precision of such calculation will be no more than the time
> between probes).
signature.asc
Description: This is a digitally signed message part
Re: [IEP-35] GridJobProcessorMetrics migration
Hi Nikolay, Alex,
A couple of my humble comments
> Aggregation should be done with the metric collect system(Prometheus,
> Graphite, etc.).
I like that statement very much!
> But, what if a user doesn't use any external monitoring system and wants to
> know the health of Ignite instance?
I think that we can add more capabilities if a real user demand
appears in future. Generally, Ignite is a cluster which almost every
time assumes an external monitoring for a production use.
And a couple of general questions regarding monitoring. If they are
answered in IEP you can simply redirect me there.
1. Are we going to preserve a compatibility with metrics present
before? Or are we going to keep only those making sense today?
2. Can we configure which supported metrics are calculated/exposed? Or
do we calculate/expose everything every time?
пн, 24 июн. 2019 г. в 12:46, Alex Plehanov :
>
> Hi Nikolay,
>
> I think "idle time" is a useful metric, but it can be calculated outside of
> Ignite using external monitoring system.
>
> About execution and waiting time, it's not the right way to calculate it
> using a jobs list. Will jobs list contain only active jobs? In this case,
> you can't calculate these metrics at all, since you don't know the time of
> finished jobs. If the list will contain all jobs (will it be unlimited?),
> iterating over this list will be resource consuming. In any way, it's much
> simpler (and sometimes only possible) for an external monitoring system to
> just get some scalar metric than iterate over a list with some condition.
>
> About aggregation, yes, in an ideal world aggregation should be done with
> the external monitoring system. But, what if a user doesn't use any
> external monitoring system and wants to know the health of Ignite instance?
> Do we have any plans to implement some simple aggregator and ship it with
> Ignite? Do we have plans to provide some presets for Ignite monitoring for
> popular monitoring systems? (These questions not related to this PR, but
> related to IEP at all)
>
> Also, some aggregation metrics ("max" for example) can't be effectively
> calculated using the external system (you should iterate over a jobs list
> again and still precision of such calculation will be no more than the time
> between probes).
--
Best regards,
Ivan Pavlukhin
Re: [RESULT] [VOTE] Release Apache Ignite 2.7.5-rc4
Hi Denis, I see docs now have the correct version. Open issues I list at the end of the document about the release https://cwiki.apache.org/confluence/display/IGNITE/Release+Process#ReleaseProcess-OpenIssues Now issues 3, 5, 6 need to be addressed. Sincerely, Dmitriy Pavlov чт, 20 июн. 2019 г. в 16:24, Dmitriy Pavlov : > Hi Denis, > Artem explained to me what to do, but I need permissions for all > supplementary docs to switch version. > > Could you please grant it for > C#/.NET > C++ > SQL > Integrations* > Ignite for Spark > Tools > ? > > вт, 11 июн. 2019 г. в 23:21, Dmitriy Pavlov : > >> Thank you, Mauricio, >> >> I'm going to apply this patch on Monday. I will be traveling with limited >> access to the Internet. >> >> I'll also add more explanation on why we should update these tags to the >> wiki. >> >> вт, 11 июн. 2019 г. в 20:57, Mauricio Stekl : >> >>> Hi Dmitriy, >>> Yes, that would be enough to set the URL for latest docs. >>> >>> Additionally I usually do some SEO updates to new/old .html files, like >>> adding NOINDEX metatag to older version; add canonical URL to latest docs; >>> and add GA code. >>> >>> I am attaching a svn patch which includes all these updates, including >>> the .htaccess one. >>> >>> >>> Thanks. >>> Mauricio Stekl >>> >>> >>> >>> >>> >>> On Tue, Jun 11, 2019 at 1:38 PM Dmitriy Pavlov >>> wrote: >>> Ok, Denis, thank you. Garrett, could you please advice me on how to do a copy of 2.7.0 docs to 2.7.5? I have an account on the readme.io, so I would like to do it and describe in the Release process https://cwiki.apache.org/confluence/display/IGNITE/Release+Process Mauricio, is my understanding correct? in the htaccess I can replace RewriteRule ^releases/latest/(.*)$ /releases/2.7.0/$1 [L] to RewriteRule ^releases/latest/(.*)$ /releases/2.7.5/$1 [L] and that's it? Sincerely Dmitriy Pavlov вт, 11 июн. 2019 г. в 19:26, Denis Magda : > Igniters, > > We have to create version 2.7.5 for readme.io even if the changes are > minimal. That's a general practice. Copying Garrett and Artem who can > facilitate on the documentation end. > > Also, feel free to reach out Mauricio (copied) who can help to run the > scripts that will make JavaDocs 2.7.5 latest for Ignite website. > > - > Denis > > > On Tue, Jun 11, 2019 at 6:00 AM Dmitriy Pavlov wrote: > > > Sorry for the second email. I want to clarify: I've already uploaded > > Javadoc,scaladoc,c++ docs (generated). Question is related only to > > readme.io. > > We have links named as 2.7 there, so both 2.7.5 & 2.7.0 seems to be > > applicable. > > > > There were no changes in the public API/parameters. > > > > вт, 11 июн. 2019 г. в 15:44, Nikolay Izhikov : > > > > > Hello, Dmitriy. > > > > > > I think we should write down diff between 2.7.0 and 2.7.5 on some page > in > > > 2.7 documentation. > > > Also, we should make this page very noticable. > > > > > > В Вт, 11/06/2019 в 15:22 +0300, Dmitriy Pavlov пишет: > > > > Hi Igniters, PMCs, > > > > > > > > could you please advice me about step > > > > > > > >1. Release all the documentation (Java, .NET, C++, etc.) on > > > >apacheignite.readme.io. > > > > > > > > from the release process. > > > > > > > > Do we have some documentation to publish for 2.7.5? Since it is minor > > > > release I propose to keep documentation for 2.7.0. > > > > > > > > Sincerely, > > > > Dmitriy Pavlov > > > > > > > > пн, 10 июн. 2019 г. в 14:30, Dmitriy Pavlov >>> >: > > > > > > > > > The vote for a new release candidate is closed, now > > > > > > > > > > > > > > > > > > > > Vote result: Vote passes with 9 votes +1 (4 binding +1 votes), no 0 > > > and no > > > > > -1. > > > > > > > > > > > > > > > > > > > > +1 votes: > > > > > > > > > > - Ilya Kasnacheev > > > > > > > > > > - Nikolay Izhikov (binding) > > > > > > > > > > - Denis Magda (binding) > > > > > > > > > > - Andrey Gura (binding) > > > > > > > > > > - Alexey Goncharuk (binding) > > > > > > > > > > - Igor Sapego > > > > > > > > > > - Ivan Pavlukhin > > > > > > > > > > - Yuriy Babak > > > > > > > > > > - Vyacheslav Daradur > > > > > > > > > > > > > > > > > > > > Vote thread > > > > > > > > > > > https://lists.apache.org/thread.html/35cbc2d4c5b769155dc8aec15edd808a25c5cf48a5e12637528e931d@%3Cdev.ignite.apache.org%3E > > > > > > > > > > > > > > > > >>>
Re: [IEP-35] GridJobProcessorMetrics migration
Hi Nikolay,
I think "idle time" is a useful metric, but it can be calculated outside of
Ignite using external monitoring system.
About execution and waiting time, it's not the right way to calculate it
using a jobs list. Will jobs list contain only active jobs? In this case,
you can't calculate these metrics at all, since you don't know the time of
finished jobs. If the list will contain all jobs (will it be unlimited?),
iterating over this list will be resource consuming. In any way, it's much
simpler (and sometimes only possible) for an external monitoring system to
just get some scalar metric than iterate over a list with some condition.
About aggregation, yes, in an ideal world aggregation should be done with
the external monitoring system. But, what if a user doesn't use any
external monitoring system and wants to know the health of Ignite instance?
Do we have any plans to implement some simple aggregator and ship it with
Ignite? Do we have plans to provide some presets for Ignite monitoring for
popular monitoring systems? (These questions not related to this PR, but
related to IEP at all)
Also, some aggregation metrics ("max" for example) can't be effectively
calculated using the external system (you should iterate over a jobs list
again and still precision of such calculation will be no more than the time
between probes).
[jira] [Created] (IGNITE-11940) MVCC SysPropWalDeltaConsistencyTest.testPutRemoveMultinode fails frequently
Ivan Pavlukhin created IGNITE-11940: --- Summary: MVCC SysPropWalDeltaConsistencyTest.testPutRemoveMultinode fails frequently Key: IGNITE-11940 URL: https://issues.apache.org/jira/browse/IGNITE-11940 Project: Ignite Issue Type: Bug Components: mvcc Affects Versions: 2.7.5 Reporter: Ivan Pavlukhin Assignee: Ivan Pavlukhin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
