Re: Ignite Modularization
Hi Ivan, Thanks for looking into this. 1. I did not get from IEP whether Thin Clients have a separate > repository and a release lifecycle or not. Sorry for the confusion. Yes, such clients have to be in separate repos and might have their own lifecycles. Updated the wiki. > 2. Are we going to exclude tests for unsupported modules from Ignite > TeamCity? Yes, that's my thinking, an unsupported integration won't be part of Ignite modular ecosystem and won't be tested by the community. Any objections? > 3. Will we adress implementing Java 9+ modules during that process? I think this should be optional. Do you think we need to do it in the first instance? - Denis On Sun, Jun 30, 2019 at 10:45 PM Павлухин Иван wrote: > Hi Denis, > > I fully support the idea. Could you please clarify on following points: > 1. I did not get from IEP whether Thin Clients have a separate > repository and a release lifecycle or not. > 2. Are we going to exclude tests for unsupported modules from Ignite > TeamCity? > 3. Will we adress implementing Java 9+ modules during that process? > > чт, 27 июн. 2019 г. в 18:11, Denis Magda : > > > > Ignite developers and users, > > > > I'd like us to consider Ignite modularization as part of Ignite 3.0 > > timeframe. Presently, Ignite codebase mixes both core capabilities with > 3rd > > party integrations. It leads to the following: > > > >- Cumbersome and continuously growing codebase with many 3rd-party > >dependencies. > >- Some of the integrations are questionable and should no longer be > >supported by the community at all. > >- Integrations evolution is bound to Ignite release cycles even though > >no changes are needed in the core. > >- Ignite community has to support everything (test, release, fix, > >continue development) which requires to have particular integration > experts > >on a permanent basis - doesn't work. > > > > Here is an IEP: > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization > > > > Please review it, share feedback. Pay attention to the list of > integrations > > that should no longer be supported by the community. > > > > > > - > > Denis > > > > -- > Best regards, > Ivan Pavlukhin >
Re: [DISCUSSION] Ignite 3.0 and to be removed list
I wouldn't kick off dozens of voting discussions. Instead, the content on the wiki page needs to be cleaned and rearranged. This will make the content readable and comprehensible. I can do that. Next, let's ask the user community for an opinion. After reviewing and incorporating the latter we can do one more dev list discussion with the last call for opinions. Next, will be the voting time. If there is a feature someone from the dev list is against of removing, then we can start a separate vote for it later. But let's get into those cases first. - Denis On Mon, Jul 1, 2019 at 1:47 AM Dmitriy Pavlov wrote: > I propose each removal should have separated formal vote thread with > consensus approval (since it is code modification). > > This means a single binding objection with justification is a blocker for > removal. > > We need separation to let community members pick up an interesting topic > from email subject. Not all members reading carefully each post in > mile-long threads. > > пн, 1 июл. 2019 г. в 11:17, Anton Vinogradov : > > > +1 to email survey with following types of votes > > - silence (agree with all proposed removals) > > - we have to keep XXX because ... > > > > As a result, will gain lists > > "to be removed" - no one objected > > "can be removed" - single objection > > "should be kept" - multi objections > > > > Denis or Dmitry Pavlov, could you please lead this thread? > > > > On Sat, Jun 29, 2019 at 12:27 AM Denis Magda wrote: > > > > > Alex, > > > > > > I would do an email survey to hear an opinion of why someone believes a > > > feature A has to stay. It makes sense to ask about the APIs to be > removed > > > as well as integrations to go out of community support [1] in the same > > > thread. > > > > > > Has everyone expressed an opinion? If yes, I can go ahead and format > the > > > wishlist page and make it structured for the user thread. > > > > > > [1] > > > > > > > > > http://apache-ignite-developers.2346864.n4.nabble.com/Ignite-Modularization-td42486.html > > > - > > > Denis > > > > > > > > > On Fri, Jun 28, 2019 at 8:54 AM Alexey Goncharuk < > > > alexey.goncha...@gmail.com> > > > wrote: > > > > > > > Anton, good point. > > > > > > > > Do you have any idea how we can keep track of the voting? Should we > > > launch > > > > a google survey or survey monkey? Voting by email? > > > > > > > > пт, 28 июн. 2019 г. в 11:24, Anton Vinogradov : > > > > > > > > > Alexey, > > > > > > > > > > Thank's for keeping an eye on page updates. > > > > > Near Caches is not a bad feature, but it should be used with > caution. > > > > > At least we have to explain how it works on readme.io, why and > when > > it > > > > > should be used because usage can drop the performance instead of > > > > increasing > > > > > it. > > > > > > > > > > Anyway, I added near caches because I never heard someone used them > > > > > meaningfully, not like a silver bullet. > > > > > So, that's just a proposal :) > > > > > > > > > > Also, I'd like to propose to have some voting about full list later > > to > > > > gain > > > > > "must be removed", "can be removed" and "should be kept" lists. > > > > > > > > > > On Thu, Jun 27, 2019 at 1:03 PM Alexey Goncharuk < > > > > > alexey.goncha...@gmail.com> > > > > > wrote: > > > > > > > > > > > Anton, > > > > > > > > > > > > I would like to pull-up the discussion regarding the near caches > - > > I > > > > > cannot > > > > > > agree this is a feature that needs to be removed. Near caches > > provide > > > > > > significant read performance improvements and, to the best of my > > > > > knowledge, > > > > > > are used in several cases in production. Can you elaborate on the > > > > > > shortcomings you faced? Maybe we can improve both internal code > and > > > > user > > > > > > experience? > > > > > > > > > > > > пт, 21 июн. 2019 г. в 10:42, Dmitry Melnichuk < > > > > > > dmitry.melnic...@nobitlost.com>: > > > > > > > > > > > > > Dmitry, > > > > > > > As a Python thin client developer, I think that separate > > repository > > > > is > > > > > > > a truly great idea! > > > > > > > On Tue, 2019-06-18 at 21:29 +0300, Dmitriy Pavlov wrote: > > > > > > > > - Move to separate repositories: thin clients (at least > > non-Java > > > > > > > > > > > > > > > > > ones) > > > > > > > > > > > > > > > > > > > > > > > > > > > >
SpiUriDeploy TC job fails on Java 9+
Hi, SpiUriDeploy fails on TC when it is run with any Java 9+ version [1]. The reason is very simple, it contains one preliminary build step which is forced to run with Java 8. But for Java 9+ builds we pass module-related arguments and consequently it leads to Java 8 failure (unknown argument). I tried to run that step on JDK used for all other steps but faced a compilation error. It was on Java 9 and it said that target version 11 was not supported. Due to some reason we configure 11 in parent pom for java-9+ profile. In my mind simple solution here is to set target version to 9. But perhaps there is a clever way to use maximum version supported by a used JDK. Do you have an ideas? [1] https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_SpiUriDeploy_IgniteTests24Java8=%3Cdefault%3E=buildTypeStatusDiv -- Best regards, Ivan Pavlukhin
Re: [MTCGA]: new failures in builds [4172317, 4172313, 4172278] needs to be handled
I've checked Pavel's change and it is .NET only https://issues.apache.org/jira/browse/IGNITE-11867 I guess this change may have an effect on other tests пн, 1 июл. 2019 г. в 21:24, Dmitriy Pavlov : > Alexey, Pavel, could you please double check reasons for the failure? > > пн, 1 июл. 2019 г. в 21:11, : > >> Hi Igniters, >> >> I've detected some new issue on TeamCity to be handled. You are more >> than welcomed to help. >> >> If your changes can lead to this failure(s): We're grateful that you >> were a volunteer to make the contribution to this project, but things >> change and you may no longer be able to finalize your contribution. >> Could you respond to this email and indicate if you wish to continue and >> fix test failures or step down and some committer may revert you commit. >> >> *New stable failure of a flaky test in master >> LocalWalModeChangeDuringRebalancingSelfTest.testWithExchangesMerge >> https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-6585115376754732686=%3Cdefault%3E=testDetails >> Changes may lead to failure were done by >> - alexey.scherbak...@gmail.com >> https://ci.ignite.apache.org/viewModification.html?modId=886764 >> - ptupit...@apache.org >> https://ci.ignite.apache.org/viewModification.html?modId=886762 >> >> *New stable failure of a flaky test in master >> GridCacheRebalancingWithAsyncClearingMvccTest.testPartitionClearingNotBlockExchange >> >> https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-7007912051428984819=%3Cdefault%3E=testDetails >> Changes may lead to failure were done by >> - alexey.scherbak...@gmail.com >> https://ci.ignite.apache.org/viewModification.html?modId=886764 >> - ptupit...@apache.org >> https://ci.ignite.apache.org/viewModification.html?modId=886762 >> >> *New stable failure of a flaky test in master >> GridCacheRebalancingAsyncSelfTest.testComplexRebalancing >> https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-8829809273565657995=%3Cdefault%3E=testDetails >> Changes may lead to failure were done by >> - alexey.scherbak...@gmail.com >> https://ci.ignite.apache.org/viewModification.html?modId=886764 >> - ptupit...@apache.org >> https://ci.ignite.apache.org/viewModification.html?modId=886762 >> >> - Here's a reminder of what contributors were agreed to do >> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute >> - Should you have any questions please contact >> dev@ignite.apache.org >> >> Best Regards, >> Apache Ignite TeamCity Bot >> https://github.com/apache/ignite-teamcity-bot >> Notification generated at 21:10:24 01-07-2019 >> >
[jira] [Created] (IGNITE-11951) Set ThreadLocal node name only once in JdkMarshaller
Ivan Pavlukhin created IGNITE-11951:
---
Summary: Set ThreadLocal node name only once in JdkMarshaller
Key: IGNITE-11951
URL: https://issues.apache.org/jira/browse/IGNITE-11951
Project: Ignite
Issue Type: Improvement
Affects Versions: 2.7.5
Reporter: Ivan Pavlukhin
Assignee: Ivan Pavlukhin
Currently {{JdkMarshaller}} saves a node name twice in couple of
marshall/unmarshall methods. Code can be improved to do it only once.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: [MTCGA]: new failures in builds [4172317, 4172313, 4172278] needs to be handled
Alexey, Pavel, could you please double check reasons for the failure? пн, 1 июл. 2019 г. в 21:11, : > Hi Igniters, > > I've detected some new issue on TeamCity to be handled. You are more than > welcomed to help. > > If your changes can lead to this failure(s): We're grateful that you were > a volunteer to make the contribution to this project, but things change and > you may no longer be able to finalize your contribution. > Could you respond to this email and indicate if you wish to continue and > fix test failures or step down and some committer may revert you commit. > > *New stable failure of a flaky test in master > LocalWalModeChangeDuringRebalancingSelfTest.testWithExchangesMerge > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-6585115376754732686=%3Cdefault%3E=testDetails > Changes may lead to failure were done by > - alexey.scherbak...@gmail.com > https://ci.ignite.apache.org/viewModification.html?modId=886764 > - ptupit...@apache.org > https://ci.ignite.apache.org/viewModification.html?modId=886762 > > *New stable failure of a flaky test in master > GridCacheRebalancingWithAsyncClearingMvccTest.testPartitionClearingNotBlockExchange > > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-7007912051428984819=%3Cdefault%3E=testDetails > Changes may lead to failure were done by > - alexey.scherbak...@gmail.com > https://ci.ignite.apache.org/viewModification.html?modId=886764 > - ptupit...@apache.org > https://ci.ignite.apache.org/viewModification.html?modId=886762 > > *New stable failure of a flaky test in master > GridCacheRebalancingAsyncSelfTest.testComplexRebalancing > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-8829809273565657995=%3Cdefault%3E=testDetails > Changes may lead to failure were done by > - alexey.scherbak...@gmail.com > https://ci.ignite.apache.org/viewModification.html?modId=886764 > - ptupit...@apache.org > https://ci.ignite.apache.org/viewModification.html?modId=886762 > > - Here's a reminder of what contributors were agreed to do > https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute > - Should you have any questions please contact > dev@ignite.apache.org > > Best Regards, > Apache Ignite TeamCity Bot > https://github.com/apache/ignite-teamcity-bot > Notification generated at 21:10:24 01-07-2019 >
[MTCGA]: new failures in builds [4172317, 4172313, 4172278] needs to be handled
Hi Igniters, I've detected some new issue on TeamCity to be handled. You are more than welcomed to help. If your changes can lead to this failure(s): We're grateful that you were a volunteer to make the contribution to this project, but things change and you may no longer be able to finalize your contribution. Could you respond to this email and indicate if you wish to continue and fix test failures or step down and some committer may revert you commit. *New stable failure of a flaky test in master LocalWalModeChangeDuringRebalancingSelfTest.testWithExchangesMerge https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-6585115376754732686=%3Cdefault%3E=testDetails Changes may lead to failure were done by - alexey.scherbak...@gmail.com https://ci.ignite.apache.org/viewModification.html?modId=886764 - ptupit...@apache.org https://ci.ignite.apache.org/viewModification.html?modId=886762 *New stable failure of a flaky test in master GridCacheRebalancingWithAsyncClearingMvccTest.testPartitionClearingNotBlockExchange https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-7007912051428984819=%3Cdefault%3E=testDetails Changes may lead to failure were done by - alexey.scherbak...@gmail.com https://ci.ignite.apache.org/viewModification.html?modId=886764 - ptupit...@apache.org https://ci.ignite.apache.org/viewModification.html?modId=886762 *New stable failure of a flaky test in master GridCacheRebalancingAsyncSelfTest.testComplexRebalancing https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-8829809273565657995=%3Cdefault%3E=testDetails Changes may lead to failure were done by - alexey.scherbak...@gmail.com https://ci.ignite.apache.org/viewModification.html?modId=886764 - ptupit...@apache.org https://ci.ignite.apache.org/viewModification.html?modId=886762 - Here's a reminder of what contributors were agreed to do https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute - Should you have any questions please contact dev@ignite.apache.org Best Regards, Apache Ignite TeamCity Bot https://github.com/apache/ignite-teamcity-bot Notification generated at 21:10:24 01-07-2019
[jira] [Created] (IGNITE-11950) Typo in log message about automatic adjustment of WAL archive size
Jan Schäfer created IGNITE-11950:
Summary: Typo in log message about automatic adjustment of WAL
archive size
Key: IGNITE-11950
URL: https://issues.apache.org/jira/browse/IGNITE-11950
Project: Ignite
Issue Type: Bug
Components: general
Affects Versions: 2.7.5
Reporter: Jan Schäfer
Fix For: 2.8
IF the WAL archive size is not set correctly the resulting log message contains
a typo:
{noformat}
Automatically adjusted max WAL archive size to 8.0 GiB (to override, use
DataStorageConfiguration.setMaxWalArhiveSize){noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IGNITE-11949) Yardstick benchmarks for WAL page snapshot compression
Aleksey Plekhanov created IGNITE-11949: -- Summary: Yardstick benchmarks for WAL page snapshot compression Key: IGNITE-11949 URL: https://issues.apache.org/jira/browse/IGNITE-11949 Project: Ignite Issue Type: Improvement Components: yardstick Reporter: Aleksey Plekhanov Assignee: Aleksey Plekhanov WAL page snapshots compression (implemented by IGNITE-11336) can be enabled by modifying config.xml file. It will be better to configure benchmarks by command line options. Also, some new probes (WAL size for example) can be added. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Migration to JUnit 5
Hi, Igniters
During work on IEP-30, which is about JUnit migration, I found that some
tests in examples module were commented [1] with the remark, that they
should be fixed in the ticket IGNITE-711 [2] which is about the
implementation of Java 8 examples.
In the context of the ticket IGNITE-10973 [3] I want to uncomment them and
mark as @Disabled. Is it really need to disable mentioned tests or I can
just remove them as outdated?
[1]
https://github.com/apache/ignite/pull/6606/files#diff-ed48193d25d777a2c30c187fa20a1a65L65
[2] https://issues.apache.org/jira/browse/IGNITE-711
[3] https://issues.apache.org/jira/browse/IGNITE-10973
вт, 26 февр. 2019 г. в 18:51, Ivan Fedotov :
> Ivan,
> I will investigate GridAbstractTest refactoring issue more precisely when
> I finish with JUnit3Legacy classes. Anyway, I will keep in touch with you
> and the community on the most significant moments.
>
> JUnit5 docs say that functionality is not full "especially with regard to
> reporting". On the other hand, I also agree with docs that it is the
> easiest way that does not require to touch CI infrastructure. I am going to
> try @RunWith(JUnitPlatform.class) construction with features from IEP to
> make sure that we will have the full support of them. The alternative way
> is dynamic tests [1], but the problem is that we add methods to suites
> manually, not via @Test annotation. It is some kind of rollback to JUnit3
> syntax.
>
> Anton,
> thank you for the reminder, I will update IEP according to the
> conversation.
>
> [1] https://www.baeldung.com/junit5-dynamic-tests
>
> вт, 26 февр. 2019 г. в 17:56, Anton Vinogradov :
>
>> Folks,
>>
>> Please make sure you keep IEP updated and each issue mentioned.
>>
>> On Tue, Feb 26, 2019 at 4:28 PM Павлухин Иван
>> wrote:
>>
>> > Ivan,
>> >
>> > Thank you for detailed answers! I would put a great care to
>> > @RunWith(JUnitPlatform.class) construction. As stated in junit5 docs
>> > [1] it does not support all features and unfortunately it is not clear
>> > how limited it is. Also, it is some kind of transitional mechanism
>> > which was not designed for being a long term solution.
>> >
>> > And I fully support an idea of refactoring GridAbstractTest. I think
>> > it is possible to make a significant improvement here.
>> >
>> > [1]
>> >
>> https://junit.org/junit5/docs/current/user-guide/#running-tests-junit-platform-runner
>> >
>> > пн, 25 февр. 2019 г. в 17:41, Ivan Fedotov :
>> > >
>> > > Hello Nikolay.
>> > >
>> > > The prime benefits are more comfortable work with flaky tests, Java 8
>> > tests
>> > > compatibility, user-friendly syntaxis in parametrized tests and
>> others.
>> > > The most significant features list you can find in IEP-30 Motivation
>> > > section.
>> > >
>> > > If you have any specific questions about JUnit5 feel free to ask me.
>> > >
>> > > пн, 25 февр. 2019 г. в 16:55, Nikolay Izhikov :
>> > >
>> > > > Hello, Ivan.
>> > > >
>> > > > May be I miss some mail - if yes, can you repeat it.
>> > > > What is advantages of migration from junit 4 to 5?
>> > > > Why we should do it?
>> > > >
>> > > >
>> > > > пн, 25 февр. 2019 г. в 16:33, Ivan Fedotov :
>> > > >
>> > > > > Ivan,
>> > > > > That is my thoughts according to your questions.
>> > > > >
>> > > > > 1. I tried to implement test suits with JUnit4 compatibility
>> layer.
>> > The
>> > > > > basic concept is to use @RunWith(JUnitPlatform.class)
>> @SelectClasses
>> > > > > ({...})[1] and on
>> > > > > CI Ignite it works fine.
>> > > > >
>> > > > > 2. According to @Rules, there are several ways to solve it:
>> > > > > 2.1 Leave JUnit4 code without changes. It will work because of
>> > > > Vintage
>> > > > > module
>> > > > > 2.2 Rewrite the @Rule as an Extension. The work of extension
>> is
>> > > > similar
>> > > > > to the @Rules work, but it is extracted in an Extension class.
>> > > > > For more information about extensions, please, follow the IEP
>> > [2].
>> > > > > In my opinion, the easiest and the most understandable way is to
>> > leave
>> > > > > GridAbstractTest in current form. It will work with JUnit5
>> > > > > syntaxis and abilities.
>> > > > >
>> > > > > 3. I faced a couple of problems during dealing with dynamic and
>> > static
>> > > > > tests in one project with JUnit5. The problem occurs with surefire
>> > > > version:
>> > > > > static tests work fine with 2.21x and earlier and with dynamic
>> > tests, the
>> > > > > situation is vice versa, it works with > 2.21x surefire version.
>> > > > > We can use helpful surefire dependency to use static tests with
>> the
>> > > > newest
>> > > > > surefire version [3], but dynamic tests become unavailable from
>> pure
>> > > > > Maven and accordingly from CI Ignite (from IDE all is fine).
>> > > > > I can suggest leaving this type of tests on JUnit4 on the current
>> > stage -
>> > > > > they are in the vast minority.
>> > > > >
>> > > > > Let me comment on your side notes.
>> > > > >
>> > > > > I am not against the stable and
Re: Fwd: [DISCUSSION][IEP-35] Metrics configuration
Hello, Ivan.
Thanks for the feedback!
> 1. Specifying subset of metrics which are exported to an external system.
Covered by exporter filter [1]
> 2. Subset of metrics which is collected (enable/disable sensor).
What is "disabled" sensor?
Why do we need it? (as long as sensort is just a AtomicLong)
> 3. A particular metric (sensor) parameters.
This is the point of the discussed ticket [2]
[1]
https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/metric/PushMetricsExporterAdapter.java#L41
[2] https://issues.apache.org/jira/browse/IGNITE-11927
В Пн, 01/07/2019 в 15:39 +0300, Павлухин Иван пишет:
> Nikolay, Igniters,
>
> In my mind there are several configuration aspects:
> 1. Specifying subset of metrics which are exported to an external system.
> 2. Subset of metrics which is collected (enable/disable sensor).
> 3. A particular metric (sensor) parameters.
>
> Are we going to address all points in the same config (file)? In my
> experience with metrics (actually not so rich) I have not seen metric
> configs similar to logging configs. Are there examples of such
> practice in industry?
>
> Also, looking at a line like "cache.my-cahe.GetLatency=50,100,250,500"
> I cannot tell much from scratch. My first thought is that having named
> parameters can be more readable, e.g. (roughly):
> cache.my-cahe.GetLatency={
> intervals: [50, 100, 250, 500]
> }
>
> пт, 28 июн. 2019 г. в 13:02, Nikolay Izhikov :
> >
> > Hello, Alexey.
> >
> > Thanks for the feedback!
> >
> > > My only concert is that we should have the metrics framework configuration
> > > as the first-citizen of the framework itself
> >
> > Yes. I planned to add `void configure(String param)` method to the metric
> > API.
> >
> > > but change the metrics parameters in
> > > runtime from JMX or command-line, etc.
> >
> > I've add requirement of JMX method to the ticket:
> >
> > https://issues.apache.org/jira/browse/IGNITE-11927
> >
> > > Another concern is to have an
> > > ability to disable/enable metrics per metrics group/prefix.
> >
> > Yes, we discusss it.
> > But, let's make it clear:
> >
> > *What is disabling metric?*
> >
> > Looks like exporter filter solve this task.
> >
> > В Чт, 27/06/2019 в 16:24 +0300, Alexey Goncharuk пишет:
> > > Nikolay,
> > >
> > > My only concert is that we should have the metrics framework configuration
> > > as the first-citizen of the framework itself. This way, we can configure
> > > the metrics not only from file, but change the metrics parameters in
> > > runtime from JMX or command-line, etc. Another concern is to have an
> > > ability to disable/enable metrics per metrics group/prefix.
> > >
> > > The logger-like configuration meets these suggestions given that the
> > > configuration is generalized into the metrics framework.
> > >
> > > What do you think?
> > >
> > > чт, 27 июн. 2019 г. в 12:30, Nikolay Izhikov :
> > >
> > > > Hello, Igniters.
> > > >
> > > > As you may know, I've contributed Phase1 [1] for IEP-35 [2].
> > > > Now we have metrics subsystem and can create and export any metrics from
> > > > Ignite.
> > > >
> > > > I think user(administrator of Ignite) should be able to configure some
> > > > metrics params in a common way [3]
> > > >
> > > > I propose to use the same way from logging frameworks.
> > > > We should define some file format Ignite can understand.
> > > > An administrator fills configuration file to configure one or several
> > > > metrics.
> > > > Ignite will analyze the file and use provided params during metrics
> > > > creation.
> > > >
> > > > For now, we have 2 types of metrics that should be configured:
> > > >
> > > > * HistrogramMetric [4]
> > > > This metric is a count of measurement that falls into
> > > > predefined intervals.
> > > > An example is "Request processing time distribution".
> > > > We want to calculate a count of requests processed
> > > > quicker
> > > > then 50ms, 50-100, 100-250, 250-500 and slower.
> > > >
> > > > * HitRateMetric [5]
> > > > This metric is a count of events in the last time
> > > > interval.
> > > > An example is the "Count of requests processed in the
> > > > last
> > > > 5 seconds".
> > > >
> > > > Example of file content:
> > > >
> > > >
> > > > cache.my-cahe.GetLatency=50,100,250,500 #Params for the histogram metric
> > > > with the name `cache.my-cahe.get`
> > > > cache.my-cache.RebalancingKeysRate=6 #Param for existing
> > > > HitRateMetric
> > > > that hold "Estimated rebalancing speed in keys".
> > > >
> > > >
> > > > Please, share your vision.
> > > >
> > > > [1]
> > > > https://github.com/apache/ignite/commit/fdaa310430aefff07994eb35510d3416886b5bbe
> > > > [2]
> > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392
> > > > [3] https://issues.apache.org/jira/browse/IGNITE-11927
>
Fwd: [DISCUSSION][IEP-35] Metrics configuration
Nikolay, Igniters,
In my mind there are several configuration aspects:
1. Specifying subset of metrics which are exported to an external system.
2. Subset of metrics which is collected (enable/disable sensor).
3. A particular metric (sensor) parameters.
Are we going to address all points in the same config (file)? In my
experience with metrics (actually not so rich) I have not seen metric
configs similar to logging configs. Are there examples of such
practice in industry?
Also, looking at a line like "cache.my-cahe.GetLatency=50,100,250,500"
I cannot tell much from scratch. My first thought is that having named
parameters can be more readable, e.g. (roughly):
cache.my-cahe.GetLatency={
intervals: [50, 100, 250, 500]
}
пт, 28 июн. 2019 г. в 13:02, Nikolay Izhikov :
>
> Hello, Alexey.
>
> Thanks for the feedback!
>
> > My only concert is that we should have the metrics framework configuration
> > as the first-citizen of the framework itself
>
> Yes. I planned to add `void configure(String param)` method to the metric API.
>
> > but change the metrics parameters in
> > runtime from JMX or command-line, etc.
>
> I've add requirement of JMX method to the ticket:
>
> https://issues.apache.org/jira/browse/IGNITE-11927
>
> > Another concern is to have an
> > ability to disable/enable metrics per metrics group/prefix.
>
> Yes, we discusss it.
> But, let's make it clear:
>
> *What is disabling metric?*
>
> Looks like exporter filter solve this task.
>
> В Чт, 27/06/2019 в 16:24 +0300, Alexey Goncharuk пишет:
> > Nikolay,
> >
> > My only concert is that we should have the metrics framework configuration
> > as the first-citizen of the framework itself. This way, we can configure
> > the metrics not only from file, but change the metrics parameters in
> > runtime from JMX or command-line, etc. Another concern is to have an
> > ability to disable/enable metrics per metrics group/prefix.
> >
> > The logger-like configuration meets these suggestions given that the
> > configuration is generalized into the metrics framework.
> >
> > What do you think?
> >
> > чт, 27 июн. 2019 г. в 12:30, Nikolay Izhikov :
> >
> > > Hello, Igniters.
> > >
> > > As you may know, I've contributed Phase1 [1] for IEP-35 [2].
> > > Now we have metrics subsystem and can create and export any metrics from
> > > Ignite.
> > >
> > > I think user(administrator of Ignite) should be able to configure some
> > > metrics params in a common way [3]
> > >
> > > I propose to use the same way from logging frameworks.
> > > We should define some file format Ignite can understand.
> > > An administrator fills configuration file to configure one or several
> > > metrics.
> > > Ignite will analyze the file and use provided params during metrics
> > > creation.
> > >
> > > For now, we have 2 types of metrics that should be configured:
> > >
> > > * HistrogramMetric [4]
> > > This metric is a count of measurement that falls into
> > > predefined intervals.
> > > An example is "Request processing time distribution".
> > > We want to calculate a count of requests processed quicker
> > > then 50ms, 50-100, 100-250, 250-500 and slower.
> > >
> > > * HitRateMetric [5]
> > > This metric is a count of events in the last time
> > > interval.
> > > An example is the "Count of requests processed in the last
> > > 5 seconds".
> > >
> > > Example of file content:
> > >
> > >
> > > cache.my-cahe.GetLatency=50,100,250,500 #Params for the histogram metric
> > > with the name `cache.my-cahe.get`
> > > cache.my-cache.RebalancingKeysRate=6 #Param for existing HitRateMetric
> > > that hold "Estimated rebalancing speed in keys".
> > >
> > >
> > > Please, share your vision.
> > >
> > > [1]
> > > https://github.com/apache/ignite/commit/fdaa310430aefff07994eb35510d3416886b5bbe
> > > [2]
> > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392
> > > [3] https://issues.apache.org/jira/browse/IGNITE-11927
> > > [4]
> > > https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/metric/impl/HistogramMetric.java
> > > [5]
> > > https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/metric/impl/HitRateMetric.java
> > >
--
Best regards,
Ivan Pavlukhin
-BEGIN PGP SIGNATURE-
iQEzBAABCgAdFiEEOiTcLcdgyP2exB5ZbiaPbjg91GUFAl0V5iYACgkQbiaPbjg9
1GWWDQf/Tfy5+lmuCTTBlrp8QetoTKrr+xdU4BuanBqLwwz8Yk+/OvA6fj1HmKDM
eIK+xGqIeYk7u8NMTURuIjTGZ+jBSd+q6riB2qaVBiR5ksRFxXCa7P+guDg8Kmm5
qQ9ocXy3tfCsQgG9Ig6d60GRuM7dW8GlUFtA/wt4zxpAA5zsdNyCJUNxjXY5zo6V
r8uSvKprxgMsXKiSiZsjoQXJtqlIoMJldeaNrq7i2zUV1w/nIeLKygLfMfI8Ty5n
2v2ocbIa07QekVBEz8JMx88grs3otmGBdJyCFwt1JZxRPsg2nEU+sJcRSs8jJl8P
tOGZbrfTByGFhwOTh8zFP55NfLm87g==
=T+jn
-END PGP SIGNATURE-
Re: Performance drop with New Monitoring.
Hello, Igniter. I think we should fix the issue with slight changes in internal API (was not released yet). Changes in API was independently proposed by me and Alexey Plekhanov [1]: 1. We should introduce `MetricGroup` - collection of metric of one specific Ignite entity(io subsystem, data region, cache, etc.). 2. We can remove entire `MetricGroup` (on cache or cache group destroy) with complexity of O(1). 3. We can remove metric from `MetricGroup` with complexity of O(1). [1] https://issues.apache.org/jira/browse/IGNITE-11920?jql=labels%20%3D%20IEP-35%20ORDER%20BY%20status%20ASC В Пт, 28/06/2019 в 19:10 +0300, Nikolay Izhikov пишет: > Hello, Maksim. > > I will take a look shortly. > > В Пт, 28/06/2019 в 19:02 +0300, Maksim Stepachev пишет: > > Hello, Igniters. > > > > I see that the ticket - "[IGNITE-11848] New Monitoring. Phase1" breaks > > cache stop performance. It affects next tests: testParallelStartAndStop, > > testStartManyCaches > > > > The spent time was increased 6 times:( > > > > Look at: > > https://ci.ignite.apache.org/project.html?tab=testDetails_IgniteTests24Java8=%3Cdefault%3E=IgniteTests24Java8=-145777872614811891=4 signature.asc Description: This is a digitally signed message part
[jira] [Created] (IGNITE-11948) Row count statistics tests fail frequently
Ivan Pavlukhin created IGNITE-11948: --- Summary: Row count statistics tests fail frequently Key: IGNITE-11948 URL: https://issues.apache.org/jira/browse/IGNITE-11948 Project: Ignite Issue Type: Bug Components: sql Reporter: Ivan Pavlukhin RowCountTableStatisticsSurvivesNodeRestartTest and RowCountTableStatisticsUsageTest fail frequently on TC https://ci.ignite.apache.org/viewLog.html?buildId=4238573=IgniteTests24Java8_BinaryObjectsSimpleMapperQueries -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-11947) [TC Bot] New failures notification are absent for several new failures
Dmitriy Pavlov created IGNITE-11947: --- Summary: [TC Bot] New failures notification are absent for several new failures Key: IGNITE-11947 URL: https://issues.apache.org/jira/browse/IGNITE-11947 Project: Ignite Issue Type: Bug Reporter: Dmitriy Pavlov Assignee: Dmitriy Pavlov Notifications to email are missed sometimes. One possible reason is buildStartTs, which was not filled correctly in Issue, but after fix https://github.com/apache/ignite-teamcity-bot/commit/5573467d04eedbe9f79dd2b19e9675b76af78985 the situation is still the same, notifications not send -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Data loading with thin clients
Denis, We have streaming support for ODBC. If they only need SQL maybe they should use ODBC instead of thin client. Best Regards, Igor On Sat, Jun 29, 2019 at 1:01 AM Denis Magda wrote: > Igor, > > The user prefers using SQL syntax which is widespread. For sure, we'll > recommend alternate solutions as long as INSERT is not suited for loading > if a thin client is used. > > However, how have we supported the streaming mode for INSERTS for ODBC? > Isn't it already based on the thin client protocol? > > - > Denis > > > On Fri, Jun 28, 2019 at 8:02 AM Igor Sapego wrote: > > > If they use python client, why wont they just use put_all()? > > > > To me it just looks like they are using wrong tool here. > > > > As for your questing, Affinity Awareness for thin clients is only for > Cache > > API now, it does not affect SQL. > > > > Best Regards, > > Igor > > > > > > On Thu, Jun 27, 2019 at 7:33 PM Denis Magda wrote: > > > > > Igor, Igniters, > > > > > > I've come across a couple of posts recently where users attempted to > > > preload initial data with a thin client like Python: > > > > > > https://stackoverflow.com/questions/56778778/apache-ignite-inserts-extremely-slow/56791088#56791088 > > > > > > Our JDBC/ODBC drivers have a special streaming mode that boosts the > > > loading for such use cases. > > > > > > Will there be any benefits from the initial data loading standpoint > when > > > partition-awareness is released or do we need to enhance our thin > > clients' > > > protocol? Users use SQL for preloading. > > > > > > > > > Denis Magda > > > > > >
Re: [DISCUSSION] Ignite 3.0 and to be removed list
I propose each removal should have separated formal vote thread with consensus approval (since it is code modification). This means a single binding objection with justification is a blocker for removal. We need separation to let community members pick up an interesting topic from email subject. Not all members reading carefully each post in mile-long threads. пн, 1 июл. 2019 г. в 11:17, Anton Vinogradov : > +1 to email survey with following types of votes > - silence (agree with all proposed removals) > - we have to keep XXX because ... > > As a result, will gain lists > "to be removed" - no one objected > "can be removed" - single objection > "should be kept" - multi objections > > Denis or Dmitry Pavlov, could you please lead this thread? > > On Sat, Jun 29, 2019 at 12:27 AM Denis Magda wrote: > > > Alex, > > > > I would do an email survey to hear an opinion of why someone believes a > > feature A has to stay. It makes sense to ask about the APIs to be removed > > as well as integrations to go out of community support [1] in the same > > thread. > > > > Has everyone expressed an opinion? If yes, I can go ahead and format the > > wishlist page and make it structured for the user thread. > > > > [1] > > > > > http://apache-ignite-developers.2346864.n4.nabble.com/Ignite-Modularization-td42486.html > > - > > Denis > > > > > > On Fri, Jun 28, 2019 at 8:54 AM Alexey Goncharuk < > > alexey.goncha...@gmail.com> > > wrote: > > > > > Anton, good point. > > > > > > Do you have any idea how we can keep track of the voting? Should we > > launch > > > a google survey or survey monkey? Voting by email? > > > > > > пт, 28 июн. 2019 г. в 11:24, Anton Vinogradov : > > > > > > > Alexey, > > > > > > > > Thank's for keeping an eye on page updates. > > > > Near Caches is not a bad feature, but it should be used with caution. > > > > At least we have to explain how it works on readme.io, why and when > it > > > > should be used because usage can drop the performance instead of > > > increasing > > > > it. > > > > > > > > Anyway, I added near caches because I never heard someone used them > > > > meaningfully, not like a silver bullet. > > > > So, that's just a proposal :) > > > > > > > > Also, I'd like to propose to have some voting about full list later > to > > > gain > > > > "must be removed", "can be removed" and "should be kept" lists. > > > > > > > > On Thu, Jun 27, 2019 at 1:03 PM Alexey Goncharuk < > > > > alexey.goncha...@gmail.com> > > > > wrote: > > > > > > > > > Anton, > > > > > > > > > > I would like to pull-up the discussion regarding the near caches - > I > > > > cannot > > > > > agree this is a feature that needs to be removed. Near caches > provide > > > > > significant read performance improvements and, to the best of my > > > > knowledge, > > > > > are used in several cases in production. Can you elaborate on the > > > > > shortcomings you faced? Maybe we can improve both internal code and > > > user > > > > > experience? > > > > > > > > > > пт, 21 июн. 2019 г. в 10:42, Dmitry Melnichuk < > > > > > dmitry.melnic...@nobitlost.com>: > > > > > > > > > > > Dmitry, > > > > > > As a Python thin client developer, I think that separate > repository > > > is > > > > > > a truly great idea! > > > > > > On Tue, 2019-06-18 at 21:29 +0300, Dmitriy Pavlov wrote: > > > > > > > - Move to separate repositories: thin clients (at least > non-Java > > > > > > > > > > > > > > > ones) > > > > > > > > > > > > > > > > > > > > >
Re: [DISCUSSION] Ignite 3.0 and to be removed list
+1 to email survey with following types of votes - silence (agree with all proposed removals) - we have to keep XXX because ... As a result, will gain lists "to be removed" - no one objected "can be removed" - single objection "should be kept" - multi objections Denis or Dmitry Pavlov, could you please lead this thread? On Sat, Jun 29, 2019 at 12:27 AM Denis Magda wrote: > Alex, > > I would do an email survey to hear an opinion of why someone believes a > feature A has to stay. It makes sense to ask about the APIs to be removed > as well as integrations to go out of community support [1] in the same > thread. > > Has everyone expressed an opinion? If yes, I can go ahead and format the > wishlist page and make it structured for the user thread. > > [1] > > http://apache-ignite-developers.2346864.n4.nabble.com/Ignite-Modularization-td42486.html > - > Denis > > > On Fri, Jun 28, 2019 at 8:54 AM Alexey Goncharuk < > alexey.goncha...@gmail.com> > wrote: > > > Anton, good point. > > > > Do you have any idea how we can keep track of the voting? Should we > launch > > a google survey or survey monkey? Voting by email? > > > > пт, 28 июн. 2019 г. в 11:24, Anton Vinogradov : > > > > > Alexey, > > > > > > Thank's for keeping an eye on page updates. > > > Near Caches is not a bad feature, but it should be used with caution. > > > At least we have to explain how it works on readme.io, why and when it > > > should be used because usage can drop the performance instead of > > increasing > > > it. > > > > > > Anyway, I added near caches because I never heard someone used them > > > meaningfully, not like a silver bullet. > > > So, that's just a proposal :) > > > > > > Also, I'd like to propose to have some voting about full list later to > > gain > > > "must be removed", "can be removed" and "should be kept" lists. > > > > > > On Thu, Jun 27, 2019 at 1:03 PM Alexey Goncharuk < > > > alexey.goncha...@gmail.com> > > > wrote: > > > > > > > Anton, > > > > > > > > I would like to pull-up the discussion regarding the near caches - I > > > cannot > > > > agree this is a feature that needs to be removed. Near caches provide > > > > significant read performance improvements and, to the best of my > > > knowledge, > > > > are used in several cases in production. Can you elaborate on the > > > > shortcomings you faced? Maybe we can improve both internal code and > > user > > > > experience? > > > > > > > > пт, 21 июн. 2019 г. в 10:42, Dmitry Melnichuk < > > > > dmitry.melnic...@nobitlost.com>: > > > > > > > > > Dmitry, > > > > > As a Python thin client developer, I think that separate repository > > is > > > > > a truly great idea! > > > > > On Tue, 2019-06-18 at 21:29 +0300, Dmitriy Pavlov wrote: > > > > > > - Move to separate repositories: thin clients (at least non-Java > > > > > > > > > > > > > ones) > > > > > > > > > > > > > > >
Re: Lightweight version of partitions map exchange
Hi, Igniters. I'm working on the implementation of lightweight PME for a baseline node leave case. [1] In my implementation, each node recalculates a new affinity and completes PME locally without distributed communication. This is possible because there are all partitions are distributed according to the baseline topology. And I found two possible blockers to do it without blocking updates: 1. Finalize partitions counter. It seems that we can't correctly collect gaps and process them without completing all txs. See the GridDhtPartitionTopologyImpl#finalizeUpdateCounters method. 2. Apply update counters. We can't correctly set HWM counter if primary left the cluster and sent updates to part of backups. Such updates can be processed later and break guarantee that LWM<=HWM. Is it impossible to leave a baseline node without waiting for all txs completed? 1. https://issues.apache.org/jira/browse/IGNITE-9913 ср, 5 июн. 2019 г. в 12:15, Nikita Amelchev : > > Maksim, > > I agree with you that we should implement current issue and do not > allow lightweight PME if there are MOVING partitions in the cluster. > > But now I'm investigating issue about finalizing update counters cause > it assumes that finalizing happens on exchange and all cache updates > are completed. Here we can wrong process update counters gaps and > break recently merged IGNITE-10078. > > And about phase 2, correct me if I misunderstood you. > You suggest do not move primary partitions on rebalancing completing > (do not change affinity assignment)? In this case, nodes recently join > to cluster will not have primary partitions and won't get a load after > rebalancing. > > чт, 30 мая 2019 г. в 19:55, Maxim Muzafarov : > > > > Igniters, > > > > > > I've looked through Nikita's changes and I think for the current issue > > [1] we should not allow the existence of MOVING partitions in the > > cluster (it must be stable) to run the lightweight PME on BLT node > > leave event occurred to achieve truly unlocked operations and here are > > my thoughts why. > > > > In general, as Nikita mentioned above, the existence of MOVING > > partitions in the cluster means that the rebalance procedure is > > currently running. It owns cache partitions locally and sends in the > > background (with additional timeout) the actual statuses of his local > > partitions to the coordinator node. So, we will always have a lag > > between local node partition states and all other cluster nodes > > partitions states. This lag can be very huge since previous > > #scheduleResendPartitions() is cancelled when a new cache group > > rebalance finished. Without the fair partition states synchronization > > (without full PME) and in case of local affinity recalculation on BLT > > node leave event, other nodes will mark such partitions LOST in most > > of the cases, which in fact are present in the cluster and saved on > > some node under checkpoint. I see that it cannot be solved by saving > > transition states of such partitions on each node. > > > > As for the case when the coordinator will calculate affinity and send > > "full map" to other nodes, I think it is better here to focus on > > designing a new lightweight PME when the rebalancing process finishes. > > Сurrently full distributed PME will occur anyway by the coordinator by > > sending CacheAffinityChaneMessage, but I think we can avoid it here, > > since no new MOVING or OWNING node partition states are introduced and > > all the previous mappings are still valid. We don't need a distributed > > PME if we will leave partition primaries on those nodes where they > > were, just set correct partition statuses via a light discovery > > message. > > > > So, my plan here can be: > > Phase 1. Lightweight PME on BLT node leave on a stable cluster (no > > MOVING partitions); > > Phase 2. Lightweight PME on BLT node finishes its rebalance procedure. > > > > Folks, Nikita, > > WDYT? > > > > [1] https://issues.apache.org/jira/browse/IGNITE-9913 > > > > On Fri, 24 May 2019 at 13:31, Nikita Amelchev wrote: > > > > > > Hello, Igniters! > > > > > > I am working on the implementation of lightweight PME for the case of > > > a BLT node leave. [1] > > > > > > There is a question: whether to allow lightweight PME if the cluster > > > has MOVING partitions? > > > > > > The problems that may happen if allow: > > > - Nodes can differently select the primary node from current OWNING > > > backups. > > > - One part of nodes can mark a partition as LOST and another one as > > > OWNING. > > > > > > We can take states of the partitions from the node2part map. The root > > > cause of those problems is that when rebalancing ends (get the last > > > message), it updates partition state of the local node to OWNING (and > > > schedules partitions resend). This may lead to different affinity > > > re-calculations on nodes. > > > > > > I see two solutions: > > > > > > 1. Nodes will store “moving-owning” transition of partitions state > > >
