Re: IGNITE-12358 Migrate ZeroMQ module to ignite-extensions
Hello, Can someone please help in review for these following PRs. I have received approval for release process from Alexey and would need a code review approval for following PR. Jira https://issues.apache.org/jira/browse/IGNITE-12358 PR https://github.com/apache/ignite-extensions/pull/5 https://github.com/apache/ignite/pull/7240 Regards, Saikat On Fri, Jan 17, 2020 at 8:38 PM Saikat Maitra wrote: > Yes sure, thank you Denis > > Regards, > Saikat > > On Thu, Jan 16, 2020 at 3:53 PM Denis Magda wrote: > >> Hi Saikat, >> >> Let's wait while Alex Goncharuk checks a similar PR here: >> >> http://apache-ignite-developers.2346864.n4.nabble.com/IGNITE-12361-Migrate-Flume-module-to-ignite-extensions-td45010.html >> >> After that, we can return to this pending pull request. >> >> - >> Denis >> >> >> On Sun, Jan 12, 2020 at 12:59 PM Saikat Maitra >> wrote: >> >> > Hi, >> > >> > I have raised PR for Ignite zeromq migration to Ignite Extensions repo. >> > >> > Jira https://issues.apache.org/jira/browse/IGNITE-12358 >> > >> > PR https://github.com/apache/ignite-extensions/pull/5 >> > https://github.com/apache/ignite/pull/7240 >> > >> > Please review and share feedback. >> > >> > This is part of our Modularization effort for Streamer modules >> > >> > https://issues.apache.org/jira/browse/IGNITE-12355 >> > >> > Regards, >> > Saikat >> > >> >
Re: IGNITE-12361 Migrate Flume module to ignite-extensions
Hi, Can someone please help in review for these following PRs. I have received approval for release process from Alexey and would need a code review approval for following PR. Jira https://issues.apache.org/jira/browse/IGNITE-12361 PR https://github.com/apache/ignite-extensions/pull/4 https://github.com/apache/ignite/pull/7227 Regards, Saikat On Tue, Feb 11, 2020 at 6:47 PM Saikat Maitra wrote: > Hi Alexey, > > > I think we can release for spring boot autoconfigure module. > > Nikolay - Do you have tentative timeline when you are planning for release > of spring boot autoconfigure module. > > > After that we are planning to make release for flink ext. > > > Since, each module are independent so they will be released independently. > > > Regards, > Saikat > > On Mon, 10 Feb 2020 at 7:33 AM, Alexey Goncharuk < > alexey.goncha...@gmail.com> wrote: > >> Saikat, >> >> Yes, I think we can go ahead with the modules PRs as long as reviewers are >> ok with the changes. Given that there is an activity around the spring >> module, which modules do you think will get to the first release? >> >> сб, 1 февр. 2020 г. в 21:37, Saikat Maitra : >> >> > Hi Alexey, >> > >> > Please let me know if I can share more info on the release process. I >> have >> > updated the issue confluence page on discussed approach for Ignite >> > Extensions. Do you think the open PRs can be merged in Ignite Extensions >> > repo? >> > >> > Independent Integrations: >> > >> > >> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations >> > Discussion Links: >> > >> > >> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-DiscussionLinks >> > Tickets: >> > >> > >> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-Tickets >> > >> > Regards, >> > Saikat >> > >> > On Sun, Jan 26, 2020 at 3:11 PM Saikat Maitra >> > wrote: >> > >> > > Hi Alexey, >> > > >> > > As discussed I have updated the wiki with agreed solution. >> > > >> > > Independent Integrations: >> > > >> > >> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations >> > > >> > > Discussion Links: >> > > >> > >> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-DiscussionLinks >> > > >> > > Tickets: >> > > >> > >> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-Tickets >> > > >> > > Please let me know if I can share more information. >> > > >> > > Regards, >> > > Saikat >> > > >> > > >> > > On Fri, Jan 17, 2020 at 9:16 PM Saikat Maitra < >> saikat.mai...@gmail.com> >> > > wrote: >> > > >> > >> Hello Alexey, >> > >> >> > >> Thank you for your email. >> > >> >> > >> 1. Yes, we discussed in dev list and agreed on creating a new >> repository >> > >> for hosting our Ignite integrations. Please find the discussion >> thread >> > >> below. I will update the wiki page as well and share updates. >> > >> >> > >> >> > >> >> > >> http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSS-Proposal-for-Ignite-Extensions-as-a-separate-Bahir-module-or-Incubator-project-td44064.html >> > >> >> > >> 2. I was hoping to complete migration of the following modules >> before we >> > >> go ahead with release. I am tracking the jira story here >> > >> https://issues.apache.org/jira/browse/IGNITE-12355 >> > >> >> > >>- Flink >> > >>- Twitter >> > >>- Storm >> > >>- ZeroMQ >> > >>- RocketMQ >> > >>- Flume >> > >>- MQTT >> > >>- Camel >> > >>- JMS >> > >> >> > >> 3. The dependencies for modules are pointing to latest snapshot of >> > >> ignite project and if there are changes in ignite master branch then >> > >> related affected Ignite extensions module also need to be modified. >> We >> > will >> > >> verify all the extensions for upcoming release but release only the >> one >> > >> that are impacted. We will plan to avoid publishing any extension >> unless >> > >> there are changes. Here is the discussion thread on release process: >> > >> >> > >> >> > >> >> > >> http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSS-dependencies-and-release-process-for-Ignite-Extensions-td44478.html >> > >> >> > >> 4. Sounds good, we can maintain a compatibility matrix to ensure we >> can >> > >> share Ignite Core and related Ignite Extensions. I have seen >> something >> > >> similar in Apache Beam and we can also create a similar wiki page to >> > >> maintain Ignite Core and Ignite Extensions dependencies >> > >> >> > >> https://beam.apache.org/documentation/runners/capability-matrix/ >> > >> >> > >> Regards, >> > >> Saikat >> > >> >> > >> >> > >> >> > >> On Fri, Jan 17, 2020 at 6:27 AM Alexey Goncharuk < >> > >> alexey.goncha...@gmail.com> wrote: >> > >> >> > >>> Saikat, >> > >>> >> > >>> Code-wise the PR looks ok because it basically moves the module
Re: IGNITE-12361 Migrate Flume module to ignite-extensions
Hi Alexey, I think we can release for spring boot autoconfigure module. Nikolay - Do you have tentative timeline when you are planning for release of spring boot autoconfigure module. After that we are planning to make release for flink ext. Since, each module are independent so they will be released independently. Regards, Saikat On Mon, 10 Feb 2020 at 7:33 AM, Alexey Goncharuk wrote: > Saikat, > > Yes, I think we can go ahead with the modules PRs as long as reviewers are > ok with the changes. Given that there is an activity around the spring > module, which modules do you think will get to the first release? > > сб, 1 февр. 2020 г. в 21:37, Saikat Maitra : > > > Hi Alexey, > > > > Please let me know if I can share more info on the release process. I > have > > updated the issue confluence page on discussed approach for Ignite > > Extensions. Do you think the open PRs can be merged in Ignite Extensions > > repo? > > > > Independent Integrations: > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations > > Discussion Links: > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-DiscussionLinks > > Tickets: > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-Tickets > > > > Regards, > > Saikat > > > > On Sun, Jan 26, 2020 at 3:11 PM Saikat Maitra > > wrote: > > > > > Hi Alexey, > > > > > > As discussed I have updated the wiki with agreed solution. > > > > > > Independent Integrations: > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations > > > > > > Discussion Links: > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-DiscussionLinks > > > > > > Tickets: > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-Tickets > > > > > > Please let me know if I can share more information. > > > > > > Regards, > > > Saikat > > > > > > > > > On Fri, Jan 17, 2020 at 9:16 PM Saikat Maitra > > > > wrote: > > > > > >> Hello Alexey, > > >> > > >> Thank you for your email. > > >> > > >> 1. Yes, we discussed in dev list and agreed on creating a new > repository > > >> for hosting our Ignite integrations. Please find the discussion thread > > >> below. I will update the wiki page as well and share updates. > > >> > > >> > > >> > > > http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSS-Proposal-for-Ignite-Extensions-as-a-separate-Bahir-module-or-Incubator-project-td44064.html > > >> > > >> 2. I was hoping to complete migration of the following modules before > we > > >> go ahead with release. I am tracking the jira story here > > >> https://issues.apache.org/jira/browse/IGNITE-12355 > > >> > > >>- Flink > > >>- Twitter > > >>- Storm > > >>- ZeroMQ > > >>- RocketMQ > > >>- Flume > > >>- MQTT > > >>- Camel > > >>- JMS > > >> > > >> 3. The dependencies for modules are pointing to latest snapshot of > > >> ignite project and if there are changes in ignite master branch then > > >> related affected Ignite extensions module also need to be modified. We > > will > > >> verify all the extensions for upcoming release but release only the > one > > >> that are impacted. We will plan to avoid publishing any extension > unless > > >> there are changes. Here is the discussion thread on release process: > > >> > > >> > > >> > > > http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSS-dependencies-and-release-process-for-Ignite-Extensions-td44478.html > > >> > > >> 4. Sounds good, we can maintain a compatibility matrix to ensure we > can > > >> share Ignite Core and related Ignite Extensions. I have seen something > > >> similar in Apache Beam and we can also create a similar wiki page to > > >> maintain Ignite Core and Ignite Extensions dependencies > > >> > > >> https://beam.apache.org/documentation/runners/capability-matrix/ > > >> > > >> Regards, > > >> Saikat > > >> > > >> > > >> > > >> On Fri, Jan 17, 2020 at 6:27 AM Alexey Goncharuk < > > >> alexey.goncha...@gmail.com> wrote: > > >> > > >>> Saikat, > > >>> > > >>> Code-wise the PR looks ok because it basically moves the module to > > >>> another > > >>> repo. I have some infrastructure/process questions though before we > > merge > > >>> the PRs > > >>> * I see that there are some objections from Alexey Zinoviev [1] on > > >>> whether > > >>> the streaming modules should be placed in extensions or in a separate > > >>> streaming project. Sorry if I missed the corresponding discussion on > > the > > >>> dev-list. If there is one, we need to edit the IEP page, if not, I > > think > > >>> we > > >>> should come to a consensus before making this change > > >>> * When are we planning to release the extensions? Are we going to > > >>> release > > >>>
Re: ML examples EvaluatorExample and MultipleMetricsExample looks the same
Sorry, for that, but I it touches ML-related stuff only and doesn't influence on any another module. I merged them to master later (currently, I have a 4 tickets in a queue). Will keep in mind for the next fixes вт, 11 февр. 2020 г. в 18:58, Maxim Muzafarov : > Alexey, > > Is the approach when we cherry-pick fixes to the 2.8 only after all > fixes have verified in the master branch better? I just want to be > sure the release branch to be stable enough. > > On Tue, 11 Feb 2020 at 18:36, Alexey Zinoviev > wrote: > > > > Fixed both in release branch 2.8. Please verify and let me know > > > > вт, 11 февр. 2020 г. в 14:58, Stepan Pilschikov < > pilshchikov@gmail.com>: > > > > > And one more > > > https://issues.apache.org/jira/browse/IGNITE-12658 - > > > TutorialStepByStepExample example failed if cluster with more then 1 > node > > > > > > вт, 11 февр. 2020 г. в 11:52, Alexey Zinoviev >: > > > > > >> Great, thank you so much, will fix next week > > >> > > >> вт, 11 февр. 2020 г., 11:39 Stepan Pilschikov < > pilshchikov@gmail.com > > >> >: > > >> > > >>> Hi, Alexey, could you please looking on one small accident happen in > ML > > >>> examples: > > >>> https://issues.apache.org/jira/browse/IGNITE-12657 - 2 examples > exactly > > >>> the same. > > >>> > > >>> Also want to say that all previous examples have one common approach > for > > >>> meaningful output, like ">>> Something important". In this two i > seeing > > >>> different one, could you please also pay attention to it. > > >>> > > >>> Best regards, > > >>> Stepan > > >>> > > >> >
[REVIEW] IGNITE-12344 Remote listener of IgniteMessaging has to run with appropriate SecurityContext
Hi, Igniters! I've prepared the PR[1] for the ticket[2], could somebody do a review? Thanks. 1. https://github.com/apache/ignite/pull/7338 2. https://issues.apache.org/jira/browse/IGNITE-12344
Re: ML examples EvaluatorExample and MultipleMetricsExample looks the same
Alexey, Is the approach when we cherry-pick fixes to the 2.8 only after all fixes have verified in the master branch better? I just want to be sure the release branch to be stable enough. On Tue, 11 Feb 2020 at 18:36, Alexey Zinoviev wrote: > > Fixed both in release branch 2.8. Please verify and let me know > > вт, 11 февр. 2020 г. в 14:58, Stepan Pilschikov : > > > And one more > > https://issues.apache.org/jira/browse/IGNITE-12658 - > > TutorialStepByStepExample example failed if cluster with more then 1 node > > > > вт, 11 февр. 2020 г. в 11:52, Alexey Zinoviev : > > > >> Great, thank you so much, will fix next week > >> > >> вт, 11 февр. 2020 г., 11:39 Stepan Pilschikov >> >: > >> > >>> Hi, Alexey, could you please looking on one small accident happen in ML > >>> examples: > >>> https://issues.apache.org/jira/browse/IGNITE-12657 - 2 examples exactly > >>> the same. > >>> > >>> Also want to say that all previous examples have one common approach for > >>> meaningful output, like ">>> Something important". In this two i seeing > >>> different one, could you please also pay attention to it. > >>> > >>> Best regards, > >>> Stepan > >>> > >>
Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]
All ML bugs are fixed, tested on TC and merged to 2.8 release branch. Hope I'm not a blocker man now. вт, 11 февр. 2020 г. в 17:23, Alexey Zinoviev : > Hello, Igniters > > Stepan found and reported bug related to lambda > serialization/deserialization > https://issues.apache.org/jira/browse/IGNITE-12657 > The problem is the next: the ParamGrid object has a lambda in interface > and this is an very bad for ML component, I've created a blocker bug > https://issues.apache.org/jira/browse/IGNITE-12660 > > Also as a result of good testing from both side (from me and Stepan) we > found a lot of bugs and CVEs in hadoop related components that should be > removed in release branch too. > > I'll notify then it will be finished > > Will work hardly on these bugs. > > > > вт, 11 февр. 2020 г. в 15:08, Ivan Bessonov : > >> Hello Igniters, >> >> I'd like to add one more fix to the release: [1] >> It adds versioning to internal classes of distributed metastorage >> component. >> Without this fix it would be much harder to update these classes without >> breaking binary compatibility. >> >> [1] https://issues.apache.org/jira/browse/IGNITE-12638 >> >> ср, 5 февр. 2020 г. в 22:33, Maxim Muzafarov : >> >> > Ivan, >> > >> > > Should not we state in release notes what new experimental API was >> added? >> > >> > I think we should. Will do. >> > Just not to miss anything that we should mark with >> > @IgniteExperimental: Consistency Check [1], Monitoring [2] anything >> > else? >> > >> > > As Flink integration was moved to external repository how Ignite 2.8 >> > users will be able to use that integration? >> > >> > Since ignite-extension has a separate release cycle (right?), it is >> > better to release ignite-extension rather than cherry-pick this change >> > back to 2.8. I also think it is not a blocker for the release, but we >> > should do our best make the first ignite-extension release as earlier >> > as possible. >> > >> > >> > [1] https://issues.apache.org/jira/browse/IGNITE-10663 >> > [2] https://issues.apache.org/jira/browse/IGNITE-11848 >> > >> > On Wed, 5 Feb 2020 at 22:07, Ivan Pavlukhin >> wrote: >> > > >> > > Maxim, >> > > >> > > A couple of questions: >> > > 1. We added an annotation to designate experimental API. Should not we >> > > state in release notes what new experimental API was added? Perhaps in >> > > a separate block. >> > > 2. As Flink integration was moved to external repository how Ignite >> > > 2.8 users will be able to use that integration? >> > > >> > > ср, 5 февр. 2020 г. в 21:21, Maxim Muzafarov : >> > > > >> > > > Igniters, >> > > > >> > > > >> > > > I've prepared RELEASE_NOTES pull-request [1] to the 2.8 release. >> > > > >> > > > Currently, IEP-35 monitoring issues are not included in this PR. >> Will >> > > > do it soon. >> > > > Please, take a look. >> > > > >> > > > >> > > > [1] https://github.com/apache/ignite/pull/7367/files >> > > > >> > > > On Mon, 3 Feb 2020 at 14:38, Maxim Muzafarov >> > wrote: >> > > > > >> > > > > Igniters, >> > > > > >> > > > > >> > > > > Let me share the current status of the release. >> > > > > >> > > > > 1. >> > > > > Waiting for the issues [1] [2] (discussed previously this thread) >> to >> > > > > be tested by TC.Bot and merged to the 2.8 release branch. >> > > > > >> > > > > 2. >> > > > > Only 2 release BLOCKER issues left. I'm planning to move these >> issues >> > > > > to 2.8.1 release. >> > > > > The issue [4] (Error during purges by expiration: Unknown page >> type) >> > > > > will be covered by [1] [2]. >> > > > > The issue [3] (Apache Ignite Cluster(Amazon S3 Based Discovery) >> Nodes >> > > > > getting down) probably require additional info to reproduce the >> > issue. >> > > > > >> > > > > 3. >> > > > > A potential performance drop on `putAll` operations on an >> in-memory >> > > > > cluster (see [5] for details). >> > > > > I'll try to reproduce in another test environment. >> > > > > >> > > > > >> > > > > Will keep you posted. >> > > > > >> > > > > >> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-12593 >> > > > > [2] https://issues.apache.org/jira/browse/IGNITE-12594 >> > > > > [3] https://issues.apache.org/jira/browse/IGNITE-12398 >> > > > > [4] https://issues.apache.org/jira/browse/IGNITE-12489 >> > > > > [5] >> > >> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8#ApacheIgnite2.8-Benchmarks(LATEST) >> > > > > >> > > > > On Thu, 30 Jan 2020 at 15:02, Alexey Goncharuk >> > > > > wrote: >> > > > > > >> > > > > > Sounds good, will do! >> > > >> > > >> > > >> > > -- >> > > Best regards, >> > > Ivan Pavlukhin >> > >> >> >> -- >> Sincerely yours, >> Ivan Bessonov >> >
Re: ML examples EvaluatorExample and MultipleMetricsExample looks the same
Fixed both in release branch 2.8. Please verify and let me know вт, 11 февр. 2020 г. в 14:58, Stepan Pilschikov : > And one more > https://issues.apache.org/jira/browse/IGNITE-12658 - > TutorialStepByStepExample example failed if cluster with more then 1 node > > вт, 11 февр. 2020 г. в 11:52, Alexey Zinoviev : > >> Great, thank you so much, will fix next week >> >> вт, 11 февр. 2020 г., 11:39 Stepan Pilschikov > >: >> >>> Hi, Alexey, could you please looking on one small accident happen in ML >>> examples: >>> https://issues.apache.org/jira/browse/IGNITE-12657 - 2 examples exactly >>> the same. >>> >>> Also want to say that all previous examples have one common approach for >>> meaningful output, like ">>> Something important". In this two i seeing >>> different one, could you please also pay attention to it. >>> >>> Best regards, >>> Stepan >>> >>
[jira] [Created] (IGNITE-12661) Fix GridIoManagerSelfTest
Ilya Kasnacheev created IGNITE-12661: Summary: Fix GridIoManagerSelfTest Key: IGNITE-12661 URL: https://issues.apache.org/jira/browse/IGNITE-12661 Project: Ignite Issue Type: Test Components: cache Reporter: Ilya Kasnacheev It is an old test which tries to create some Processors and stich them together, which just does no longer work. We need to rewrite this test, I think. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]
Hello, Igniters Stepan found and reported bug related to lambda serialization/deserialization https://issues.apache.org/jira/browse/IGNITE-12657 The problem is the next: the ParamGrid object has a lambda in interface and this is an very bad for ML component, I've created a blocker bug https://issues.apache.org/jira/browse/IGNITE-12660 Also as a result of good testing from both side (from me and Stepan) we found a lot of bugs and CVEs in hadoop related components that should be removed in release branch too. I'll notify then it will be finished Will work hardly on these bugs. вт, 11 февр. 2020 г. в 15:08, Ivan Bessonov : > Hello Igniters, > > I'd like to add one more fix to the release: [1] > It adds versioning to internal classes of distributed metastorage > component. > Without this fix it would be much harder to update these classes without > breaking binary compatibility. > > [1] https://issues.apache.org/jira/browse/IGNITE-12638 > > ср, 5 февр. 2020 г. в 22:33, Maxim Muzafarov : > > > Ivan, > > > > > Should not we state in release notes what new experimental API was > added? > > > > I think we should. Will do. > > Just not to miss anything that we should mark with > > @IgniteExperimental: Consistency Check [1], Monitoring [2] anything > > else? > > > > > As Flink integration was moved to external repository how Ignite 2.8 > > users will be able to use that integration? > > > > Since ignite-extension has a separate release cycle (right?), it is > > better to release ignite-extension rather than cherry-pick this change > > back to 2.8. I also think it is not a blocker for the release, but we > > should do our best make the first ignite-extension release as earlier > > as possible. > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-10663 > > [2] https://issues.apache.org/jira/browse/IGNITE-11848 > > > > On Wed, 5 Feb 2020 at 22:07, Ivan Pavlukhin wrote: > > > > > > Maxim, > > > > > > A couple of questions: > > > 1. We added an annotation to designate experimental API. Should not we > > > state in release notes what new experimental API was added? Perhaps in > > > a separate block. > > > 2. As Flink integration was moved to external repository how Ignite > > > 2.8 users will be able to use that integration? > > > > > > ср, 5 февр. 2020 г. в 21:21, Maxim Muzafarov : > > > > > > > > Igniters, > > > > > > > > > > > > I've prepared RELEASE_NOTES pull-request [1] to the 2.8 release. > > > > > > > > Currently, IEP-35 monitoring issues are not included in this PR. Will > > > > do it soon. > > > > Please, take a look. > > > > > > > > > > > > [1] https://github.com/apache/ignite/pull/7367/files > > > > > > > > On Mon, 3 Feb 2020 at 14:38, Maxim Muzafarov > > wrote: > > > > > > > > > > Igniters, > > > > > > > > > > > > > > > Let me share the current status of the release. > > > > > > > > > > 1. > > > > > Waiting for the issues [1] [2] (discussed previously this thread) > to > > > > > be tested by TC.Bot and merged to the 2.8 release branch. > > > > > > > > > > 2. > > > > > Only 2 release BLOCKER issues left. I'm planning to move these > issues > > > > > to 2.8.1 release. > > > > > The issue [4] (Error during purges by expiration: Unknown page > type) > > > > > will be covered by [1] [2]. > > > > > The issue [3] (Apache Ignite Cluster(Amazon S3 Based Discovery) > Nodes > > > > > getting down) probably require additional info to reproduce the > > issue. > > > > > > > > > > 3. > > > > > A potential performance drop on `putAll` operations on an in-memory > > > > > cluster (see [5] for details). > > > > > I'll try to reproduce in another test environment. > > > > > > > > > > > > > > > Will keep you posted. > > > > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-12593 > > > > > [2] https://issues.apache.org/jira/browse/IGNITE-12594 > > > > > [3] https://issues.apache.org/jira/browse/IGNITE-12398 > > > > > [4] https://issues.apache.org/jira/browse/IGNITE-12489 > > > > > [5] > > > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8#ApacheIgnite2.8-Benchmarks(LATEST) > > > > > > > > > > On Thu, 30 Jan 2020 at 15:02, Alexey Goncharuk > > > > > wrote: > > > > > > > > > > > > Sounds good, will do! > > > > > > > > > > > > -- > > > Best regards, > > > Ivan Pavlukhin > > > > > -- > Sincerely yours, > Ivan Bessonov >
Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]
Igniters, Current the 2.8 release status 1. The PR with RELEASE_NOTES fully updated [1]. 2. Previously mentioned performance drop has not been confirmed. Run many times in different environments. All test results within the margin of error. In-memory, putAll, 4 nodes, 1 client IgnitePutAllBenchmark: +1% IgnitePutAllTxBenchmark: -6% 3. Waiting for the vote completion (Allow or prohibit a joint use of @deprecated and @IgniteExperimental) 4. Mark MVCC with IgniteExperimental [2]. 5. Wait for ML examples to be fixed [3]. [1] https://github.com/apache/ignite/pull/7367/files [2] http://apache-ignite-developers.2346864.n4.nabble.com/Mark-MVCC-with-IgniteExperimental-td45669.html [3] https://issues.apache.org/jira/browse/IGNITE-12657 On Tue, 11 Feb 2020 at 15:08, Ivan Bessonov wrote: > > Hello Igniters, > > I'd like to add one more fix to the release: [1] > It adds versioning to internal classes of distributed metastorage component. > Without this fix it would be much harder to update these classes without > breaking binary compatibility. > > [1] https://issues.apache.org/jira/browse/IGNITE-12638 > > ср, 5 февр. 2020 г. в 22:33, Maxim Muzafarov : > > > Ivan, > > > > > Should not we state in release notes what new experimental API was added? > > > > I think we should. Will do. > > Just not to miss anything that we should mark with > > @IgniteExperimental: Consistency Check [1], Monitoring [2] anything > > else? > > > > > As Flink integration was moved to external repository how Ignite 2.8 > > users will be able to use that integration? > > > > Since ignite-extension has a separate release cycle (right?), it is > > better to release ignite-extension rather than cherry-pick this change > > back to 2.8. I also think it is not a blocker for the release, but we > > should do our best make the first ignite-extension release as earlier > > as possible. > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-10663 > > [2] https://issues.apache.org/jira/browse/IGNITE-11848 > > > > On Wed, 5 Feb 2020 at 22:07, Ivan Pavlukhin wrote: > > > > > > Maxim, > > > > > > A couple of questions: > > > 1. We added an annotation to designate experimental API. Should not we > > > state in release notes what new experimental API was added? Perhaps in > > > a separate block. > > > 2. As Flink integration was moved to external repository how Ignite > > > 2.8 users will be able to use that integration? > > > > > > ср, 5 февр. 2020 г. в 21:21, Maxim Muzafarov : > > > > > > > > Igniters, > > > > > > > > > > > > I've prepared RELEASE_NOTES pull-request [1] to the 2.8 release. > > > > > > > > Currently, IEP-35 monitoring issues are not included in this PR. Will > > > > do it soon. > > > > Please, take a look. > > > > > > > > > > > > [1] https://github.com/apache/ignite/pull/7367/files > > > > > > > > On Mon, 3 Feb 2020 at 14:38, Maxim Muzafarov > > wrote: > > > > > > > > > > Igniters, > > > > > > > > > > > > > > > Let me share the current status of the release. > > > > > > > > > > 1. > > > > > Waiting for the issues [1] [2] (discussed previously this thread) to > > > > > be tested by TC.Bot and merged to the 2.8 release branch. > > > > > > > > > > 2. > > > > > Only 2 release BLOCKER issues left. I'm planning to move these issues > > > > > to 2.8.1 release. > > > > > The issue [4] (Error during purges by expiration: Unknown page type) > > > > > will be covered by [1] [2]. > > > > > The issue [3] (Apache Ignite Cluster(Amazon S3 Based Discovery) Nodes > > > > > getting down) probably require additional info to reproduce the > > issue. > > > > > > > > > > 3. > > > > > A potential performance drop on `putAll` operations on an in-memory > > > > > cluster (see [5] for details). > > > > > I'll try to reproduce in another test environment. > > > > > > > > > > > > > > > Will keep you posted. > > > > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-12593 > > > > > [2] https://issues.apache.org/jira/browse/IGNITE-12594 > > > > > [3] https://issues.apache.org/jira/browse/IGNITE-12398 > > > > > [4] https://issues.apache.org/jira/browse/IGNITE-12489 > > > > > [5] > > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8#ApacheIgnite2.8-Benchmarks(LATEST) > > > > > > > > > > On Thu, 30 Jan 2020 at 15:02, Alexey Goncharuk > > > > > wrote: > > > > > > > > > > > > Sounds good, will do! > > > > > > > > > > > > -- > > > Best regards, > > > Ivan Pavlukhin > > > > > -- > Sincerely yours, > Ivan Bessonov
[jira] [Created] (IGNITE-12660) [ML] The ParamGrid uses unserialized lambdas in interface to get an access to the trainer fields
Alexey Zinoviev created IGNITE-12660: Summary: [ML] The ParamGrid uses unserialized lambdas in interface to get an access to the trainer fields Key: IGNITE-12660 URL: https://issues.apache.org/jira/browse/IGNITE-12660 Project: Ignite Issue Type: Bug Reporter: Alexey Zinoviev Assignee: Alexey Zinoviev Fix For: 2.8 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12659) [ML] Remove broken sub-modules in ML
Alexey Zinoviev created IGNITE-12659: Summary: [ML] Remove broken sub-modules in ML Key: IGNITE-12659 URL: https://issues.apache.org/jira/browse/IGNITE-12659 Project: Ignite Issue Type: Bug Components: ml Reporter: Alexey Zinoviev Assignee: Alexey Zinoviev Fix For: 2.8 After testing phase we investigated that the next sub-modules are broken and should be removed from the release branch 2.8 * tensorflow (it uses deprecated IGFS and current solution in tf.contrib could n't connect to Ignite now) * tensorflow-ml-parser (it has a lot of bugs and doesn't work with tf 2.0) * mleap-ml-parser (CVE, Ignite ML couldn't work with metadata) also due to connect with Hadoop, Spark and another libraries it has a few CVE related to the old dependencies in Hadoop libraries P.S. I suppose all these solutions should be moved to the separate project ignite-ml-extensions for the independent developement -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]
Hello Igniters, I'd like to add one more fix to the release: [1] It adds versioning to internal classes of distributed metastorage component. Without this fix it would be much harder to update these classes without breaking binary compatibility. [1] https://issues.apache.org/jira/browse/IGNITE-12638 ср, 5 февр. 2020 г. в 22:33, Maxim Muzafarov : > Ivan, > > > Should not we state in release notes what new experimental API was added? > > I think we should. Will do. > Just not to miss anything that we should mark with > @IgniteExperimental: Consistency Check [1], Monitoring [2] anything > else? > > > As Flink integration was moved to external repository how Ignite 2.8 > users will be able to use that integration? > > Since ignite-extension has a separate release cycle (right?), it is > better to release ignite-extension rather than cherry-pick this change > back to 2.8. I also think it is not a blocker for the release, but we > should do our best make the first ignite-extension release as earlier > as possible. > > > [1] https://issues.apache.org/jira/browse/IGNITE-10663 > [2] https://issues.apache.org/jira/browse/IGNITE-11848 > > On Wed, 5 Feb 2020 at 22:07, Ivan Pavlukhin wrote: > > > > Maxim, > > > > A couple of questions: > > 1. We added an annotation to designate experimental API. Should not we > > state in release notes what new experimental API was added? Perhaps in > > a separate block. > > 2. As Flink integration was moved to external repository how Ignite > > 2.8 users will be able to use that integration? > > > > ср, 5 февр. 2020 г. в 21:21, Maxim Muzafarov : > > > > > > Igniters, > > > > > > > > > I've prepared RELEASE_NOTES pull-request [1] to the 2.8 release. > > > > > > Currently, IEP-35 monitoring issues are not included in this PR. Will > > > do it soon. > > > Please, take a look. > > > > > > > > > [1] https://github.com/apache/ignite/pull/7367/files > > > > > > On Mon, 3 Feb 2020 at 14:38, Maxim Muzafarov > wrote: > > > > > > > > Igniters, > > > > > > > > > > > > Let me share the current status of the release. > > > > > > > > 1. > > > > Waiting for the issues [1] [2] (discussed previously this thread) to > > > > be tested by TC.Bot and merged to the 2.8 release branch. > > > > > > > > 2. > > > > Only 2 release BLOCKER issues left. I'm planning to move these issues > > > > to 2.8.1 release. > > > > The issue [4] (Error during purges by expiration: Unknown page type) > > > > will be covered by [1] [2]. > > > > The issue [3] (Apache Ignite Cluster(Amazon S3 Based Discovery) Nodes > > > > getting down) probably require additional info to reproduce the > issue. > > > > > > > > 3. > > > > A potential performance drop on `putAll` operations on an in-memory > > > > cluster (see [5] for details). > > > > I'll try to reproduce in another test environment. > > > > > > > > > > > > Will keep you posted. > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-12593 > > > > [2] https://issues.apache.org/jira/browse/IGNITE-12594 > > > > [3] https://issues.apache.org/jira/browse/IGNITE-12398 > > > > [4] https://issues.apache.org/jira/browse/IGNITE-12489 > > > > [5] > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8#ApacheIgnite2.8-Benchmarks(LATEST) > > > > > > > > On Thu, 30 Jan 2020 at 15:02, Alexey Goncharuk > > > > wrote: > > > > > > > > > > Sounds good, will do! > > > > > > > > -- > > Best regards, > > Ivan Pavlukhin > -- Sincerely yours, Ivan Bessonov
Re: ML examples EvaluatorExample and MultipleMetricsExample looks the same
And one more https://issues.apache.org/jira/browse/IGNITE-12658 - TutorialStepByStepExample example failed if cluster with more then 1 node вт, 11 февр. 2020 г. в 11:52, Alexey Zinoviev : > Great, thank you so much, will fix next week > > вт, 11 февр. 2020 г., 11:39 Stepan Pilschikov : > >> Hi, Alexey, could you please looking on one small accident happen in ML >> examples: >> https://issues.apache.org/jira/browse/IGNITE-12657 - 2 examples exactly >> the same. >> >> Also want to say that all previous examples have one common approach for >> meaningful output, like ">>> Something important". In this two i seeing >> different one, could you please also pay attention to it. >> >> Best regards, >> Stepan >> >
Re: [VOTE] Allow or prohibit a joint use of @deprecated and @IgniteExperimental
-1 Prohibit On Mon, Feb 10, 2020 at 11:02 AM Alexey Goncharuk wrote: > > Dear Apache Ignite community, > > We would like to conduct a formal vote on the subject of whether to allow > or prohibit a joint existence of @deprecated annotation for an old API > and @IgniteExperimental [1] for a new (replacement) API. The result of this > vote will be formalized as an Apache Ignite development rule to be used in > future. > > The discussion thread where you can address all non-vote messages is [2]. > > The votes are: > *[+1 Allow]* Allow to deprecate the old APIs even when new APIs are marked > with @IgniteExperimental to explicitly notify users that an old APIs will > be removed in the next major release AND new APIs are available. > *[-1 Prohibit]* Never deprecate the old APIs unless the new APIs are stable > and released without @IgniteExperimental. The old APIs javadoc may be > updated with a reference to new APIs to encourage users to evaluate new > APIs. The deprecation and new API release may happen simultaneously if the > new API is not marked with @IgniteExperimental or the annotation is removed > in the same release. > > Neither of the choices prohibits deprecation of an API without a > replacement if community decides so. > > The vote will hold for 72 hours and will end on February 13th 2020 08:00 > UTC: > https://www.timeanddate.com/countdown/to?year=2020=2=13=8=0=0=utc-1 > > All votes count, there is no binding/non-binding status for this. > > [1] > https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/lang/IgniteExperimental.java > [2] > http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSS-Public-API-deprecation-rules-td45647.html > > Thanks, > --AG
[jira] [Created] (IGNITE-12658) [ML][Examples] TutorialStepByStepExample failed on cluster with more then 1 node
Stepan Pilschikov created IGNITE-12658:
--
Summary: [ML][Examples] TutorialStepByStepExample failed on
cluster with more then 1 node
Key: IGNITE-12658
URL: https://issues.apache.org/jira/browse/IGNITE-12658
Project: Ignite
Issue Type: Bug
Components: examples, ml
Affects Versions: 2.8
Environment: Ubuntu/Win
Java 8
Reporter: Stepan Pilschikov
Steps to reproduce:
1. Run Ignite node with org.apache.ignite.examples.ExampleNodeStartup (1 node
will be enough)
2. Run org.apache.ignite.examples.ml.tutorial.TutorialStepByStepExample
Actual:
On Step_8_CV_with_Param_Grid_and_metrics starting to throw a lot of exceptions
{code:java}
Train with p: 2 and maxDeep: 1
>>> Trained model: if (x1 > 0.4368) then return 1. else return 0.
>>> Accuracy 0.7679083094555874
>>> Test Error 0.2320916905444126
>>> Tutorial step 8 (cross-validation) example completed.
[13:25:40] Ignite node stopped OK [uptime=00:00:17.453]
>>> Tutorial step 8 (cross-validation with param grid) example started.
[13:25:40]__
[13:25:40] / _/ ___/ |/ / _/_ __/ __/
[13:25:40] _/ // (7 7// / / / / _/
[13:25:40] /___/\___/_/|_/___/ /_/ /___/
[13:25:40]
[13:25:40] ver. 2.8.0#20200130-sha1:f478aa56
[13:25:40] 2020 Copyright(C) Apache Software Foundation
[13:25:40]
[13:25:40] Ignite documentation: http://ignite.apache.org
[13:25:40]
[13:25:40] Quiet mode.
[13:25:40] ^-- Logging to file
'/opt/buildagent/work/d501ae8146bd8253/i2test/var/suite-examples/app-ignite/work/log/ignite-e156b2f2.log'
[13:25:40] ^-- Logging by 'Log4JLogger [quiet=true, config=null]'
[13:25:40] ^-- To see **FULL** console log here add -DIGNITE_QUIET=false or
"-v" to ignite.{sh|bat}
[13:25:40]
[13:25:40] OS: Linux 4.15.0-65-generic amd64
[13:25:40] VM information: Java(TM) SE Runtime Environment 1.8.0_221-b11 Oracle
Corporation Java HotSpot(TM) 64-Bit Server VM 25.221-b11
[13:25:40] Please set system property '-Djava.net.preferIPv4Stack=true' to
avoid possible problems in mixed environments.
[13:25:40] Configured plugins:
[13:25:40] ^-- ml-inference-plugin 1.0.0
[13:25:40] ^-- null
[13:25:40]
[13:25:40] Configured failure handler: [hnd=StopNodeOrHaltFailureHandler
[tryStop=false, timeout=0, super=AbstractFailureHandler
[ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED,
SYSTEM_CRITICAL_OPERATION_TIMEOUT
[13:25:40] Message queue limit is set to 0 which may lead to potential OOMEs
when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to
message queues growth on sender and receiver sides.
[13:25:40] Security status [authentication=off, tls/ssl=off]
[13:25:41] Performance suggestions for grid (fix if possible)
[13:25:41] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
[13:25:41] ^-- Disable grid events (remove 'includeEventTypes' from
configuration)
[13:25:41] ^-- Enable G1 Garbage Collector (add '-XX:+UseG1GC' to JVM options)
[13:25:41] ^-- Set max direct memory size if getting 'OOME: Direct buffer
memory' (add '-XX:MaxDirectMemorySize=[g|G|m|M|k|K]' to JVM options)
[13:25:41] ^-- Disable processing of calls to System.gc() (add
'-XX:+DisableExplicitGC' to JVM options)
[13:25:41] Refer to this page for more performance suggestions:
https://apacheignite.readme.io/docs/jvm-and-system-tuning
[13:25:41]
[13:25:41] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat}
[13:25:41] Data Regions Configured:
[13:25:41] ^-- Default_Region [initSize=500.0 MiB, maxSize=18.9 GiB,
persistence=false, lazyMemoryAllocation=true]
[13:25:41]
[13:25:41] Ignite node started OK (id=e156b2f2)
[13:25:41] Topology snapshot [ver=20, locNode=e156b2f2, servers=2, clients=0,
state=ACTIVE, CPUs=5, offheap=38.0GB, heap=3.0GB]
[13:25:41] ^-- Baseline [id=0, size=2, online=2, offline=0]
[2020-02-11 13:25:42,428][ERROR][sys-#593][GridTaskWorker] Failed to obtain
remote job result policy for result from ComputeTask.result(..) method (will
fail the whole task): GridJobResultImpl [job=C2
[c=o.a.i.ml.dataset.impl.cache.util.ComputeUtils$DeployableCallable@30e27659],
sib=GridJobSiblingImpl [sesId=f9aced33071-e156b2f2-d116-4389-bd43-8536dc59,
jobId=1aaced33071-e156b2f2-d116-4389-bd43-8536dc59,
nodeId=f1135598-73c8-4324-92af-45c110a6a637, isJobDone=false],
jobCtx=GridJobContextImpl
[jobId=1aaced33071-e156b2f2-d116-4389-bd43-8536dc59, timeoutObj=null,
attrs=HashMap {}], node=TcpDiscoveryNode
[id=f1135598-73c8-4324-92af-45c110a6a637,
consistentId=127.0.0.1,192.168.112.2:47500, addrs=ArrayList [127.0.0.1,
192.168.112.2], sockAddrs=HashSet [/127.0.0.1:47500,
87bf29244900/192.168.112.2:47500], discPort=47500, order=1, intOrder=1,
lastExchangeTime=1581416741501, loc=false, ver=2.8.0#20200130-sha1:f478aa56,
isClient=false], ex=class o.a.i.IgniteException: Failed to deserialize object
Re: ML examples EvaluatorExample and MultipleMetricsExample looks the same
Great, thank you so much, will fix next week вт, 11 февр. 2020 г., 11:39 Stepan Pilschikov : > Hi, Alexey, could you please looking on one small accident happen in ML > examples: > https://issues.apache.org/jira/browse/IGNITE-12657 - 2 examples exactly > the same. > > Also want to say that all previous examples have one common approach for > meaningful output, like ">>> Something important". In this two i seeing > different one, could you please also pay attention to it. > > Best regards, > Stepan >
Re: [jira] [Created] (IGNITE-12657) ML examples EvaluatorExample and MultipleMetricsExample looks the same
Thank you so much, will fix it next week вт, 11 февр. 2020 г., 11:33 Stepan Pilschikov (Jira) : > Stepan Pilschikov created IGNITE-12657: > -- > > Summary: ML examples EvaluatorExample and > MultipleMetricsExample looks the same > Key: IGNITE-12657 > URL: https://issues.apache.org/jira/browse/IGNITE-12657 > Project: Ignite > Issue Type: Bug > Components: examples, ml > Affects Versions: 2.8 > Reporter: Stepan Pilschikov > > > Examples > org.apache.ignite.examples.ml.selection.scoring.EvaluatorExample > and > org.apache.ignite.examples.ml.selection.scoring.MultipleMetricsExample > Looks exactly the same > I think MultipleMetricsExample is wrong because description told about > using KNNClassificationTrainer but actually used > SVMLinearClassificationTrainer > > > > > -- > This message was sent by Atlassian Jira > (v8.3.4#803005) >
ML examples EvaluatorExample and MultipleMetricsExample looks the same
Hi, Alexey, could you please looking on one small accident happen in ML examples: https://issues.apache.org/jira/browse/IGNITE-12657 - 2 examples exactly the same. Also want to say that all previous examples have one common approach for meaningful output, like ">>> Something important". In this two i seeing different one, could you please also pay attention to it. Best regards, Stepan
[jira] [Created] (IGNITE-12657) ML examples EvaluatorExample and MultipleMetricsExample looks the same
Stepan Pilschikov created IGNITE-12657: -- Summary: ML examples EvaluatorExample and MultipleMetricsExample looks the same Key: IGNITE-12657 URL: https://issues.apache.org/jira/browse/IGNITE-12657 Project: Ignite Issue Type: Bug Components: examples, ml Affects Versions: 2.8 Reporter: Stepan Pilschikov Examples org.apache.ignite.examples.ml.selection.scoring.EvaluatorExample and org.apache.ignite.examples.ml.selection.scoring.MultipleMetricsExample Looks exactly the same I think MultipleMetricsExample is wrong because description told about using KNNClassificationTrainer but actually used SVMLinearClassificationTrainer -- This message was sent by Atlassian Jira (v8.3.4#803005)
