[ML] IGNITE-9282 Naive Bayes task split

[Ravil Galeyev]

Hi Team,
I work on implementing Naive Bayes classifiers.

Withing IGNITE-9282  I
implemented a Gaussian Bayes and created a PR
https://github.com/apache/ignite/pull/4869

But  there are already a lot of changes
That's' why I'd like to create separate tasks for
multinomial and Bernoulli Bayes classifiers and continue work.

Any objections?

Best regards,
Ravil

[GitHub] ignite pull request #4875: IGNITE-9742 Illegal change of DiscoveryCustomEven...

[daradurvs]

GitHub user daradurvs opened a pull request:

https://github.com/apache/ignite/pull/4875

IGNITE-9742 Illegal change of DiscoveryCustomEvent in 
GridDhtPartitionsExchangeFuture#onDone



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/daradurvs/ignite ignite-9742

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4875.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4875


commit eabe386acef14647e92ff36d2ffeb0b1ad642df2
Author: Vyacheslav Daradur 
Date:   2018-09-29T21:07:04Z

fixed




---

[jira] [Created] (IGNITE-9742) Illegal change of DiscoveryCustomEvent in GridDhtPartitionsExchangeFuture#onDone

[Vyacheslav Daradur (JIRA)]

Vyacheslav Daradur created IGNITE-9742:
--

 Summary: Illegal change of DiscoveryCustomEvent in 
GridDhtPartitionsExchangeFuture#onDone
 Key: IGNITE-9742
 URL: https://issues.apache.org/jira/browse/IGNITE-9742
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.6
Reporter: Vyacheslav Daradur
Assignee: Vyacheslav Daradur
 Fix For: 2.8


The following code is called in {{GridDhtPartitionsExchangeFuture#onDone}}:
{code}
((DiscoveryCustomEvent)firstDiscoEvt).customMessage(null);
{code}

That means a global instance of 'DiscoveryCustomEvent' is being
mutated outside discovery-spi infrastructure. It also means that
discovery listeners receive 'DiscoveryCustomEvent' with 'null' field
instead of 'CustomMessage' which they may rely on.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Danger (?) change of DiscoveryCustomEvent in GridDhtPartitionsExchangeFuture#onDone

[Vyacheslav Daradur]

Hi Igniters!

I think I found an illegal behavior in
GridDhtPartitionsExchangeFuture#onDone, the following code is called
here:
((DiscoveryCustomEvent)firstDiscoEvt).customMessage(null);

That means a global instance of 'DiscoveryCustomEvent' is being
mutated outside discovery-spi infrastructure. It also means that
discovery listeners receive 'DiscoveryCustomEvent' with 'null' field
instead of 'CustomMessage' which they may rely on.

Could someone confirm if it is wrong behavior and should be fixed?

-- 
Best Regards, Vyacheslav D.

Re: [QUESTION] SegmentAware holder initialization (fix of WAL blocked rollOver)

[Maxim Muzafarov]

Anton,

Thank you for your reply and clarification.

I've created a ticket [1] for this fix. Can you take it, please?

Probably, I have the solution for the issue described above and at the
first the glance it looks not too bad.
But I can't guarantee the completeness of my fix and the fact that I took
all the details into account. As an
author of the IGNITE-8559 commit, you definitely know all details better
rather than me.

I've marked the issue to be included in the 2.7 release, but it also
discussable.


[1] https://issues.apache.org/jira/browse/IGNITE-9741

On Fri, 28 Sep 2018 at 19:53 Anton Kalashnikov  wrote:

> Hello.
>
> Looks like you are right about interrupted flag. I don't understand why
> tests was pass successfully in master. Your solution  also looks correct.
>
> SegmentAware - it's info object of actual state of WAL segments like
> "current work segment", "last archived segment" etc. Actually it's nothing
> new here. It just was extracted from FileWriteAheadLogManager for
> centralization and future simplification. It should not change a old
> behaviour if it was happened it  will need to be investigate.
>
> If you already have fix for it, please will create task and add me to the
> watchers. Or I can fix it by myself.
>
> --
> Best regards,
> Anton Kalashnikov
>
>
> 28.09.2018, 16:06, "Maxim Muzafarov" :
> > Igniters,
> >
> > I would like to discuss this question here and create a separate topic
> for
> > it. Previously, I've posted some comments on the probable issue in Apache
> > Ignite 2.7 topic [2]. My question is related to the IGNITE-8559 [3]
> (commit
> > [4]). We've added a new SegmentAware class and change the
> > FileWriteAheadLogManager behaviour.
> >
> > The FileWriteAheadLogManager now contains:
> >
> > `private final SegmentAware segmentAware;`.
> >
> > The SegmentAware have the `interrupt()` method which at manager
> > de-activation sets (e.g. for SegmentArchivedStorage) `interrupted` filed
> to
> > `true` value [5] but never revert it to `false` after activation. So, the
> > SegmentArchivedStorage after de-activation always remain interrupted.
> >
> > I think it can lead us to unpredictable issues with multiple cluster
> > activation/de-activation. I don't know why all tests in master branch
> > suppose to be successful but in my local branch, they hang on exchange
> > future get() method.
> >
> > My local solution (probably not ideal) is:
> > 1) make it volatile - `private volatile SegmentAware segmentAware;`
> > 2) move field init to the `start0()` method of FileWriteAheadLogManager;
> >
> > With these changes, everything begins to work fine but I can miss
> something
> > because I don’t understand the whole this change well enough.
> >
> > Can anyone comment on this and share details IGNITE-8559 implementation?
> >
> > [1]
> >
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/FileWriteAheadLogManager.java#L319
> > [2]
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/Apache-Ignite-2-7-release-tp34076p35863.html
> > [3] https://issues.apache.org/jira/browse/IGNITE-8559
> > [4]
> >
> https://github.com/apache/ignite/commit/2f72fe758d4256c4eb4610e5922ad3d174b43dc5
> > [5]
> >
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/aware/SegmentArchivedStorage.java#L117
> > --
> > --
> > Maxim Muzafarov
>
-- 
--
Maxim Muzafarov

[jira] [Created] (IGNITE-9741) SegmentArchivedStorage and SegmentCompressStorage remain `iterrupted` after de-activation occurs before activation

[Maxim Muzafarov (JIRA)]

Maxim Muzafarov created IGNITE-9741:
---

 Summary: SegmentArchivedStorage and SegmentCompressStorage remain 
`iterrupted` after de-activation occurs before activation
 Key: IGNITE-9741
 URL: https://issues.apache.org/jira/browse/IGNITE-9741
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.6
Reporter: Maxim Muzafarov
 Fix For: 2.7


The {{FileWriteAheadLogManager}} now contains:
 {{private final SegmentAware segmentAware;}}

 

The SegmentAware have the `interrupt()` method:
{code:java|title=SegmentAware:216}
/**
 * Interrupt waiting on related objects.
 */
public void interrupt() {
segmentArchivedStorage.interrupt();

segmentCompressStorage.interrupt();

segmentCurrStateStorage.interrupt();
}
{code}
 

Method at the {{FileWriteAheadLogManager}} de-activation sets (e.g. for 
SegmentArchivedStorage) `interrupted` filed to `true` value but never revert it 
to `false` after activation.
{code:java|title=SegmentArchivedStorage:114}
/**
 * Interrupt waiting on this object.
 */
synchronized void interrupt() {
interrupted = true;

notifyAll();
}
{code}
 

So, the SegmentArchivedStorage after de-activation always remain interrupted. 
 This can lead to undefined behaviour e.g. exchange worker hangs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Published articles on Data streaming in Dzone and Tech@Target

[Saikat Maitra]

Hi,

We have published an article on Data streaming at Dzone. Thank you Prachi
for help for publishing the article

https://dzone.com/articles/data-streaming-using-apache-flink-and-apache-ignit

Also wrote an post at Tech@Target about how I started with Apache Ignite

https://tech.target.com/2018/09/14/data-streaming-using-apache-flink-and-apache-ignite.html

Please let me know if you have any feedback.

Regards,
Saikat

Re: Apache Ignite 2.7 release

[Vladimir Ozerov]

I fully support Dmitry's concerns. I do not see a single chance for Service
Grid to be included into 2.7 scope, but we are still keeping hope alive.
What for? Our experience shows, that any feature involving consensus
algorithms typically take weeks to be finalized after initial public
review. Yes, it is possible that at some day in October we will see it
completed, but what about testing? What about regressions with other
components? Let me reming you that from Monday a lot of efforts will be put
into project stabilization. This is the sole purpose of that pre-release
phase. Accepting a patch with deep rework of Ignite internals in the middle
of that process, means that our effrots will be lost. This is simply a
matter of respect to contributor's time.

Another problem is that this false hope puts us at rush. Rush during
design, development, review, testing. Result is known - bad features, which
makes damage to the project.

So my question is - why don't we just want to move it to AI 2.8 right now?
Feature is big, feature is very far from being ready. This simple action
immediately shifts the focus from dates to quality of the product, and
remove any risks that potential merge will defeat stabilization effrots of
other contributors.



On Sat, Sep 29, 2018 at 8:32 AM Vyacheslav Daradur 
wrote:

> Dmitriy,
>
> Hot redeployment and versioning will not be implemented in phase 1,
> but it is scheduled once it is finished.
>
> Here is an umbrella ticket [1] to track phase 1 scope.
>
> It includes very few new features, but we completely rework component
> to improve guarantees to be more reliable and we build the base for
> new features.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-9607
> On Fri, Sep 28, 2018 at 9:38 PM Dmitriy Setrakyan 
> wrote:
> >
> > I am not sure I can agree. SG redesign includes:
> > - hot redeployment
> > - versioning
> >
> > In my experience, features like these take about 1 month to test properly
> > and fix all the bugs, including redeployment tests and restart tests on
> > larger topologies, together with overnight runs. If this type of testing
> > has not been performed, I think it would be unreasonable to expect this
> > feature making it into the release.
> >
> > Can someone comment on the testing?
> >
> > D.
> >
> >
> > On Fri, Sep 28, 2018 at 10:38 AM Dmitriy Pavlov 
> > wrote:
> >
> > > Nikolay, because I think you're a do'er, but not a commenter, like me,
> for
> > > example, I can trust your opinion.
> > >
> > > I will join review if I have spare cycles.
> > >
> > > пт, 28 сент. 2018 г. в 20:34, Denis Magda :
> > >
> > > > Nikolay,
> > > >
> > > > Thanks for stepping in and giving more context. In general, I'm
> fully for
> > > > your proposal below:
> > > >
> > > > My vote goes to option *a*.
> > > > > I think we should release 2.7 with the bunch of new cool features.
> > > > > *AND* we should plan 2.8 release at the end of the year with SG
> > > redesign
> > > > > and MVCC stabilization tasks.
> > > >
> > > >
> > > > --
> > > > Denis
> > > >
> > > > On Fri, Sep 28, 2018 at 10:30 AM Nikolay Izhikov <
> nizhi...@apache.org>
> > > > wrote:
> > > >
> > > > > Hello, Igniters.
> > > > >
> > > > > I think we shouldn't put so many emotions in discussion of any
> > > > > contribution.
> > > > > Even so big and important as SG redesign.
> > > > >
> > > > > The crucial point we all agreed about: Service Grid redesign a big
> > > > feature
> > > > > that can significally improve Ignite.
> > > > > We all want to have it in the product.
> > > > >
> > > > > Let me write my vision of the current task state:
> > > > >
> > > > > 1. Design of SG is discussed *and aligned* both: privately with
> Ignite
> > > > > experts(Vladimir Ozerov, Alexey Goncharyuk, Anton Vinogradov, Denis
> > > > > Mekhanikov)
> > > > > and publicly on dev-list. This task is done.
> > > > >
> > > > > 2. Current PR state - *on my review*.
> > > > > I spend some time with this contribution and gave Vyacheslav a
> > > feedback.
> > > > > I expect he can fix my comments in a day or two.
> > > > > Seem we can ask of Anton Vinogradov review on the beginning of next
> > > week.
> > > > >
> > > > > If Dmitriy or any community member wants to help *by doing things,
> not
> > > > > discussing them on dev-list*.
> > > > > Please, join to the review - you are welcome. PR is here [1]
> > > > >
> > > > > 3. I think, we have two mutually exclusive options regarding of
> release
> > > > > 2.7
> > > > > a. We release 2.7 in planned dates.
> > > > > b. We include SG in release scope.
> > > > >
> > > > > My vote goes to option *a*.
> > > > > I think we should release 2.7 with the bunch of new cool features.
> > > > > *AND* we should plan 2.8 release at the end of the year with SG
> > > redesign
> > > > > and MVCC stabilization tasks.
> > > > >
> > > > > Anyway, while we preparing release a lot of things can happen.
> > > > > Let's come back to discussion of SG release version *when it will
> be
> > > > ready
> > 

Re: [Discussion] revert of commit MVCC, ignite-9320

[Dmitriy Pavlov]

Vladimir, I agree more with Andrey's fix in the parity test, because it
disables only specific property failure.

If we keep .NET parity test failed for a long time, some other
contributor will introduce new property- it will begin to fail always - it
is like a snowball.

I hope the metrics test will also be fixed before 2.7

Sincerely,
Dmitriy Pavlov

пт, 28 сент. 2018 г. в 14:43, Павлухин Иван :

> Hi guys!
>
> By the way, is it practically feasible to revert a single commit without
> making harm? If I am getting it right in current case reverting commit will
> lead to compilation errors for commits depending on commit in question.
>
> 2018-09-28 14:22 GMT+03:00 Dmitriy Pavlov :
>
> > Hi Dmitriy S.,
> >
> > I really prefer avoiding reverts, which why I've started this topic. If I
> > were reverting-fan, I could just write: "Vetoing commit  because of
> > test failures , commit reverted, ticket IGNITE- reopened."
> >
> > But some time ago I several times asked newbie contributors to fix missed
> > test failures and they managed to do it in 1-2 days, I'm waiting these
> test
> > to be fixed by Ignite veteran(s) for 11 days.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> >
> > пт, 28 сент. 2018 г. в 13:16, Vladimir Ozerov :
> >
> > > Andrey,
> > >
> > > This is not a fix, but a hack, which covers real state of affairs.
> > >
> > > пт, 28 сент. 2018 г. в 13:00, Andrey Mashenkov <
> > andrey.mashen...@gmail.com
> > > >:
> > >
> > > > Hi,
> > > >
> > > > Fix is trivial and ready.
> > > > Hope, it will be merged within IGNITE-7764 today.
> > > >
> > > > https://issues.apache.org/jira/browse/IGNITE-7764
> > > >
> > > > On Fri, Sep 28, 2018 at 12:26 PM Dmitriy Setrakyan <
> > > dsetrak...@apache.org>
> > > > wrote:
> > > >
> > > > > Guys, let's just fix the tests without reverting commits.
> Reverting a
> > > > > commit may trigger a time machine, where all following commits may
> be
> > > > > broken because of it. Fixing that scenario will be much harder.
> > > > >
> > > > > Going forward, I would agree that we should not merge anything that
> > > > breaks
> > > > > tests. This is about following a basic engineering discipline. We
> > > should
> > > > > all do it.
> > > > >
> > > > > D.
> > > > >
> > > > >
> > > > > On Fri, Sep 28, 2018 at 12:47 AM Dmitriy Pavlov <
> > dpavlov@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Yep, we're humans and we constantly make mistakes. It is a very
> > human
> > > > > thing
> > > > > > to do mistakes.
> > > > > >
> > > > > > So I suggest we will be under the control and protection of robot
> > to
> > > > > avoid
> > > > > > mistakes, I suggest robot will revert such commits in 72h without
> > its
> > > > own
> > > > > > personal attitudes, emotions, etc.
> > > > > >
> > > > > > Someone who is interested in contribution usually can find time
> to
> > > make
> > > > > > contribution perfect.
> > > > > >
> > > > > > I'm not aware of project priorities, please share it. I believe
> > > > different
> > > > > > priorities can co-exist. A number of contributors are fixing
> tests,
> > > so
> > > > it
> > > > > > is a priority for them, isn't it? So why to add work to that guys
> > > > because
> > > > > > of you have other priorities?
> > > > > >
> > > > > > пт, 28 сент. 2018 г. в 10:39, Vladimir Ozerov <
> > voze...@gridgain.com
> > > >:
> > > > > >
> > > > > > > Because a lot of other activities depended on configuration in
> > > Java,
> > > > > and
> > > > > > we
> > > > > > > didn't have expertise to fix .NET immediately.
> > > > > > >
> > > > > > > If you want to revert it - please go ahead. But I'd better
> > suggest
> > > > you
> > > > > to
> > > > > > > think about the impact and project priorities first, instead of
> > > > trying
> > > > > to
> > > > > > > apply the some sort rules blindly. We are not robots.
> > > > > > >
> > > > > > > On Fri, Sep 28, 2018 at 10:19 AM Dmitriy Pavlov <
> > > > dpavlov@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Vladimir,
> > > > > > > >
> > > > > > > > https://issues.apache.org/jira/browse/IGNITE-9320 is named
> > > > > > configuration
> > > > > > > > finalization.
> > > > > > > >
> > > > > > > > Why finalization was considered as done without tests
> passing?
> > > > > > > >
> > > > > > > > Why can't ve revert finalization change, re-do finalization
> > with
> > > > > > passing
> > > > > > > > tests and merge changes?
> > > > > > > >
> > > > > > > > Sincerely,
> > > > > > > > Dmitriy Pavlov
> > > > > > > >
> > > > > > > > пт, 28 сент. 2018 г. в 8:16, Vladimir Ozerov <
> > > voze...@gridgain.com
> > > > >:
> > > > > > > >
> > > > > > > > > Test is going to be fixed in the scope of AI 2.7 [1]. This
> is
> > > not
> > > > > > > > > one-minute fix as there are multiple places where
> > configuration
> > > > > > should
> > > > > > > be
> > > > > > > > > passed, and changes should be covered with tests. I muted
> the
> > > > test
> > > > > > for
> > > > > > > > now.
> > > > > > > > >
> 

Re: [MTCGA]: new failures in builds [1871897] needs to be handled

[Dmitriy Pavlov]

Folks,

both tests are failed in ignite-2.7 IgniteStandByClusterTest.testSimple

 &
IgniteChangeGlobalStateFailOverTest.testActivateDeActivateOnFixTopologyWithPutValues



Can I hope these failures will be fixed in master and 2.7 before release?

https://issues.apache.org/jira/browse/IGNITE-7618




пт, 21 сент. 2018 г. в 11:33, Dmitrii Ryabov :

> Hi, Dmitriy,
> I checked 7618 and previous commits: test fails locally starting from 7618.
> It fails because `cache.get()` remembers deactivated state and doesn't
> check current state.
>
> 2018-09-20 18:41 GMT+03:00 Dmitriy Pavlov :
>
> > Hi,
> >
> > IgniteStandByClusterTest seems to fail, Dmitriy G., Ivan, would it be
> > reasonable to revert commit?
> >
> > Dmitriy Ryabov, is it related to recent fix or is it a standalone
> problem?
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > пн, 17 сент. 2018 г. в 18:45, Dmitrii Ryabov :
> >
> > > Looks like problem I had described in the ticket.
> > >
> > >
> > > https://issues.apache.org/jira/browse/IGNITE-7618?
> > focusedCommentId=16506923=com.atlassian.jira.
> > plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16506923
> > >
> > > 2018-09-15 12:01 GMT+03:00 Dmitriy Pavlov :
> > >
> > > > Dmitriy G, Ivan B,
> > > >
> > > > could you please double-check if failure is not coming from
> > > > https://issues.apache.org/jira/browse/IGNITE-7618
> > > >
> > > > Sincerely,
> > > > Dmitriy Pavlov
> > > >
> > > > сб, 15 сент. 2018 г. в 5:42, :
> > > >
> > > > > Hi Ignite Developer,
> > > > >
> > > > > I am MTCGA.Bot, and I've detected some issue on TeamCity to be
> > > addressed.
> > > > > I hope you can help.
> > > > >
> > > > >  *New test failure in master
> IgniteStandByClusterTest.testSimple
> > > > > https://ci.ignite.apache.org/project.html?projectId=
> > > > IgniteTests24Java8=1332314705000986815=%
> > > > 3Cdefault%3E=testDetails
> > > > >  Changes may led to failure were done by
> > > > >  - bessonov.ip
> > > > > http://ci.ignite.apache.org/viewModification.html?modId=
> > > > 831651=false
> > > > >
> > > > > - If your changes can led to this failure(s), please create
> > > issue
> > > > > with label MakeTeamCityGreenAgain and assign it to you.
> > > > > -- If you have fix, please set ticket to PA state and write
> > to
> > > > dev
> > > > > list fix is ready
> > > > > -- For case fix will require some time please mute test and
> > set
> > > > > label Muted_Test to issue
> > > > > - If you know which change caused failure please contact
> > change
> > > > > author directly
> > > > > - If you don't know which change caused failure please send
> > > > > message to dev list to find out
> > > > > Should you have any questions please contact dev@ignite.apache.org
> > > > > Best Regards,
> > > > > MTCGA.Bot
> > > > > Notification generated at Sat Sep 15 05:42:21 MSK 2018
> > > > >
> > > >
> > >
> >
>