Re: JSR 381 (Visual Recognition in Java) and Apache Ignite

[Zoran Sevarac]

Great, thanks, looking forward to it. In the meantime, I'll take a more
detailed look at the existing Apache Ignite ML support

Best,
Zoran

On Wed, Apr 1, 2020 at 7:59 PM Alexey Zinoviev 
wrote:

> Thanks, I will check it out. Great dlj library with the reference JSR
> implementation, I have not seen yet this repo.
>
> Will return in a few weeks with suggestions on JSR mailing list
>
> Sincerely yours,
>Alexey Zinoviev
>
> ср, 1 апр. 2020 г., 18:49 Zoran Sevarac :
>
>> I look on the https://djl.ai, it seems a little bit different with JSR
>>> (but close in builders), also as I understand (looking in sources) they
>>> don't support TF engine yet
>>> Is it a mature framework, how do you think?
>>>
>>
>> It is relatively new but very promising in my opinion. TF support is not
>> publically available but they do have it.
>> And also they provide implementation of JSR381 at
>> https://github.com/JavaVisRec/visrec-djl
>> They have recently joined JSR381
>>
>>
>>> Happy to join the unification movement in Java DL, where is the best
>>> place to leave feedback about specification? Here, in this thread or on
>>> another mailing list?
>>>
>>
>> For the initial feedback please use the official JSR381 mailing list at
>> https://groups.io/g/visrec
>>
>> Best
>> Zoran
>>
>>
>>>
>>> Sincerely yours,
>>>   Alexey Zinoviev
>>>
>>> ср, 1 апр. 2020 г. в 16:59, Zoran Sevarac :
>>>
 On Tue, Mar 31, 2020 at 3:03 PM Alexey Zinoviev 
 wrote:

> Nice to see all of you guys (thanks Vladimir) here on dev-list of
> Apache Ignite that includes distributed ML.
>

 Nice to meet you too Alexey, and thanks to Vladimir for introducing us
 to this community.


> First of all, want to say thank you for your great work to build specs
> to the Java world, hope it will be helpful for Java devs in the nearest
> future.
> In the second, a few days ago I found a link
>  in Frank
> Greco twitter to this JSR and read a few examples to understand the
> proposed API.
> Also, we (in Apache Ignite ML) have experimental integration with
> TensorFlow for inference and distributed training purposes and I'm
> interested in TF Java API development  and DL4j/PyTorch possible
> integration.
> Will be great to have common API for different implementations.
>

 Great, we're interested to hear about your experience with these libs,
 and requirements for distributed training. I also saw
 https://apacheignite.readme.io/docs/machine-learning  it looks like
 something different from Tensorflow integration? I guess it would make
 sense for end-users to use the same API for the same task regardless of the
 backend library.
 By the way, I recommend that you take a look at https://djl.ai it's
 deep learning library from Amazon which is a Java wrapper around MxNet, TF
 and PyTorch, they also participate in JSR381, and have provided initial
 implementation.


> P.S. From my side, I've ready to join your team if the specification
> is not finished yet, because I have a few comments and thoughts based on 
> my
> experience of different API usage. A few weeks ago I also joined to the
> developer group of JVM API for TensorFlow (SIG JVM), hope it will be
> helpful in both directions to communicate each other.
>

 Sounds good,  at this phase we're looking for the input and feedback
 from various communities in order to make API that will meet their needs.
 So we would definitely like to hear your thoughts and we're open to
 adapt the specification.

 Best regards
 Zoran



>
> пн, 30 мар. 2020 г. в 21:00, Vladimir Ozerov :
>
>> Hi Alexey, Igniters,
>>
>> Let me introduce you Heather, Zoran, and Frank. Heather is the Chair
>> of the JCP. Zoran and Frank are JSR 381 spec leads. They are interested 
>> in
>> discussing the upcoming visual recognition specification with the Apache
>> Ignite community, to understand whether the community has any interest in
>> implementing it.
>>
>> Zoran and Frank,
>>
>> Please meet Alexey Zinoviev, who is the principal maintainer of the
>> Apache ML module. I hope he will be able to help you with your questions.
>>
>> Regards,
>> Vladimir.
>>
>

 --
 Zoran Sevarac, PhD, Associate Professor
 University of Belgrade, Faculty of Organisational Sciences, Department
 for Software Engineering
 Java Champion
  | Oracle
 Groundbreaker Ambassador | Deep Netts 
 Co-founder & CEO

>>>
>>
>> --
>> Zoran Sevarac, PhD, Associate Professor
>> University of Belgrade, Faculty of Organisational Sciences, Department
>> for Software Engineering
>> Java Champion
>> 

Re: .Net core versions

[Pavel Tupitsyn]

Great to hear that it works well for you!

Personally I can't wait to get rid of all the legacy stuff in Ignite code
base,
and use newer language version, hopefully that comes soon enough :)

On Wed, Apr 1, 2020 at 11:47 PM Raymond Wilson 
wrote:

> I get the legacy argument :)
>
> I have just upgraded out Ignite system to Ignite 2.8 on .Net Core 3.1
> exclusively across a collection of services and large collection of
> supporting libraries. It works very well. This project involved killing all
> legacy .Net Framework dependencies. It was quite cathartic.
>
> It is features like Span, ValueTask etc, and all the new goodies coming
> down the line which would be good to take advantage of in the Ignite C#
> client.
>
>
>
> On Thu, Apr 2, 2020 at 9:23 AM Pavel Tupitsyn 
> wrote:
>
> > Hi Raymond,
> >
> > > .Net Standard 2.0 is the end of line for .Net Standard
> > > All new .Net and C# language features are only being added to .Net Core
> >
> > Do you mean .NET Framework here instead of .NET Standard?
> > Yes, ".NET Framework", old Windows-only thing, is not going to be updated
> > with new features anymore,
> > and will be eventually killed off.
> >
> > This has nothing to do with .NET Standard, however, which is not going
> > anywhere.
> > .NET Core 3.1 was introduced along with .NET Standard 2.1, they evolve
> > together.
> >
> > .NET Standard is the recommended target for library authors:
> >
> >
> https://docs.microsoft.com/en-us/dotnet/standard/library-guidance/cross-platform-targeting
> >
> >
> > Just to be clear: as a user, you will get all the performance benefits of
> > .NET Core 3.1,
> > even if Ignite library is targeting any old .NET Standard.
> > Even today, you can run Ignite 2.8 on .NET Core 3.1 and get all those
> > benefits.
> > I did some benchmarks, blog post is coming soon on that.
> >
> > We will probably do multitargeting with conditional compilation in order
> to
> > use some
> > new performance-critical features and APIs (ValueTask, Span,
> > Reflection.Emit).
> >
> > We won't target only .NET Core 3.1. We can't afford losing all the users
> of
> > older .NET versions.
> >
> >
> >
> > On Wed, Apr 1, 2020 at 10:18 PM Raymond Wilson <
> raymond_wil...@trimble.com
> > >
> > wrote:
> >
> > > Hi Pavel,
> > >
> > > It might be worth considering moving to .Net Core 3.1 instead of .Net
> > > Standard 2.0 for Ignite 3.0.
> > >
> > > .Net Standard 2.0 is the end of line for .Net Standard. While this does
> > > permit cross compatibility with .Net Framework, this is also at an end
> of
> > > line.
> > >
> > > All new .Net and C# language features are only being added to .Net
> > > Core, and there are significant improvements in many areas within .Net
> > > Core. With .Net 5.0 being the  successor to .Net Framework and .Net
> Core
> > > would moving to .Net Core 3.1 be a useful stage before targeting .Net
> > Core
> > > 5.0 in the future?
> > >
> > > Thanks,
> > > Raymond.
> > >
> > >
> > > On Thu, Apr 2, 2020 at 1:00 AM Pavel Tupitsyn 
> > > wrote:
> > >
> > > > Nikolay,
> > > >
> > > > We are targeting multiple platforms this way.
> > > > Our lowest supported .NET version is 4.0, that's what
> Apache.Ignite.sln
> > > is
> > > > for, and all the related projects.
> > > > However, .NET 4.0 is old and not cross-platform, so we also target
> .NET
> > > > Core 2.0.
> > > >
> > > > We have to have separate solution and project files for that
> > > > (Apache.Ignite.DotNetCore.sln) for multiple reasons:
> > > > * Normally you can multitarget as part of one csproj file, but not
> with
> > > > with a combination of .NET 4.0 and .NET Core 2.0
> > > > * There are legacy integrations (ASP.NET, EF) that do not compile
> > under
> > > > .NET Core.
> > > >
> > > > This is going to change in Ignite 3.0, we will get rid of the legacy
> > > parts
> > > > and move on to .NET Standard 2.0.
> > > >
> > > > Thanks,
> > > > Pavel
> > > >
> > > > On Wed, Apr 1, 2020 at 1:08 PM Ilya Kasnacheev <
> > > ilya.kasnach...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hello!
> > > > >
> > > > > As far as my understanding goes, v4.0 is for .Net classic, such as
> > > Mono.
> > > > >
> > > > > Regards,
> > > > > --
> > > > > Ilya Kasnacheev
> > > > >
> > > > >
> > > > > ср, 1 апр. 2020 г. в 13:04, Nikolay Izhikov :
> > > > >
> > > > > > Hello. Igniters.
> > > > > >
> > > > > > I working on some issue in .Net platforms and found that
> different
> > > > > project
> > > > > > files for .Net contains different platform versions.
> > > > > > Can someone explain why?
> > > > > >
> > > > > > 1. Apache.Ignite.Core.csproj [1]
> > > > > >
> > > > > > ```
> > > > > > 
> > > > > > ...
> > > > > >   v4.0
> > > > > > ...
> > > > > > 
> > > > > > ```
> > > > > >
> > > > > > 2. Apache.Ignite.Core.DotNetCore.csproj
> > > > > >
> > > > > > ```
> > > > > > 

Re: .Net core versions

[Raymond Wilson]

I get the legacy argument :)

I have just upgraded out Ignite system to Ignite 2.8 on .Net Core 3.1
exclusively across a collection of services and large collection of
supporting libraries. It works very well. This project involved killing all
legacy .Net Framework dependencies. It was quite cathartic.

It is features like Span, ValueTask etc, and all the new goodies coming
down the line which would be good to take advantage of in the Ignite C#
client.



On Thu, Apr 2, 2020 at 9:23 AM Pavel Tupitsyn  wrote:

> Hi Raymond,
>
> > .Net Standard 2.0 is the end of line for .Net Standard
> > All new .Net and C# language features are only being added to .Net Core
>
> Do you mean .NET Framework here instead of .NET Standard?
> Yes, ".NET Framework", old Windows-only thing, is not going to be updated
> with new features anymore,
> and will be eventually killed off.
>
> This has nothing to do with .NET Standard, however, which is not going
> anywhere.
> .NET Core 3.1 was introduced along with .NET Standard 2.1, they evolve
> together.
>
> .NET Standard is the recommended target for library authors:
>
> https://docs.microsoft.com/en-us/dotnet/standard/library-guidance/cross-platform-targeting
>
>
> Just to be clear: as a user, you will get all the performance benefits of
> .NET Core 3.1,
> even if Ignite library is targeting any old .NET Standard.
> Even today, you can run Ignite 2.8 on .NET Core 3.1 and get all those
> benefits.
> I did some benchmarks, blog post is coming soon on that.
>
> We will probably do multitargeting with conditional compilation in order to
> use some
> new performance-critical features and APIs (ValueTask, Span,
> Reflection.Emit).
>
> We won't target only .NET Core 3.1. We can't afford losing all the users of
> older .NET versions.
>
>
>
> On Wed, Apr 1, 2020 at 10:18 PM Raymond Wilson  >
> wrote:
>
> > Hi Pavel,
> >
> > It might be worth considering moving to .Net Core 3.1 instead of .Net
> > Standard 2.0 for Ignite 3.0.
> >
> > .Net Standard 2.0 is the end of line for .Net Standard. While this does
> > permit cross compatibility with .Net Framework, this is also at an end of
> > line.
> >
> > All new .Net and C# language features are only being added to .Net
> > Core, and there are significant improvements in many areas within .Net
> > Core. With .Net 5.0 being the  successor to .Net Framework and .Net Core
> > would moving to .Net Core 3.1 be a useful stage before targeting .Net
> Core
> > 5.0 in the future?
> >
> > Thanks,
> > Raymond.
> >
> >
> > On Thu, Apr 2, 2020 at 1:00 AM Pavel Tupitsyn 
> > wrote:
> >
> > > Nikolay,
> > >
> > > We are targeting multiple platforms this way.
> > > Our lowest supported .NET version is 4.0, that's what Apache.Ignite.sln
> > is
> > > for, and all the related projects.
> > > However, .NET 4.0 is old and not cross-platform, so we also target .NET
> > > Core 2.0.
> > >
> > > We have to have separate solution and project files for that
> > > (Apache.Ignite.DotNetCore.sln) for multiple reasons:
> > > * Normally you can multitarget as part of one csproj file, but not with
> > > with a combination of .NET 4.0 and .NET Core 2.0
> > > * There are legacy integrations (ASP.NET, EF) that do not compile
> under
> > > .NET Core.
> > >
> > > This is going to change in Ignite 3.0, we will get rid of the legacy
> > parts
> > > and move on to .NET Standard 2.0.
> > >
> > > Thanks,
> > > Pavel
> > >
> > > On Wed, Apr 1, 2020 at 1:08 PM Ilya Kasnacheev <
> > ilya.kasnach...@gmail.com>
> > > wrote:
> > >
> > > > Hello!
> > > >
> > > > As far as my understanding goes, v4.0 is for .Net classic, such as
> > Mono.
> > > >
> > > > Regards,
> > > > --
> > > > Ilya Kasnacheev
> > > >
> > > >
> > > > ср, 1 апр. 2020 г. в 13:04, Nikolay Izhikov :
> > > >
> > > > > Hello. Igniters.
> > > > >
> > > > > I working on some issue in .Net platforms and found that different
> > > > project
> > > > > files for .Net contains different platform versions.
> > > > > Can someone explain why?
> > > > >
> > > > > 1. Apache.Ignite.Core.csproj [1]
> > > > >
> > > > > ```
> > > > > 
> > > > > ...
> > > > >   v4.0
> > > > > ...
> > > > > 
> > > > > ```
> > > > >
> > > > > 2. Apache.Ignite.Core.DotNetCore.csproj
> > > > >
> > > > > ```
> > > > > 

Re: .Net core versions

[Pavel Tupitsyn]

Hi Raymond,

> .Net Standard 2.0 is the end of line for .Net Standard
> All new .Net and C# language features are only being added to .Net Core

Do you mean .NET Framework here instead of .NET Standard?
Yes, ".NET Framework", old Windows-only thing, is not going to be updated
with new features anymore,
and will be eventually killed off.

This has nothing to do with .NET Standard, however, which is not going
anywhere.
.NET Core 3.1 was introduced along with .NET Standard 2.1, they evolve
together.

.NET Standard is the recommended target for library authors:
https://docs.microsoft.com/en-us/dotnet/standard/library-guidance/cross-platform-targeting


Just to be clear: as a user, you will get all the performance benefits of
.NET Core 3.1,
even if Ignite library is targeting any old .NET Standard.
Even today, you can run Ignite 2.8 on .NET Core 3.1 and get all those
benefits.
I did some benchmarks, blog post is coming soon on that.

We will probably do multitargeting with conditional compilation in order to
use some
new performance-critical features and APIs (ValueTask, Span,
Reflection.Emit).

We won't target only .NET Core 3.1. We can't afford losing all the users of
older .NET versions.



On Wed, Apr 1, 2020 at 10:18 PM Raymond Wilson 
wrote:

> Hi Pavel,
>
> It might be worth considering moving to .Net Core 3.1 instead of .Net
> Standard 2.0 for Ignite 3.0.
>
> .Net Standard 2.0 is the end of line for .Net Standard. While this does
> permit cross compatibility with .Net Framework, this is also at an end of
> line.
>
> All new .Net and C# language features are only being added to .Net
> Core, and there are significant improvements in many areas within .Net
> Core. With .Net 5.0 being the  successor to .Net Framework and .Net Core
> would moving to .Net Core 3.1 be a useful stage before targeting .Net Core
> 5.0 in the future?
>
> Thanks,
> Raymond.
>
>
> On Thu, Apr 2, 2020 at 1:00 AM Pavel Tupitsyn 
> wrote:
>
> > Nikolay,
> >
> > We are targeting multiple platforms this way.
> > Our lowest supported .NET version is 4.0, that's what Apache.Ignite.sln
> is
> > for, and all the related projects.
> > However, .NET 4.0 is old and not cross-platform, so we also target .NET
> > Core 2.0.
> >
> > We have to have separate solution and project files for that
> > (Apache.Ignite.DotNetCore.sln) for multiple reasons:
> > * Normally you can multitarget as part of one csproj file, but not with
> > with a combination of .NET 4.0 and .NET Core 2.0
> > * There are legacy integrations (ASP.NET, EF) that do not compile under
> > .NET Core.
> >
> > This is going to change in Ignite 3.0, we will get rid of the legacy
> parts
> > and move on to .NET Standard 2.0.
> >
> > Thanks,
> > Pavel
> >
> > On Wed, Apr 1, 2020 at 1:08 PM Ilya Kasnacheev <
> ilya.kasnach...@gmail.com>
> > wrote:
> >
> > > Hello!
> > >
> > > As far as my understanding goes, v4.0 is for .Net classic, such as
> Mono.
> > >
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > >
> > >
> > > ср, 1 апр. 2020 г. в 13:04, Nikolay Izhikov :
> > >
> > > > Hello. Igniters.
> > > >
> > > > I working on some issue in .Net platforms and found that different
> > > project
> > > > files for .Net contains different platform versions.
> > > > Can someone explain why?
> > > >
> > > > 1. Apache.Ignite.Core.csproj [1]
> > > >
> > > > ```
> > > > 
> > > > ...
> > > >   v4.0
> > > > ...
> > > > 
> > > > ```
> > > >
> > > > 2. Apache.Ignite.Core.DotNetCore.csproj
> > > >
> > > > ```
> > > > 

Re: .Net core versions

[Raymond Wilson]

Hi Pavel,

It might be worth considering moving to .Net Core 3.1 instead of .Net
Standard 2.0 for Ignite 3.0.

.Net Standard 2.0 is the end of line for .Net Standard. While this does
permit cross compatibility with .Net Framework, this is also at an end of
line.

All new .Net and C# language features are only being added to .Net
Core, and there are significant improvements in many areas within .Net
Core. With .Net 5.0 being the  successor to .Net Framework and .Net Core
would moving to .Net Core 3.1 be a useful stage before targeting .Net Core
5.0 in the future?

Thanks,
Raymond.


On Thu, Apr 2, 2020 at 1:00 AM Pavel Tupitsyn  wrote:

> Nikolay,
>
> We are targeting multiple platforms this way.
> Our lowest supported .NET version is 4.0, that's what Apache.Ignite.sln is
> for, and all the related projects.
> However, .NET 4.0 is old and not cross-platform, so we also target .NET
> Core 2.0.
>
> We have to have separate solution and project files for that
> (Apache.Ignite.DotNetCore.sln) for multiple reasons:
> * Normally you can multitarget as part of one csproj file, but not with
> with a combination of .NET 4.0 and .NET Core 2.0
> * There are legacy integrations (ASP.NET, EF) that do not compile under
> .NET Core.
>
> This is going to change in Ignite 3.0, we will get rid of the legacy parts
> and move on to .NET Standard 2.0.
>
> Thanks,
> Pavel
>
> On Wed, Apr 1, 2020 at 1:08 PM Ilya Kasnacheev 
> wrote:
>
> > Hello!
> >
> > As far as my understanding goes, v4.0 is for .Net classic, such as Mono.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > ср, 1 апр. 2020 г. в 13:04, Nikolay Izhikov :
> >
> > > Hello. Igniters.
> > >
> > > I working on some issue in .Net platforms and found that different
> > project
> > > files for .Net contains different platform versions.
> > > Can someone explain why?
> > >
> > > 1. Apache.Ignite.Core.csproj [1]
> > >
> > > ```
> > > 
> > > ...
> > >   v4.0
> > > ...
> > > 
> > > ```
> > >
> > > 2. Apache.Ignite.Core.DotNetCore.csproj
> > >
> > > ```
> > > 

Re: JSR 381 (Visual Recognition in Java) and Apache Ignite

[Alexey Zinoviev]

Thanks, I will check it out. Great dlj library with the reference JSR
implementation, I have not seen yet this repo.

Will return in a few weeks with suggestions on JSR mailing list

Sincerely yours,
   Alexey Zinoviev

ср, 1 апр. 2020 г., 18:49 Zoran Sevarac :

> I look on the https://djl.ai, it seems a little bit different with JSR
>> (but close in builders), also as I understand (looking in sources) they
>> don't support TF engine yet
>> Is it a mature framework, how do you think?
>>
>
> It is relatively new but very promising in my opinion. TF support is not
> publically available but they do have it.
> And also they provide implementation of JSR381 at
> https://github.com/JavaVisRec/visrec-djl
> They have recently joined JSR381
>
>
>> Happy to join the unification movement in Java DL, where is the best
>> place to leave feedback about specification? Here, in this thread or on
>> another mailing list?
>>
>
> For the initial feedback please use the official JSR381 mailing list at
> https://groups.io/g/visrec
>
> Best
> Zoran
>
>
>>
>> Sincerely yours,
>>   Alexey Zinoviev
>>
>> ср, 1 апр. 2020 г. в 16:59, Zoran Sevarac :
>>
>>> On Tue, Mar 31, 2020 at 3:03 PM Alexey Zinoviev 
>>> wrote:
>>>
 Nice to see all of you guys (thanks Vladimir) here on dev-list of
 Apache Ignite that includes distributed ML.

>>>
>>> Nice to meet you too Alexey, and thanks to Vladimir for introducing us
>>> to this community.
>>>
>>>
 First of all, want to say thank you for your great work to build specs
 to the Java world, hope it will be helpful for Java devs in the nearest
 future.
 In the second, a few days ago I found a link
  in Frank
 Greco twitter to this JSR and read a few examples to understand the
 proposed API.
 Also, we (in Apache Ignite ML) have experimental integration with
 TensorFlow for inference and distributed training purposes and I'm
 interested in TF Java API development  and DL4j/PyTorch possible
 integration.
 Will be great to have common API for different implementations.

>>>
>>> Great, we're interested to hear about your experience with these libs,
>>> and requirements for distributed training. I also saw
>>> https://apacheignite.readme.io/docs/machine-learning  it looks like
>>> something different from Tensorflow integration? I guess it would make
>>> sense for end-users to use the same API for the same task regardless of the
>>> backend library.
>>> By the way, I recommend that you take a look at https://djl.ai it's
>>> deep learning library from Amazon which is a Java wrapper around MxNet, TF
>>> and PyTorch, they also participate in JSR381, and have provided initial
>>> implementation.
>>>
>>>
 P.S. From my side, I've ready to join your team if the specification is
 not finished yet, because I have a few comments and thoughts based on my
 experience of different API usage. A few weeks ago I also joined to the
 developer group of JVM API for TensorFlow (SIG JVM), hope it will be
 helpful in both directions to communicate each other.

>>>
>>> Sounds good,  at this phase we're looking for the input and feedback
>>> from various communities in order to make API that will meet their needs.
>>> So we would definitely like to hear your thoughts and we're open to
>>> adapt the specification.
>>>
>>> Best regards
>>> Zoran
>>>
>>>
>>>

 пн, 30 мар. 2020 г. в 21:00, Vladimir Ozerov :

> Hi Alexey, Igniters,
>
> Let me introduce you Heather, Zoran, and Frank. Heather is the Chair
> of the JCP. Zoran and Frank are JSR 381 spec leads. They are interested in
> discussing the upcoming visual recognition specification with the Apache
> Ignite community, to understand whether the community has any interest in
> implementing it.
>
> Zoran and Frank,
>
> Please meet Alexey Zinoviev, who is the principal maintainer of the
> Apache ML module. I hope he will be able to help you with your questions.
>
> Regards,
> Vladimir.
>

>>>
>>> --
>>> Zoran Sevarac, PhD, Associate Professor
>>> University of Belgrade, Faculty of Organisational Sciences, Department
>>> for Software Engineering
>>> Java Champion
>>>  | Oracle
>>> Groundbreaker Ambassador | Deep Netts 
>>> Co-founder & CEO
>>>
>>
>
> --
> Zoran Sevarac, PhD, Associate Professor
> University of Belgrade, Faculty of Organisational Sciences, Department
> for Software Engineering
> Java Champion 
> | Oracle Groundbreaker Ambassador | Deep Netts 
> Co-founder & CEO
>

Re: Data vanished from cluster after INACTIVE/ACTIVE switch

[Ivan Rakov]

I don't think that making javadocs more descriptive can be considered as
harmful code base enlargement.
I'd recommend to extend the docs, but the last word is yours ;)

On Tue, Mar 31, 2020 at 2:44 PM Vladimir Steshin  wrote:

> Ivan, hi.
>
> I absolutely agree this particular description is not enough to see the
> deactivation issue. I also vote for brief code.
>
> There are about 15 places in inner logic with this description. I
> propose balance between code base size and comment completeness.
>
> Should we enlarge code even if we got several full descriptions?
>
>
> 30.03.2020 20:02, Ivan Rakov пишет:
> > Vladimir,
> >
> > @param forceDeactivation If {@code true}, cluster deactivation will be
> >> forced.
> > It's true that it's possible to infer semantics of forced deactivation
> from
> > other parts of API. I just wanted to highlight that exactly this
> > description explains something that can be guessed by the parameter name.
> > I suppose to shorten the lookup path and shed a light on deactivation
> > semantics a bit:
> >
> >> @param forceDeactivation If {@code true}, cluster will be deactivated
> even
> >> if running in-memory caches are present. All data in the corresponding
> >> caches will be vanished as a result.
> > Does this make sense?
> >
> > On Fri, Mar 27, 2020 at 12:00 PM Vladimir Steshin 
> > wrote:
> >
> >> Ivan, hi.
> >>
> >>
> >> 1) >>> Is it correct? If we are on the same page, let's proceed this way
> >>
> >> It is correct.
> >>
> >>
> >> 2) - In many places in the code I can see the following javadoc
> >>
> >>>@param forceDeactivation If {@code true}, cluster deactivation will
> be
> >> forced.
> >>
> >> In the internal params/flags. You can also find /@see
> >> ClusterState#INACTIVE/ and full description with several public APIs (
> >> like /Ignite.active(boolean)/ ):
> >>
> >> //
> >>
> >> /* /
> >>
> >> //
> >>
> >> /* NOTE:/
> >>
> >> //
> >>
> >> /* Deactivation clears in-memory caches (without persistence) including
> >> the system caches./
> >>
> >> Should be enough. Is not?
> >>
> >>
> >> 27.03.2020 10:51, Ivan Rakov пишет:
> >>> Vladimir, Igniters,
> >>>
> >>> Let's emphasize our final plan.
> >>>
> >>> We are going to add --force flags that will be necessary to pass for a
> >>> deactivation if there are in-memory caches to:
> >>> 1) Rest API (already implemented in [1])
> >>> 2) Command line utility (already implemented in [1])
> >>> 3) JMX bean (going to be implemented in [2])
> >>> We are *not* going to change IgniteCluster or any other thick Java API,
> >>> thus we are *not* going to merge [3].
> >>> We plan to *fully rollback* [1] and [2] once cache data survival after
> >>> activation-deactivation cycle will be implemented.
> >>>
> >>> Is it correct? If we are on the same page, let's proceed this way.
> >>> I propose to:
> >>> - Create a JIRA issue for in-memory-data-safe deactivation (possibly,
> >>> without IEP and detailed design so far)
> >>> - Describe in the issue description what exact parts of API should be
> >>> removed under the issue scope.
> >>>
> >>> Also, a few questions on already merged [1]:
> >>> - We have removed GridClientClusterState#state(ClusterState) from Java
> >>> client API. Is it a legitimate thing to do? Don't we have to support
> API
> >>> compatibility for thin clients as well?
> >>> - In many places in the code I can see the following javadoc
> >>>
> @param forceDeactivation If {@code true}, cluster deactivation will
> >> be forced.
>  As for me, this javadoc doesn't clarify anything. I'd suggest to
> >> describe
> >>> in which cases deactivation won't happen unless it's forced and which
> >>> impact forced deactivation will bring on the system.
> >>>
> >>> [1]: https://issues.apache.org/jira/browse/IGNITE-12701
> >>> [2]: https://issues.apache.org/jira/browse/IGNITE-12779
> >>> [3]: https://issues.apache.org/jira/browse/IGNITE-12614
> >>>
> >>> --
> >>> Ivan
> >>>
> >>> On Tue, Mar 24, 2020 at 7:18 PM Vladimir Steshin 
> >> wrote:
>  Hi, Igniters.
> 
>  I'd like to remind you that cluster can be deactivated by user with 3
>  utilities: control.sh, *JMX and the REST*. Proposed in [1] solution is
>  not about control.sh. It suggests same approach regardless of the
>  utility user executes. The task touches *only* *API of the user
> calls*,
>  not the internal APIs.
> 
>  The reasons why flag “--yes” and confirmation prompt hasn’t taken into
>  account for control.sh are:
> 
>  -Various commands widely use “--yes” just to start. Even not dangerous
>  ones require “--yes” to begin. “--force” is dedicated for *harmless
>  actions*.
> 
>  -Checking of probable data erasure works after command start and
>  “--force” may not be required at all.
> 
>  -There are also JMX and REST. They have no “—yes” but should work
> alike.
> 
> To get the deactivation safe I propose to merge last ticket
> with
>  the JMX fixes [2]. In future 

[jira] [Created] (IGNITE-12853) ThinClient: Introduce Features for thin clients

[Igor Sapego (Jira)]

Igor Sapego created IGNITE-12853:


 Summary: ThinClient: Introduce Features for thin clients
 Key: IGNITE-12853
 URL: https://issues.apache.org/jira/browse/IGNITE-12853
 Project: Ignite
  Issue Type: Bug
  Components: thin client
Reporter: Igor Sapego
Assignee: Igor Sapego
 Fix For: 2.8.1


As we have a lot of different thin clients now, maintained by different people, 
the issues with our backward compatibility mechanism becomes more and more 
prominent.

Currently, we use protocol versioning as the only approach to provide backward 
compatibility. The main issue of this approach is that we can not skip some 
change in protocol and implement i.e. protocol of version 1.5 without 
implementation of 1.4. There are many cases when one may want to do so: e.g. 
when feature provided in 1.4 is not relevant for a specific client, or when 
protocol version 1.5 contains urgent fix or feature which is easy to implement, 
but its blocked by not-so-urgent and hard-to-implement feature introduced in 
1.4.

So to fix this issue I propose to introduce another backward compatibility 
mechanism. The idea is to send "supported features" mask by a client to a 
server, which should be answered with the same mask by the server. The 
resulting set of enabled features is acquired with a simple logical "AND" 
operation on these two masks.

This change has many other positive effects: 
1. It improves readability and also potentially simplifies debugging.
2. It gives users the ability to enable or disable features of thin clients on 
both server and client as they desire.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: JSR 381 (Visual Recognition in Java) and Apache Ignite

[Zoran Sevarac]

>
> I look on the https://djl.ai, it seems a little bit different with JSR
> (but close in builders), also as I understand (looking in sources) they
> don't support TF engine yet
> Is it a mature framework, how do you think?
>

It is relatively new but very promising in my opinion. TF support is not
publically available but they do have it.
And also they provide implementation of JSR381 at
https://github.com/JavaVisRec/visrec-djl
They have recently joined JSR381


> Happy to join the unification movement in Java DL, where is the best place
> to leave feedback about specification? Here, in this thread or on another
> mailing list?
>

For the initial feedback please use the official JSR381 mailing list at
https://groups.io/g/visrec

Best
Zoran


>
> Sincerely yours,
>   Alexey Zinoviev
>
> ср, 1 апр. 2020 г. в 16:59, Zoran Sevarac :
>
>> On Tue, Mar 31, 2020 at 3:03 PM Alexey Zinoviev 
>> wrote:
>>
>>> Nice to see all of you guys (thanks Vladimir) here on dev-list of Apache
>>> Ignite that includes distributed ML.
>>>
>>
>> Nice to meet you too Alexey, and thanks to Vladimir for introducing us to
>> this community.
>>
>>
>>> First of all, want to say thank you for your great work to build specs
>>> to the Java world, hope it will be helpful for Java devs in the nearest
>>> future.
>>> In the second, a few days ago I found a link
>>>  in Frank
>>> Greco twitter to this JSR and read a few examples to understand the
>>> proposed API.
>>> Also, we (in Apache Ignite ML) have experimental integration with
>>> TensorFlow for inference and distributed training purposes and I'm
>>> interested in TF Java API development  and DL4j/PyTorch possible
>>> integration.
>>> Will be great to have common API for different implementations.
>>>
>>
>> Great, we're interested to hear about your experience with these libs,
>> and requirements for distributed training. I also saw
>> https://apacheignite.readme.io/docs/machine-learning  it looks like
>> something different from Tensorflow integration? I guess it would make
>> sense for end-users to use the same API for the same task regardless of the
>> backend library.
>> By the way, I recommend that you take a look at https://djl.ai it's deep
>> learning library from Amazon which is a Java wrapper around MxNet, TF and
>> PyTorch, they also participate in JSR381, and have provided initial
>> implementation.
>>
>>
>>> P.S. From my side, I've ready to join your team if the specification is
>>> not finished yet, because I have a few comments and thoughts based on my
>>> experience of different API usage. A few weeks ago I also joined to the
>>> developer group of JVM API for TensorFlow (SIG JVM), hope it will be
>>> helpful in both directions to communicate each other.
>>>
>>
>> Sounds good,  at this phase we're looking for the input and feedback from
>> various communities in order to make API that will meet their needs.
>> So we would definitely like to hear your thoughts and we're open to adapt
>> the specification.
>>
>> Best regards
>> Zoran
>>
>>
>>
>>>
>>> пн, 30 мар. 2020 г. в 21:00, Vladimir Ozerov :
>>>
 Hi Alexey, Igniters,

 Let me introduce you Heather, Zoran, and Frank. Heather is the Chair of
 the JCP. Zoran and Frank are JSR 381 spec leads. They are interested in
 discussing the upcoming visual recognition specification with the Apache
 Ignite community, to understand whether the community has any interest in
 implementing it.

 Zoran and Frank,

 Please meet Alexey Zinoviev, who is the principal maintainer of the
 Apache ML module. I hope he will be able to help you with your questions.

 Regards,
 Vladimir.

>>>
>>
>> --
>> Zoran Sevarac, PhD, Associate Professor
>> University of Belgrade, Faculty of Organisational Sciences, Department
>> for Software Engineering
>> Java Champion
>>  | Oracle
>> Groundbreaker Ambassador | Deep Netts 
>> Co-founder & CEO
>>
>

-- 
Zoran Sevarac, PhD, Associate Professor
University of Belgrade, Faculty of Organisational Sciences, Department for
Software Engineering
Java Champion 
| Oracle Groundbreaker Ambassador | Deep Netts 
Co-founder & CEO

Re: JSR 381 (Visual Recognition in Java) and Apache Ignite

[Alexey Zinoviev]

We removed the TF integration from the current release due to a few
problems with TF 2.0 and new policy about tf.contrib but it will turn out
after problem solving
You could find docs for previous Ignite and TF versions
https://apacheignite.readme.io/v2.7.6/docs/tensor-flow

I look on the https://djl.ai, it seems a little bit different with JSR (but
close in builders), also as I understand (looking in sources) they don't
support TF engine yet
Is it a mature framework, how do you think?

Happy to join the unification movement in Java DL, where is the best place
to leave feedback about specification? Here, in this thread or on another
mailing list?

Sincerely yours,
  Alexey Zinoviev

ср, 1 апр. 2020 г. в 16:59, Zoran Sevarac :

> On Tue, Mar 31, 2020 at 3:03 PM Alexey Zinoviev 
> wrote:
>
>> Nice to see all of you guys (thanks Vladimir) here on dev-list of Apache
>> Ignite that includes distributed ML.
>>
>
> Nice to meet you too Alexey, and thanks to Vladimir for introducing us to
> this community.
>
>
>> First of all, want to say thank you for your great work to build specs to
>> the Java world, hope it will be helpful for Java devs in the nearest
>> future.
>> In the second, a few days ago I found a link
>>  in Frank
>> Greco twitter to this JSR and read a few examples to understand the
>> proposed API.
>> Also, we (in Apache Ignite ML) have experimental integration with
>> TensorFlow for inference and distributed training purposes and I'm
>> interested in TF Java API development  and DL4j/PyTorch possible
>> integration.
>> Will be great to have common API for different implementations.
>>
>
> Great, we're interested to hear about your experience with these libs, and
> requirements for distributed training. I also saw
> https://apacheignite.readme.io/docs/machine-learning  it looks like
> something different from Tensorflow integration? I guess it would make
> sense for end-users to use the same API for the same task regardless of the
> backend library.
> By the way, I recommend that you take a look at https://djl.ai it's deep
> learning library from Amazon which is a Java wrapper around MxNet, TF and
> PyTorch, they also participate in JSR381, and have provided initial
> implementation.
>
>
>> P.S. From my side, I've ready to join your team if the specification is
>> not finished yet, because I have a few comments and thoughts based on my
>> experience of different API usage. A few weeks ago I also joined to the
>> developer group of JVM API for TensorFlow (SIG JVM), hope it will be
>> helpful in both directions to communicate each other.
>>
>
> Sounds good,  at this phase we're looking for the input and feedback from
> various communities in order to make API that will meet their needs.
> So we would definitely like to hear your thoughts and we're open to adapt
> the specification.
>
> Best regards
> Zoran
>
>
>
>>
>> пн, 30 мар. 2020 г. в 21:00, Vladimir Ozerov :
>>
>>> Hi Alexey, Igniters,
>>>
>>> Let me introduce you Heather, Zoran, and Frank. Heather is the Chair of
>>> the JCP. Zoran and Frank are JSR 381 spec leads. They are interested in
>>> discussing the upcoming visual recognition specification with the Apache
>>> Ignite community, to understand whether the community has any interest in
>>> implementing it.
>>>
>>> Zoran and Frank,
>>>
>>> Please meet Alexey Zinoviev, who is the principal maintainer of the
>>> Apache ML module. I hope he will be able to help you with your questions.
>>>
>>> Regards,
>>> Vladimir.
>>>
>>
>
> --
> Zoran Sevarac, PhD, Associate Professor
> University of Belgrade, Faculty of Organisational Sciences, Department
> for Software Engineering
> Java Champion 
> | Oracle Groundbreaker Ambassador | Deep Netts 
> Co-founder & CEO
>

Re: JSR 381 (Visual Recognition in Java) and Apache Ignite

[Zoran Sevarac]

On Tue, Mar 31, 2020 at 3:03 PM Alexey Zinoviev 
wrote:

> Nice to see all of you guys (thanks Vladimir) here on dev-list of Apache
> Ignite that includes distributed ML.
>

Nice to meet you too Alexey, and thanks to Vladimir for introducing us to
this community.


> First of all, want to say thank you for your great work to build specs to
> the Java world, hope it will be helpful for Java devs in the nearest
> future.
> In the second, a few days ago I found a link
>  in Frank
> Greco twitter to this JSR and read a few examples to understand the
> proposed API.
> Also, we (in Apache Ignite ML) have experimental integration with
> TensorFlow for inference and distributed training purposes and I'm
> interested in TF Java API development  and DL4j/PyTorch possible
> integration.
> Will be great to have common API for different implementations.
>

Great, we're interested to hear about your experience with these libs, and
requirements for distributed training. I also saw
https://apacheignite.readme.io/docs/machine-learning  it looks like
something different from Tensorflow integration? I guess it would make
sense for end-users to use the same API for the same task regardless of the
backend library.
By the way, I recommend that you take a look at https://djl.ai it's deep
learning library from Amazon which is a Java wrapper around MxNet, TF and
PyTorch, they also participate in JSR381, and have provided initial
implementation.


> P.S. From my side, I've ready to join your team if the specification is
> not finished yet, because I have a few comments and thoughts based on my
> experience of different API usage. A few weeks ago I also joined to the
> developer group of JVM API for TensorFlow (SIG JVM), hope it will be
> helpful in both directions to communicate each other.
>

Sounds good,  at this phase we're looking for the input and feedback from
various communities in order to make API that will meet their needs.
So we would definitely like to hear your thoughts and we're open to adapt
the specification.

Best regards
Zoran



>
> пн, 30 мар. 2020 г. в 21:00, Vladimir Ozerov :
>
>> Hi Alexey, Igniters,
>>
>> Let me introduce you Heather, Zoran, and Frank. Heather is the Chair of
>> the JCP. Zoran and Frank are JSR 381 spec leads. They are interested in
>> discussing the upcoming visual recognition specification with the Apache
>> Ignite community, to understand whether the community has any interest in
>> implementing it.
>>
>> Zoran and Frank,
>>
>> Please meet Alexey Zinoviev, who is the principal maintainer of the
>> Apache ML module. I hope he will be able to help you with your questions.
>>
>> Regards,
>> Vladimir.
>>
>

-- 
Zoran Sevarac, PhD, Associate Professor
University of Belgrade, Faculty of Organisational Sciences, Department for
Software Engineering
Java Champion 
| Oracle Groundbreaker Ambassador | Deep Netts 
Co-founder & CEO

[jira] [Created] (IGNITE-12852) Comma in field is not supported by COPY command

[YuJue Li (Jira)]

YuJue Li created IGNITE-12852:
-

 Summary: Comma in field is not supported by COPY command
 Key: IGNITE-12852
 URL: https://issues.apache.org/jira/browse/IGNITE-12852
 Project: Ignite
  Issue Type: Bug
  Components: sql
Affects Versions: 2.8
Reporter: YuJue Li
 Fix For: 2.8.1


CREATE TABLE test(a int,b varchar(100),c int,PRIMARY key(a)); 
 
a.csv: 
1,"a,b",2 
 
COPY FROM '/data/a.csv' INTO test (a,b,c) FORMAT CSV; 
 
The copy command fails because there is a comma in the second field,but this is 
a fully legal and compliant CSV format



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: .Net core versions

[Pavel Tupitsyn]

Do not confuse .NET Core and .NET Standard [1].
.NET Standard is an interface, .NET Core is an implementation.
.NET Core versions have end-of-support dates, .NET Standard versions do
not, since there is nothing to support.

Libraries should target .NET Standard, runnable applications can only
target .NET Core.
Ignite is a set of libraries and one runnable binary (standalone node).

* Ignite libraries will target .NET Standard 2.0 - lowest possible for us,
widely supported [1]
* Ignite standalone binary will target latest .NET Core LTS version
available during release, to maximize performance and security

[1] https://docs.microsoft.com/en-us/dotnet/standard/net-standard


On Wed, Apr 1, 2020 at 3:05 PM Nikolay Izhikov  wrote:

> Hello, Pavel.
>
> Thanks for the explanation.
>
> AFAIK .Net Core 2 is legacy also and not supported anymore [1]
>
> Is that true?
> Should we move to the .Net Core 3?
>
> [1] https://dotnet.microsoft.com/download/dotnet-core/2.2
>
> > 1 апр. 2020 г., в 15:00, Pavel Tupitsyn 
> написал(а):
> >
> > Nikolay,
> >
> > We are targeting multiple platforms this way.
> > Our lowest supported .NET version is 4.0, that's what Apache.Ignite.sln
> is
> > for, and all the related projects.
> > However, .NET 4.0 is old and not cross-platform, so we also target .NET
> > Core 2.0.
> >
> > We have to have separate solution and project files for that
> > (Apache.Ignite.DotNetCore.sln) for multiple reasons:
> > * Normally you can multitarget as part of one csproj file, but not with
> > with a combination of .NET 4.0 and .NET Core 2.0
> > * There are legacy integrations (ASP.NET, EF) that do not compile under
> > .NET Core.
> >
> > This is going to change in Ignite 3.0, we will get rid of the legacy
> parts
> > and move on to .NET Standard 2.0.
> >
> > Thanks,
> > Pavel
> >
> > On Wed, Apr 1, 2020 at 1:08 PM Ilya Kasnacheev <
> ilya.kasnach...@gmail.com>
> > wrote:
> >
> >> Hello!
> >>
> >> As far as my understanding goes, v4.0 is for .Net classic, such as Mono.
> >>
> >> Regards,
> >> --
> >> Ilya Kasnacheev
> >>
> >>
> >> ср, 1 апр. 2020 г. в 13:04, Nikolay Izhikov :
> >>
> >>> Hello. Igniters.
> >>>
> >>> I working on some issue in .Net platforms and found that different
> >> project
> >>> files for .Net contains different platform versions.
> >>> Can someone explain why?
> >>>
> >>> 1. Apache.Ignite.Core.csproj [1]
> >>>
> >>> ```
> >>> 
> >>> ...
> >>>  v4.0
> >>> ...
> >>> 
> >>> ```
> >>>
> >>> 2. Apache.Ignite.Core.DotNetCore.csproj
> >>>
> >>> ```
> >>> 

Re: .Net core versions

[Nikolay Izhikov]

Hello, Pavel.

Thanks for the explanation.

AFAIK .Net Core 2 is legacy also and not supported anymore [1]

Is that true?
Should we move to the .Net Core 3?

[1] https://dotnet.microsoft.com/download/dotnet-core/2.2

> 1 апр. 2020 г., в 15:00, Pavel Tupitsyn  написал(а):
> 
> Nikolay,
> 
> We are targeting multiple platforms this way.
> Our lowest supported .NET version is 4.0, that's what Apache.Ignite.sln is
> for, and all the related projects.
> However, .NET 4.0 is old and not cross-platform, so we also target .NET
> Core 2.0.
> 
> We have to have separate solution and project files for that
> (Apache.Ignite.DotNetCore.sln) for multiple reasons:
> * Normally you can multitarget as part of one csproj file, but not with
> with a combination of .NET 4.0 and .NET Core 2.0
> * There are legacy integrations (ASP.NET, EF) that do not compile under
> .NET Core.
> 
> This is going to change in Ignite 3.0, we will get rid of the legacy parts
> and move on to .NET Standard 2.0.
> 
> Thanks,
> Pavel
> 
> On Wed, Apr 1, 2020 at 1:08 PM Ilya Kasnacheev 
> wrote:
> 
>> Hello!
>> 
>> As far as my understanding goes, v4.0 is for .Net classic, such as Mono.
>> 
>> Regards,
>> --
>> Ilya Kasnacheev
>> 
>> 
>> ср, 1 апр. 2020 г. в 13:04, Nikolay Izhikov :
>> 
>>> Hello. Igniters.
>>> 
>>> I working on some issue in .Net platforms and found that different
>> project
>>> files for .Net contains different platform versions.
>>> Can someone explain why?
>>> 
>>> 1. Apache.Ignite.Core.csproj [1]
>>> 
>>> ```
>>> 
>>> ...
>>>  v4.0
>>> ...
>>> 
>>> ```
>>> 
>>> 2. Apache.Ignite.Core.DotNetCore.csproj
>>> 
>>> ```
>>> 

Re: .Net core versions

[Pavel Tupitsyn]

Nikolay,

We are targeting multiple platforms this way.
Our lowest supported .NET version is 4.0, that's what Apache.Ignite.sln is
for, and all the related projects.
However, .NET 4.0 is old and not cross-platform, so we also target .NET
Core 2.0.

We have to have separate solution and project files for that
(Apache.Ignite.DotNetCore.sln) for multiple reasons:
* Normally you can multitarget as part of one csproj file, but not with
with a combination of .NET 4.0 and .NET Core 2.0
* There are legacy integrations (ASP.NET, EF) that do not compile under
.NET Core.

This is going to change in Ignite 3.0, we will get rid of the legacy parts
and move on to .NET Standard 2.0.

Thanks,
Pavel

On Wed, Apr 1, 2020 at 1:08 PM Ilya Kasnacheev 
wrote:

> Hello!
>
> As far as my understanding goes, v4.0 is for .Net classic, such as Mono.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 1 апр. 2020 г. в 13:04, Nikolay Izhikov :
>
> > Hello. Igniters.
> >
> > I working on some issue in .Net platforms and found that different
> project
> > files for .Net contains different platform versions.
> > Can someone explain why?
> >
> > 1. Apache.Ignite.Core.csproj [1]
> >
> > ```
> > 
> > ...
> >   v4.0
> > ...
> > 
> > ```
> >
> > 2. Apache.Ignite.Core.DotNetCore.csproj
> >
> > ```
> > 

Re: deadlock in system pool on meta update

[Sergey]

Hi,
Here is reproducer for master :

https://github.com/macrergate/ignite/commit/9a7d2d27af30018a5f6faccb39176a35243ccfa2


Best regards,
Sergey Kosarev.


вт, 17 мар. 2020 г. в 12:27, Sergey-A Kosarev :

> Classification: Public
>
> Hi Sergey,
> Ticket is here https://issues.apache.org/jira/browse/IGNITE-12793
>
> I will try to make reproducer in the coming days.
>
> Kind regards,
> Sergey Kosarev
>
> -Original Message-
> From: Sergey Chugunov [mailto:sergey.chugu...@gmail.com]
> Sent: 17 March 2020 09:45
> To: dev 
> Subject: Re: deadlock in system pool on meta update
>
> Hello Sergey,
>
> Your analysis looks valid to me, we definitely need to investigate this
> deadlock and find out how to fix it.
>
> Could you create a ticket and write a test that reproduces the issue with
> sufficient probability?
>
> Thanks!
>
> On Mon, Mar 16, 2020 at 8:22 PM Sergey-A Kosarev 
> wrote:
>
> > Classification: Public
> >
> > Hi,
> > I've recently tried to apply Ilya's idea (
> > https://issues.apache.org/jira/browse/IGNITE-12663) of minimizing
> > thread pools and tried to set system pool to 3 in my own tests.
> > It caused deadlock on a client node and I think it can happen not only
> > on such small pool values.
> >
> > Details are following:
> > I'm not using persistence currently (if it matters).
> > On the client note I use ignite compute to  call   a job on every server
> > node (there are 3 server nodes in the tests).
> >
> > Then I've found in logs:
> >
> > [10:55:21] : [Step 1/1] [2020-03-13 10:55:21,773] {
> > grid-timeout-worker-#8} [WARN] [o.a.i.i.IgniteKernal] - Possible
> > thread pool starvation detected (no task completed in last 3ms, is
> > system thread pool size large enough?)
> > [10:55:21] : [Step 1/1] ^-- System thread pool [active=3, idle=0,
> > qSize=9]
> >
> >
> > I see in threaddumps that all 3 system pool workers do the same -
> > processing of job responses:
> >
> > "sys-#26" #605 daemon prio=5 os_prio=0 tid=0x64a0a800
> > nid=0x1f34 waiting on condition [0x7b91d000]
> >java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > at
> > java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> > at
> >
> org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
> > at
> >
> org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
> > at
> >
> org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.metadata(CacheObjectBinaryProcessorImpl.java:749)
> > at
> >
> org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl$1.metadata(CacheObjectBinaryProcessorImpl.java:250)
> > at
> >
> org.apache.ignite.internal.binary.BinaryContext.metadata(BinaryContext.java:1169)
> > at
> >
> org.apache.ignite.internal.binary.BinaryReaderExImpl.getOrCreateSchema(BinaryReaderExImpl.java:2005)
> > at
> >
> org.apache.ignite.internal.binary.BinaryReaderExImpl.(BinaryReaderExImpl.java:285)
> > at
> >
> org.apache.ignite.internal.binary.BinaryReaderExImpl.(BinaryReaderExImpl.java:184)
> > at
> >
> org.apache.ignite.internal.binary.BinaryUtils.doReadObject(BinaryUtils.java:1797)
> > at
> >
> org.apache.ignite.internal.binary.BinaryUtils.deserializeOrUnmarshal(BinaryUtils.java:2160)
> > at
> >
> org.apache.ignite.internal.binary.BinaryUtils.doReadCollection(BinaryUtils.java:2091)
> > at
> >
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1914)
> > at
> >
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1714)
> > at
> >
> org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1982)
> > at
> >
> org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:702)
> > at
> >
> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:187)
> > at
> >
> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:887)
> > at
> >
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1762)
> > at
> >
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1714)
> > at
> >
> org.apache.ignite.internal.binary.BinaryUtils.doReadObject(BinaryUtils.java:1797)
> > at
> >
> org.apache.ignite.internal.binary.BinaryUtils.deserializeOrUnmarshal(BinaryUtils.java:2160)
> > at
> >
> org.apache.ignite.internal.binary.BinaryUtils.doReadCollection(BinaryUtils.java:2091)
> > 

Re: .Net core versions

[Ilya Kasnacheev]

Hello!

As far as my understanding goes, v4.0 is for .Net classic, such as Mono.

Regards,
-- 
Ilya Kasnacheev


ср, 1 апр. 2020 г. в 13:04, Nikolay Izhikov :

> Hello. Igniters.
>
> I working on some issue in .Net platforms and found that different project
> files for .Net contains different platform versions.
> Can someone explain why?
>
> 1. Apache.Ignite.Core.csproj [1]
>
> ```
> 
> ...
>   v4.0
> ...
> 
> ```
>
> 2. Apache.Ignite.Core.DotNetCore.csproj
>
> ```
> 

.Net core versions

[Nikolay Izhikov]

Hello. Igniters.

I working on some issue in .Net platforms and found that different project 
files for .Net contains different platform versions.
Can someone explain why?

1. Apache.Ignite.Core.csproj [1]

```

...
  v4.0
...

```

2. Apache.Ignite.Core.DotNetCore.csproj

```

Re: Inconsistent API IgniteClient and REST

[Nikolay Izhikov]

Got it. Thank you.

> But we should recheck search results twice because we're talking about the 
> security.

+1

> 1 апр. 2020 г., в 11:56, Anton Vinogradov  написал(а):
> 
> That's exactly the question I have.
> Since no tests and documentation can be found - then we have no contract
> and it's safe to change the semantic.
> But we should recheck search results twice because we're talking about the
> security.
> 
> On Wed, Apr 1, 2020 at 11:44 AM Nikolay Izhikov  wrote:
> 
>> Hello, Anton.
>> 
>> What is «contract» for you?
>> Do we have this contract written somewhere?
>> 
>> 
>>> 1 апр. 2020 г., в 11:35, Anton Vinogradov  написал(а):
>>> 
>>> Folks,
>>> 
>>> The question is not about "+1" or "-1".
>>> The question is "why?".
>>> 
>>> This looks like some special feature to solve some special security issue
>>> but may be just a bad/broken/obsolete/unrefactored code.
>>> Changing this semantic we'll fix bad code or break some contract. 50% to
>>> 50%.
>>> 
>>> Let's keep REST case as is for now but start an investigation to gain
>>> security consistent across all APIs, if possible.
>>> 
>>> On Tue, Mar 31, 2020 at 4:59 PM Andrey Kuznetsov 
>> wrote:
>>> 
 I'd prefer marking ADMIN_CACHE as deprecated, but postpone its removal
>> from
 GridRestProcessor till next Ignte release (2.10 or 3.0?). For now we
>> could
 just add checks for CACHE_CREATE / CACHE_DESTROY there along
 with ADMIN_CACHE.
 
 вт, 31 мар. 2020 г. в 12:30, Nikolay Izhikov :
 
> Hello, Sergey.
> 
> 
> I’m +1 to make this change.
> 
> I think we should make security consistent across all APIs.
> 
>> 31 марта 2020 г., в 12:14, Sergei Ryzhov 
> написал(а):
>> 
>> Hello!
>> Now the work of permissions for API IgniteClient and REST is
>> different.
>> To create/delete a cache:
>> IgniteClient authorises
> CACHE_CREATE/CACHE_DESTROY.(GridCacheProcessor#authorizeCacheCreate <
> 
 
>> https://github.com/apache/ignite/blob/aefad946ebd7720f81b460aa39e205c10dc24b26/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheProcessor.java#L3983
> ,
> authorizeCacheDestroy <
> 
 
>> https://github.com/apache/ignite/blob/aefad946ebd7720f81b460aa39e205c10dc24b26/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheProcessor.java#L3973
>> )
>> REST authorises ADMIN_CACHE.(GridRestProcessor#authorize <
> 
 
>> https://github.com/apache/ignite/blob/aefad946ebd7720f81b460aa39e205c10dc24b26/modules/core/src/main/java/org/apache/ignite/internal/processors/rest/GridRestProcessor.java#L841
>> )
>> I think this is inconsistent.
>> 
>> I suggest ADMIN_CACHE mark @Deprecated
>> and replace it in the GridRestProcessor with CACHE_CREATE /
> CACHE_DESTROY
>> while maintaining backward compatibility for ADMIN_CACHE.
>> 
>> This will allow us to remove ADMIN_CACHE in the future.
>> 
>> 
>> 
>> Sergei Ryzhov
>> s.vi.ryz...@gmail.com
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
> 
> 
 
 --
 Best regards,
 Andrey Kuznetsov.
 
>> 
>> 

Re: Inconsistent API IgniteClient and REST

[Anton Vinogradov]

That's exactly the question I have.
Since no tests and documentation can be found - then we have no contract
and it's safe to change the semantic.
But we should recheck search results twice because we're talking about the
security.

On Wed, Apr 1, 2020 at 11:44 AM Nikolay Izhikov  wrote:

> Hello, Anton.
>
> What is «contract» for you?
> Do we have this contract written somewhere?
>
>
> > 1 апр. 2020 г., в 11:35, Anton Vinogradov  написал(а):
> >
> > Folks,
> >
> > The question is not about "+1" or "-1".
> > The question is "why?".
> >
> > This looks like some special feature to solve some special security issue
> > but may be just a bad/broken/obsolete/unrefactored code.
> > Changing this semantic we'll fix bad code or break some contract. 50% to
> > 50%.
> >
> > Let's keep REST case as is for now but start an investigation to gain
> > security consistent across all APIs, if possible.
> >
> > On Tue, Mar 31, 2020 at 4:59 PM Andrey Kuznetsov 
> wrote:
> >
> >> I'd prefer marking ADMIN_CACHE as deprecated, but postpone its removal
> from
> >> GridRestProcessor till next Ignte release (2.10 or 3.0?). For now we
> could
> >> just add checks for CACHE_CREATE / CACHE_DESTROY there along
> >> with ADMIN_CACHE.
> >>
> >> вт, 31 мар. 2020 г. в 12:30, Nikolay Izhikov :
> >>
> >>> Hello, Sergey.
> >>>
> >>>
> >>> I’m +1 to make this change.
> >>>
> >>> I think we should make security consistent across all APIs.
> >>>
>  31 марта 2020 г., в 12:14, Sergei Ryzhov 
> >>> написал(а):
> 
>  Hello!
>  Now the work of permissions for API IgniteClient and REST is
> different.
>  To create/delete a cache:
>  IgniteClient authorises
> >>> CACHE_CREATE/CACHE_DESTROY.(GridCacheProcessor#authorizeCacheCreate <
> >>>
> >>
> https://github.com/apache/ignite/blob/aefad946ebd7720f81b460aa39e205c10dc24b26/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheProcessor.java#L3983
> >>> ,
> >>> authorizeCacheDestroy <
> >>>
> >>
> https://github.com/apache/ignite/blob/aefad946ebd7720f81b460aa39e205c10dc24b26/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheProcessor.java#L3973
>  )
>  REST authorises ADMIN_CACHE.(GridRestProcessor#authorize <
> >>>
> >>
> https://github.com/apache/ignite/blob/aefad946ebd7720f81b460aa39e205c10dc24b26/modules/core/src/main/java/org/apache/ignite/internal/processors/rest/GridRestProcessor.java#L841
>  )
>  I think this is inconsistent.
> 
>  I suggest ADMIN_CACHE mark @Deprecated
>  and replace it in the GridRestProcessor with CACHE_CREATE /
> >>> CACHE_DESTROY
>  while maintaining backward compatibility for ADMIN_CACHE.
> 
>  This will allow us to remove ADMIN_CACHE in the future.
> 
> 
> 
>  Sergei Ryzhov
>  s.vi.ryz...@gmail.com
> 
> 
> 
> 
> 
> 
> 
> 
> >>>
> >>>
> >>
> >> --
> >> Best regards,
> >>  Andrey Kuznetsov.
> >>
>
>

[jira] [Created] (IGNITE-12851) Inconsistent API IgniteClient and REST

[Sergei Ryzhov (Jira)]

Sergei Ryzhov created IGNITE-12851:
--

 Summary: Inconsistent API IgniteClient and REST
 Key: IGNITE-12851
 URL: https://issues.apache.org/jira/browse/IGNITE-12851
 Project: Ignite
  Issue Type: Wish
  Components: security
Affects Versions: 2.8
Reporter: Sergei Ryzhov
Assignee: Sergei Ryzhov


Now the work of permissions for API IgniteClient and REST is different.
To create/delete a cache:
IgniteClient authorises 
CACHE_CREATE/CACHE_DESTROY.(GridCacheProcessor#authorizeCacheCreate, 
authorizeCacheDestroy)
REST authorises ADMIN_CACHE.(GridRestProcessor#authorize)
I think this is inconsistent.

I suggest ADMIN_CACHE mark @Deprecated
and replace it in the GridRestProcessor with CACHE_CREATE / CACHE_DESTROY 
while maintaining backward compatibility for ADMIN_CACHE.

This will allow us to remove ADMIN_CACHE in the future.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Inconsistent API IgniteClient and REST

[Nikolay Izhikov]

Hello, Anton.

What is «contract» for you?
Do we have this contract written somewhere?


> 1 апр. 2020 г., в 11:35, Anton Vinogradov  написал(а):
> 
> Folks,
> 
> The question is not about "+1" or "-1".
> The question is "why?".
> 
> This looks like some special feature to solve some special security issue
> but may be just a bad/broken/obsolete/unrefactored code.
> Changing this semantic we'll fix bad code or break some contract. 50% to
> 50%.
> 
> Let's keep REST case as is for now but start an investigation to gain
> security consistent across all APIs, if possible.
> 
> On Tue, Mar 31, 2020 at 4:59 PM Andrey Kuznetsov  wrote:
> 
>> I'd prefer marking ADMIN_CACHE as deprecated, but postpone its removal from
>> GridRestProcessor till next Ignte release (2.10 or 3.0?). For now we could
>> just add checks for CACHE_CREATE / CACHE_DESTROY there along
>> with ADMIN_CACHE.
>> 
>> вт, 31 мар. 2020 г. в 12:30, Nikolay Izhikov :
>> 
>>> Hello, Sergey.
>>> 
>>> 
>>> I’m +1 to make this change.
>>> 
>>> I think we should make security consistent across all APIs.
>>> 
 31 марта 2020 г., в 12:14, Sergei Ryzhov 
>>> написал(а):
 
 Hello!
 Now the work of permissions for API IgniteClient and REST is different.
 To create/delete a cache:
 IgniteClient authorises
>>> CACHE_CREATE/CACHE_DESTROY.(GridCacheProcessor#authorizeCacheCreate <
>>> 
>> https://github.com/apache/ignite/blob/aefad946ebd7720f81b460aa39e205c10dc24b26/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheProcessor.java#L3983
>>> ,
>>> authorizeCacheDestroy <
>>> 
>> https://github.com/apache/ignite/blob/aefad946ebd7720f81b460aa39e205c10dc24b26/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheProcessor.java#L3973
 )
 REST authorises ADMIN_CACHE.(GridRestProcessor#authorize <
>>> 
>> https://github.com/apache/ignite/blob/aefad946ebd7720f81b460aa39e205c10dc24b26/modules/core/src/main/java/org/apache/ignite/internal/processors/rest/GridRestProcessor.java#L841
 )
 I think this is inconsistent.
 
 I suggest ADMIN_CACHE mark @Deprecated
 and replace it in the GridRestProcessor with CACHE_CREATE /
>>> CACHE_DESTROY
 while maintaining backward compatibility for ADMIN_CACHE.
 
 This will allow us to remove ADMIN_CACHE in the future.
 
 
 
 Sergei Ryzhov
 s.vi.ryz...@gmail.com
 
 
 
 
 
 
 
 
>>> 
>>> 
>> 
>> --
>> Best regards,
>>  Andrey Kuznetsov.
>> 

Re: Inconsistent API IgniteClient and REST

[Anton Vinogradov]

Folks,

The question is not about "+1" or "-1".
The question is "why?".

This looks like some special feature to solve some special security issue
but may be just a bad/broken/obsolete/unrefactored code.
Changing this semantic we'll fix bad code or break some contract. 50% to
50%.

Let's keep REST case as is for now but start an investigation to gain
security consistent across all APIs, if possible.

On Tue, Mar 31, 2020 at 4:59 PM Andrey Kuznetsov  wrote:

> I'd prefer marking ADMIN_CACHE as deprecated, but postpone its removal from
> GridRestProcessor till next Ignte release (2.10 or 3.0?). For now we could
> just add checks for CACHE_CREATE / CACHE_DESTROY there along
> with ADMIN_CACHE.
>
> вт, 31 мар. 2020 г. в 12:30, Nikolay Izhikov :
>
> > Hello, Sergey.
> >
> >
> > I’m +1 to make this change.
> >
> > I think we should make security consistent across all APIs.
> >
> > > 31 марта 2020 г., в 12:14, Sergei Ryzhov 
> > написал(а):
> > >
> > > Hello!
> > > Now the work of permissions for API IgniteClient and REST is different.
> > > To create/delete a cache:
> > > IgniteClient authorises
> > CACHE_CREATE/CACHE_DESTROY.(GridCacheProcessor#authorizeCacheCreate <
> >
> https://github.com/apache/ignite/blob/aefad946ebd7720f81b460aa39e205c10dc24b26/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheProcessor.java#L3983
> >,
> > authorizeCacheDestroy <
> >
> https://github.com/apache/ignite/blob/aefad946ebd7720f81b460aa39e205c10dc24b26/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheProcessor.java#L3973
> > >)
> > > REST authorises ADMIN_CACHE.(GridRestProcessor#authorize <
> >
> https://github.com/apache/ignite/blob/aefad946ebd7720f81b460aa39e205c10dc24b26/modules/core/src/main/java/org/apache/ignite/internal/processors/rest/GridRestProcessor.java#L841
> > >)
> > > I think this is inconsistent.
> > >
> > > I suggest ADMIN_CACHE mark @Deprecated
> > > and replace it in the GridRestProcessor with CACHE_CREATE /
> > CACHE_DESTROY
> > > while maintaining backward compatibility for ADMIN_CACHE.
> > >
> > > This will allow us to remove ADMIN_CACHE in the future.
> > >
> > >
> > >
> > > Sergei Ryzhov
> > > s.vi.ryz...@gmail.com
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
> >
>
> --
> Best regards,
>   Andrey Kuznetsov.
>