Re: Changing public IgniteCompute API to improve changes in 5037 ticket

hi all I'd like to ask if somebody is going to continue with a proposal for final API design here. We have a use-case were map-reduce API with a built-in affinity feature would be a big benefit for us (project is covering 26 European countries now). Otherwise we have to think how to implement it

Re: Changing public IgniteCompute API to improve changes in 5037 ticket

> Is the final design documented in the ticket? No, since discussion in progress. > Each job should somehow know the partition it's running over, on example, > for setting it for ScanQuery or SqlQuery over local partition. Seems, partition reserve() is enough to guarantee that partition will not

Re: Changing public IgniteCompute API to improve changes in 5037 ticket

Is the final design documented in the ticket? On Sat, Jul 29, 2017 at 5:04 AM, Alexei Scherbakov < alexey.scherbak...@gmail.com> wrote: > Anton, > > I'm fine with proposed solution using AffinityComputeTask. > > Please take care of issues pointed by me while implementing. > > Each job should

Re: Changing public IgniteCompute API to improve changes in 5037 ticket

Anton, I'm fine with proposed solution using AffinityComputeTask. Please take care of issues pointed by me while implementing. Each job should somehow know the partition it's running over, on example, for setting it for ScanQuery or SqlQuery over local partition. affinityExecute must support

Re: Changing public IgniteCompute API to improve changes in 5037 ticket

Anton, This seems to be a completely separate issue. I don't see how it can be fixed by adding new APIs. -Val On Wed, Jul 26, 2017 at 3:56 AM, Anton Vinogradov wrote: > Val, > > AFAIK, affinityRun/Call has guarantee to be successfully executed on > unstable topology

Re: Changing public IgniteCompute API to improve changes in 5037 ticket

Alexey, Is there exact use case that is currently not supported? I really would like to see one, because such a big API change should add clear value. ComputeGrid is not used very often, and so far I've never seen any questions from users about using it in conjunction with affinity collocation.

Re: Changing public IgniteCompute API to improve changes in 5037 ticket

Anton, How does topology change break this functionality? Closures executed with affinityRun/Call fail over in the same way as any ComputeJob. -Val On Tue, Jul 25, 2017 at 5:48 AM, Anton Vinogradov wrote: > Alexei, > > > How would task know the partition it is

Re: Changing public IgniteCompute API to improve changes in 5037 ticket

Alexei, > How would task know the partition it is running over ? Not sure it necessary. You'll create pair partition-job at task's map phase. > How can I assign task for each cache partition ? Just implement map method generates map with size equals to partition count. > How can I enforce

Re: Changing public IgniteCompute API to improve changes in 5037 ticket

Please read job instead task 2017-07-25 15:20 GMT+03:00 Alexei Scherbakov : > Main point of the issue is to provide clean API for working with > computations requiring data collocation > > affinityCall/Run provide the ability to run closure near data, but >

Re: Changing public IgniteCompute API to improve changes in 5037 ticket

Main point of the issue is to provide clean API for working with computations requiring data collocation affinityCall/Run provide the ability to run closure near data, but map/reduce API is a way reacher: continuous mapping, task session, etc. As for proposed API, I do not understand fully how

Re: Changing public IgniteCompute API to improve changes in 5037 ticket

Val, Sure, we can, but we'd like to use map/reduce without fearing that topology can change. On Mon, Jul 24, 2017 at 11:17 PM, Valentin Kulichenko < valentin.kuliche...@gmail.com> wrote: > Anton, > > You can call affinityCallAsync multiple times and then reduce locally. > > -Val > > On Mon, Jul

Re: Changing public IgniteCompute API to improve changes in 5037 ticket

Anton, You can call affinityCallAsync multiple times and then reduce locally. -Val On Mon, Jul 24, 2017 at 3:05 AM, Anton Vinogradov wrote: > Val, > > > What is the use case for which current affinityRun/Call API doesn't work? > It does not work for map/reduce. > >

Re: Changing public IgniteCompute API to improve changes in 5037 ticket

Val, > What is the use case for which current affinityRun/Call API doesn't work? It does not work for map/reduce. On Fri, Jul 21, 2017 at 11:42 PM, Valentin Kulichenko < valentin.kuliche...@gmail.com> wrote: > Maxim, > > The issue is that it's currently assumed to support job mapping, but it >

Re: Changing public IgniteCompute API to improve changes in 5037 ticket

Maxim, The issue is that it's currently assumed to support job mapping, but it actually doesn't. However, I agree that AffinityKeyMapped annotation doesn't fit the use case well. Let's fix documentation and JavaDoc then. As for the proposed API, it's overcomplicated, took me 15 minutes to

Re: Changing public IgniteCompute API to improve changes in 5037 ticket

Valentin, The author of tiket wants to see to provide some API allows to map ComputeJobs to partitions or keys. If we use @AffinityKeyMapped then you need to enter the cache name parameter, I think this is not convenient for the user. Therefore, I propose to extend the existing API. Having

Re: Changing public IgniteCompute API to improve changes in 5037 ticket

Hi Max, This ticket doesn't assume any API changes, it's about broken functionality. I would start with checking what tests we have for @AffinityKeyMapped and creating missing one. From what I understand functionality is broken completely or almost completely, so I guess testing coverage is very

Changing public IgniteCompute API to improve changes in 5037 ticket

Igniters, jira: https://issues.apache.org/jira/browse/IGNITE-5037 How do you look to solve this ticket by adding two methods to the public IgniteCompute API? @IgniteAsyncSupported public void affinityRun(@NotNull Collection cacheNames,