Re: [DISCUSSION] IEP-100: Compute API: Phase 1 - Simple remote job execution
> Creating a compute job for each record would quickly get unmanageable and > broadcast isn’t equivalent as it doesn’t take into account the possibility of > partitions migrating during calculations. Of course creating a compute job for each record is a waste of resources. Even in current really simple API a user can execute bulk operation, it is the responsibility of the job's code. But we also should provide API which simplifies some WAs which user should use in current design. Thanks for highlighting these important aspects of Compute API. On Mon, May 15, 2023 at 5:39 PM Stephen Darlington wrote: > > Right, it doesn’t need to be the same — exposing partitions in this manner is > weird — but I think we need to support bulk operations in some capacity. > Creating a compute job for each record would quickly get unmanageable and > broadcast isn’t equivalent as it doesn’t take into account the possibility of > partitions migrating during calculations. > > > On 15 May 2023, at 14:30, Andrey Gura wrote: > > > >> In Ignite 2, for bulk operations there’s an affinityRun method that takes > >> a partition as a parameter. I don’t see an equivalent here. Is one in the > >> plan? > > > > Ideally users don't need to know a partition for execution of > > computation and the right tool for it is colocation. Method with > > partitions was introduced in order to execute scan queries on specific > > partitions and it is some kind of hack. I believe this approach should > > be revised if possible. > > > >> Are other APIs like apply, async methods, map-reduce being considered in > >> later phases? > > > > All methods are async and return CompletableFuture. Yes, Map/reduce, > > etc will be designed in later phases. > > > > On Mon, May 15, 2023 at 4:08 PM Stephen Darlington > > wrote: > >> > >> In Ignite 2, for bulk operations there’s an affinityRun method that takes > >> a partition as a parameter. I don’t see an equivalent here. Is one in the > >> plan? > >> > >> Are other APIs like apply, async methods, map-reduce being considered in > >> later phases? > >> > >>> On 15 May 2023, at 12:52, Andrey Gura wrote: > >>> > >>> Hi, Igniters! > >>> > >>> Please take a look at the first phase of proposal for Apache ignite 3 > >>> Compute API: Simple remote job execution [1]. > >>> > >>> While most issues are already resolved due to simplicity of the proposal > >>> any feedback and ideas will be really useful for the next phases. > >>> > >>> Thanks! > >>> > >>> [1] > >>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-100%3A+Compute+API%3A+Phase+1+-+Simple+remote+job+execution > >> >
Re: [DISCUSSION] IEP-100: Compute API: Phase 1 - Simple remote job execution
Right, it doesn’t need to be the same — exposing partitions in this manner is weird — but I think we need to support bulk operations in some capacity. Creating a compute job for each record would quickly get unmanageable and broadcast isn’t equivalent as it doesn’t take into account the possibility of partitions migrating during calculations. > On 15 May 2023, at 14:30, Andrey Gura wrote: > >> In Ignite 2, for bulk operations there’s an affinityRun method that takes a >> partition as a parameter. I don’t see an equivalent here. Is one in the plan? > > Ideally users don't need to know a partition for execution of > computation and the right tool for it is colocation. Method with > partitions was introduced in order to execute scan queries on specific > partitions and it is some kind of hack. I believe this approach should > be revised if possible. > >> Are other APIs like apply, async methods, map-reduce being considered in >> later phases? > > All methods are async and return CompletableFuture. Yes, Map/reduce, > etc will be designed in later phases. > > On Mon, May 15, 2023 at 4:08 PM Stephen Darlington > wrote: >> >> In Ignite 2, for bulk operations there’s an affinityRun method that takes a >> partition as a parameter. I don’t see an equivalent here. Is one in the plan? >> >> Are other APIs like apply, async methods, map-reduce being considered in >> later phases? >> >>> On 15 May 2023, at 12:52, Andrey Gura wrote: >>> >>> Hi, Igniters! >>> >>> Please take a look at the first phase of proposal for Apache ignite 3 >>> Compute API: Simple remote job execution [1]. >>> >>> While most issues are already resolved due to simplicity of the proposal >>> any feedback and ideas will be really useful for the next phases. >>> >>> Thanks! >>> >>> [1] >>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-100%3A+Compute+API%3A+Phase+1+-+Simple+remote+job+execution >>
Re: [DISCUSSION] IEP-100: Compute API: Phase 1 - Simple remote job execution
> In Ignite 2, for bulk operations there’s an affinityRun method that takes a > partition as a parameter. I don’t see an equivalent here. Is one in the plan? Ideally users don't need to know a partition for execution of computation and the right tool for it is colocation. Method with partitions was introduced in order to execute scan queries on specific partitions and it is some kind of hack. I believe this approach should be revised if possible. > Are other APIs like apply, async methods, map-reduce being considered in > later phases? All methods are async and return CompletableFuture. Yes, Map/reduce, etc will be designed in later phases. On Mon, May 15, 2023 at 4:08 PM Stephen Darlington wrote: > > In Ignite 2, for bulk operations there’s an affinityRun method that takes a > partition as a parameter. I don’t see an equivalent here. Is one in the plan? > > Are other APIs like apply, async methods, map-reduce being considered in > later phases? > > > On 15 May 2023, at 12:52, Andrey Gura wrote: > > > > Hi, Igniters! > > > > Please take a look at the first phase of proposal for Apache ignite 3 > > Compute API: Simple remote job execution [1]. > > > > While most issues are already resolved due to simplicity of the proposal > > any feedback and ideas will be really useful for the next phases. > > > > Thanks! > > > > [1] > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-100%3A+Compute+API%3A+Phase+1+-+Simple+remote+job+execution >
Re: [DISCUSSION] IEP-100: Compute API: Phase 1 - Simple remote job execution
In Ignite 2, for bulk operations there’s an affinityRun method that takes a partition as a parameter. I don’t see an equivalent here. Is one in the plan? Are other APIs like apply, async methods, map-reduce being considered in later phases? > On 15 May 2023, at 12:52, Andrey Gura wrote: > > Hi, Igniters! > > Please take a look at the first phase of proposal for Apache ignite 3 > Compute API: Simple remote job execution [1]. > > While most issues are already resolved due to simplicity of the proposal > any feedback and ideas will be really useful for the next phases. > > Thanks! > > [1] > https://cwiki.apache.org/confluence/display/IGNITE/IEP-100%3A+Compute+API%3A+Phase+1+-+Simple+remote+job+execution
[DISCUSSION] IEP-100: Compute API: Phase 1 - Simple remote job execution
Hi, Igniters! Please take a look at the first phase of proposal for Apache ignite 3 Compute API: Simple remote job execution [1]. While most issues are already resolved due to simplicity of the proposal any feedback and ideas will be really useful for the next phases. Thanks! [1] https://cwiki.apache.org/confluence/display/IGNITE/IEP-100%3A+Compute+API%3A+Phase+1+-+Simple+remote+job+execution
