Main point of the issue is to provide clean API for working with
computations requiring data collocation
affinityCall/Run provide the ability to run closure near data, but
map/reduce API is a way reacher: continuous mapping, task session, etc.
As for proposed API, I do not understand fully how it solves the problem.
Maxim, please provide detailed javadoc for each method and each argument
for presented API, and the answers to the following questions:
1. How would task know the partition it is running over ?
2. How can I assign task for each cache partition ?
3. How can I enforce partition reservation if task works with multiple
caches at once ?
2017-07-25 12:30 GMT+03:00 Anton Vinogradov <avinogra...@gridgain.com>:
> Val,
>
> Sure, we can, but we'd like to use map/reduce without fearing that topology
> can change.
>
> On Mon, Jul 24, 2017 at 11:17 PM, Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
> > Anton,
> >
> > You can call affinityCallAsync multiple times and then reduce locally.
> >
> > -Val
> >
> > On Mon, Jul 24, 2017 at 3:05 AM, Anton Vinogradov <
> > avinogra...@gridgain.com>
> > wrote:
> >
> > > Val,
> > >
> > > > What is the use case for which current affinityRun/Call API doesn't
> > work?
> > > It does not work for map/reduce.
> > >
> > > On Fri, Jul 21, 2017 at 11:42 PM, Valentin Kulichenko <
> > > valentin.kuliche...@gmail.com> wrote:
> > >
> > > > Maxim,
> > > >
> > > > The issue is that it's currently assumed to support job mapping, but
> it
> > > > actually doesn't. However, I agree that AffinityKeyMapped annotation
> > > > doesn't fit the use case well. Let's fix documentation and JavaDoc
> > then.
> > > >
> > > > As for the proposed API, it's overcomplicated, took me 15 minutes to
> > > > understand what it does :)
> > > >
> > > > What is the use case for which current affinityRun/Call API doesn't
> > work?
> > > >
> > > > -Val
> > > >
> > > > On Fri, Jul 21, 2017 at 5:57 AM, Kozlov Maxim <dreamx....@gmail.com>
> > > > wrote:
> > > >
> > > > > Valentin,
> > > > >
> > > > > The author of tiket wants to see to provide some API allows to map
> > > > > ComputeJobs to partitions or keys. If we use @AffinityKeyMapped
> then
> > > you
> > > > > need to enter the cache name parameter, I think this is not
> > convenient
> > > > for
> > > > > the user. Therefore, I propose to extend the existing API.
> > > > > Having consulted with Anton V. decided to make a separate interface
> > > > > ReducibleTask, which will allow us to have different map logic at
> > each
> > > > > inheritor.
> > > > >
> > > > > Old method, allows to map to node
> > > > > public interface ComputeTask<T, R> extends ReducibleTask<R> {
> > > > > @Nullable public Map<? extends ComputeJob, ClusterNode>
> > > > > map(List<ClusterNode> subgrid, @Nullable T arg) throws
> > IgniteException;
> > > > > }
> > > > >
> > > > > Brand new method with mapping to partitions, which solves topology
> > > change
> > > > > issues.
> > > > > public interface AffinityComputeTask<T, R> extends
> ReducibleTask<R> {
> > > > > @Nullable public Map<? extends ComputeJob, Integer>
> > > > map(@NotnullString
> > > > > cacheName, List<Integer> partIds, @Nullable T arg) throws
> > > > IgniteException;
> > > > > }
> > > > >
> > > > > public interface ReducibleTask<R> extends Serializable {
> > > > > public ComputeJobResultPolicy result(ComputeJobResult res,
> > > > > List<ComputeJobResult> rcvd) throws IgniteException;
> > > > >
> > > > > @Nullable public R reduce(List<ComputeJobResult> results)
> throws
> > > > > IgniteException;
> > > > > }
> > > > >
> > > > > We also need to implement AffinityComputeTaskAdapter and
> > > > > AffinityComputeTaskSplitAdapter, for implementation by default. It
> > is
> > > > > right?
> > > > >
> > > > > In the IgniteCompute add:
> > > > >
> > > > > @IgniteAsyncSupported
> > > > > public <T, R> R affinityExecute(Class<? extends
> > AffinityComputeTask<T,
> > > > R>>
> > > > > taskCls, List<Integer> partIds, @Nullable T arg) throws
> > > IgniteException;
> > > > > @IgniteAsyncSupported
> > > > > public <T, R> R affinityExecute(AffinityComputeTask<T, R> task,
> > > > > List<Integer> partIds, @Nullable T arg) throws IgniteException;
> > > > >
> > > > > public <T, R> ComputeTaskFuture<R> affinityExecuteAsync(Class<?
> > extends
> > > > > AffinityComputeTask<T, R>> taskCls, List<Integer> partIds,
> @Nullable
> > T
> > > > arg)
> > > > > throws IgniteException;
> > > > > public <T, R> ComputeTaskFuture<R> affinityExecuteAsync(
> > > > AffinityComputeTask<T,
> > > > > R> task, List<Integer> partIds, @Nullable T arg) throws
> > > IgniteException;
> > > > >
> > > > >
> > > > > How do you like this idea or do you insist that you need to use
> > > > > @AffinityKeyMapped to solve the problem?
> > > > >
> > > > >
> > > > > > 13 июля 2017 г., в 6:36, Valentin Kulichenko <
> > > > > valentin.kuliche...@gmail.com> написал(а):
> > > > > >
> > > > > > Hi Max,
> > > > > >
> > > > > > This ticket doesn't assume any API changes, it's about broken
> > > > > > functionality. I would start with checking what tests we have
> > > > > > for @AffinityKeyMapped and creating missing one. From what I
> > > understand
> > > > > > functionality is broken completely or almost completely, so I
> guess
> > > > > testing
> > > > > > coverage is very weak there.
> > > > > >
> > > > > > -Val
> > > > > >
> > > > > > On Wed, Jul 12, 2017 at 4:27 PM, Kozlov Maxim <
> > dreamx....@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > >> Igniters,
> > > > > >>
> > > > > >> jira: https://issues.apache.org/jira/browse/IGNITE-5037 <
> > > > > >> https://issues.apache.org/jira/browse/IGNITE-5037>
> > > > > >> How do you look to solve this ticket by adding two methods to
> the
> > > > public
> > > > > >> IgniteCompute API?
> > > > > >>
> > > > > >> @IgniteAsyncSupported
> > > > > >> public void affinityRun(@NotNull Collection<String> cacheNames,
> > > > > >> Collection<Object> keys, IgniteRunnable job)
> > > > > >> throws IgniteException;
> > > > > >>
> > > > > >> @IgniteAsyncSupported
> > > > > >> public <R> R affinityCall(@NotNull Collection<String>
> cacheNames,
> > > > > >> Collection<Object> keys, IgniteCallable<R> job)
> > > > > >> throws IgniteException;
> > > > > >>
> > > > > >> There is also a question of how to act when changing the
> topology
> > > > during
> > > > > >> the execution of the job.
> > > > > >> 1) complete with an exception;
> > > > > >> 2) stop execution and wait until the topology is rebuilt and
> > > continue
> > > > > >> execution;
> > > > > >>
> > > > > >> I think the second way, do you think?
> > > > > >>
> > > > > >> --
> > > > > >> Best Regards,
> > > > > >> Max K.
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > >
> > > > > --
> > > > > Best Regards,
> > > > > Max K.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
--
Best regards,
Alexei Scherbakov
