Hi,

Luke - Thank you for sharing the details for the portability layer for
Flink, Samza and Spark. I will look into them and will reach out if I have
any questions.

Val - Thank you for your response, yes I am planning to run the beam
pipeline using Ignite compute engine in async run. Here is a sample code
for the run method.

IgnitePipelineResult pipelineResult = new IgnitePipelineResult(job,
metricsAccumulator);
    ComputeTaskFuture<Void> computeTaskFuture =
        ignite.compute().withAsync().run(
                (r, f) -> {
                  pipelineResult.freeze(f);
                  metricsAccumulator.destroy();
                  ignite.shutdown();
                });
    pipelineResult.setComputeFuture(asyncCompute.future());

    return pipelineResult;


My understanding is for failover scenarios we will need to map the job
state from Ignite known state to Beam Job state, an example like in
JetPipelineResult
https://github.com/apache/beam/blob/master/runners/jet/src/main/java/org/apache/beam/runners/jet/JetPipelineResult.java#L68-L90

Regards,
Saikat






On Mon, Aug 17, 2020 at 2:27 PM Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Hi Saikat,
>
> This sounds very interesting - I've been thinking about how Ignite compute
> engine could be enhanced, and integration with Apache Beam is one of the
> options I have in mind. Can you please describe how you plan to implement
> this? Will it run on top of the Ignite Compute Grid? How are you going to
> handle the failover, especially in the case of async pipeline execution?
>
> -Val
>
> On Sat, Aug 15, 2020 at 12:50 PM Saikat Maitra <saikat.mai...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I have been working on implementing the Apache Ignite Runner to run
> Apache
> > Beam pipeline. I have created IgniteRunner and IgnitePipelineOptions. I
> > have implemented the normalize pipeline method and currently working on
> run
> > method implementation for Pipeline and IgnitePipelineTranslator.
> >
> > Jira : https://issues.apache.org/jira/browse/BEAM-9045
> >
> > PR : https://github.com/apache/beam/pull/12593
> >
> > Please review and feel free to share any feedback or questions.
> >
> > Regards,
> > Saikat
> >
>

Reply via email to