Hi,
Luke - Thank you for sharing the details for the portability layer for
Flink, Samza and Spark. I will look into them and will reach out if I have
any questions.
Val - Thank you for your response, yes I am planning to run the beam
pipeline using Ignite compute engine in async run. Here is a sample code
for the run method.
IgnitePipelineResult pipelineResult = new IgnitePipelineResult(job,
metricsAccumulator);
ComputeTaskFuture<Void> computeTaskFuture =
ignite.compute().withAsync().run(
(r, f) -> {
pipelineResult.freeze(f);
metricsAccumulator.destroy();
ignite.shutdown();
});
pipelineResult.setComputeFuture(asyncCompute.future());
return pipelineResult;
My understanding is for failover scenarios we will need to map the job
state from Ignite known state to Beam Job state, an example like in
JetPipelineResult
https://github.com/apache/beam/blob/master/runners/jet/src/main/java/org/apache/beam/runners/jet/JetPipelineResult.java#L68-L90
Regards,
Saikat
On Mon, Aug 17, 2020 at 2:27 PM Valentin Kulichenko <
[email protected]> wrote:
> Hi Saikat,
>
> This sounds very interesting - I've been thinking about how Ignite compute
> engine could be enhanced, and integration with Apache Beam is one of the
> options I have in mind. Can you please describe how you plan to implement
> this? Will it run on top of the Ignite Compute Grid? How are you going to
> handle the failover, especially in the case of async pipeline execution?
>
> -Val
>
> On Sat, Aug 15, 2020 at 12:50 PM Saikat Maitra <[email protected]>
> wrote:
>
> > Hi,
> >
> > I have been working on implementing the Apache Ignite Runner to run
> Apache
> > Beam pipeline. I have created IgniteRunner and IgnitePipelineOptions. I
> > have implemented the normalize pipeline method and currently working on
> run
> > method implementation for Pipeline and IgnitePipelineTranslator.
> >
> > Jira : https://issues.apache.org/jira/browse/BEAM-9045
> >
> > PR : https://github.com/apache/beam/pull/12593
> >
> > Please review and feel free to share any feedback or questions.
> >
> > Regards,
> > Saikat
> >
>
