Yes, you can use Flink's local execution mode, which is the default if
you don't provide any settings. A cluster should not be necessary to
complete the integration. Ideally, it should work out of the box :)
However, I'm first trying to solve the complicated issue of grouping records
together in Beam in a safe way so that they can batched up
I'm not sure what your use case is but Beam does batching by default.
The batches are called bundles. The Flink Runner supports setting the
bundle size.
Cheers,
Max
On 12.02.19 12:20, Matt Casters wrote:
Yes, Flink is obviously the next target. I'm not expecting too many
issues there beyond getting a cluster set up to test on. I read you can
run the Flink Runner locally so that will help a lot in testing.
However, I'm first trying to solve the complicated issue of grouping
records together in Beam in a safe way so that they can batched up.
Batching up is really important for fast loading into a lot of output
targets. I'll probably use some group by behind the scenes or something
like that, need to think about that.
Having the ability to re-use the existing Kettle steps without having to
write new code is really key.
Once that is done (in a few weeks) I'll give Flink a shot.
Cheers,
Matt
Op di 12 feb. 2019 om 12:02 schreef Maximilian Michels <[email protected]
<mailto:[email protected]>>:
@Dan: Thanks for sharing the presentation. Kettle is a great way to
make
Beam more accessible.
@Matt: Thanks for the plug. It's good to hear you enjoyed it. I think
the link to your slides got messed up: http://beam.kettle.be
Are you planning to add execution via the Flink Runner to Kettle?
Saw in
the presentation that you already support Direct, Spark, and Dataflow.
On 11.02.19 20:50, Matt Casters wrote:
> By the way, Maximilian, I linked and plugged your wonderful FOSDEM
> presentation in my slides http://beam kettle.be
<http://kettle.be> <http://kettle.be> slide
> 19. If you mind, let me know and I'll get it out of the slides.
In any
> case, great content worth promoting I thought.
>
> Op wo 6 feb. 2019 18:03 schreef Maximilian Michels
<[email protected] <mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]>>:
>
> Hi Dan,
>
> Thanks for the info. Would be great to share a video of the
> presentation.
>
> Cheers,
> Max
>
> On 30.01.19 10:00, Dan wrote:
> > Hi, in just over a week you're all welcome to come and see
the very
> > first public reveal of Kettle running on beam! (Including
spark,
> > dataflow and flink support)
> >
> >
https://www.meetup.com/Pentaho-London-User-Group/events/256773962/
> >
> > So this ingenious integration combines the power of visual
> development,
> > with the platform agnostic benefits of beam - impressive
stuff. No
> > vendor lock-in here!
> >
> >
> > See you there!
> > Dan
>
