Oh yes! I now remember that Hop uses Beam to run on Flink. Anyhow, the
basic checkpoint options are there. Very true, that we might have to add
any new ones. Maybe a chance for Hans-Peter to contribute? :)
Cheers,
Max
On 27.01.22 11:55, Matt Casters wrote:
You are correct. Hop uses the Flink runner of Apache Beam though and we
did upgrade to the Flink version 1.13 runner in our latest Hop version
so there's always the chance that new options became available in Flink
which are not yet reflected in the runner or in Hop.
There's really no good communications channel that I know of to signal
downstream projects of available changes so we just keep an eye out for
them.
Cheers,
Matt
Op do 27 jan. 2022 11:49 schreef Maximilian Michels <[email protected]
<mailto:[email protected]>>:
It looks to me, Hans-Peter's question was not about Beam but Flink. The
provided link should answer his question regarding setting the
checkpoint interval.
-Max
On 27.01.22 11:21, Matt Casters wrote:
> It can happen that we lag a bit behind when Apache Beam adds
options for
> specific runners.
> So always let us know if there's an option that's not available.
> In fact, if the option is not available in Beam either we can
create a
> change against Apache Beam itself.
> Cheers,
> Matt
>
> On Thu, Jan 27, 2022 at 10:50 AM Maximilian Michels
<[email protected] <mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]>>> wrote:
>
> Hi Hans-Peter,
>
> Checkpoints are internal to Flink. Hop does not have any
business with
> it, other than configuring the checkpoint options. Those can
also be
> configured via the Flink config file.
>
> I see that Hop has checkpointing options listed here:
>
https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options
<https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options>
>
<https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options <https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.html#_options>>
>
> Cheers,
> -Max
>
> On 23.01.22 19:03, HG wrote:
> > Hi,
> >
> > Perhaps a dumb question.
> > How is checkpointing implemented when the final framework
is Flink?
> > With Flink checkpoint barriers are inserted.
> > Can such an interval also be configured with HOP?
> >
> > Regards Hans-Peter
>
>
>
>
>
