My test is using the direct runner with the streaming option set to true.
As it is the runner that should be used for development (as far as I
understand), I suppose it should expose most of the "correct" behaviours
... including the fact that checkpointoing algorithm should be
understandable,
It is great to see that, moving forward, we will have a way to tell if
finalization is required.
>In my opinion, for such crucial behavior i would expect the pipeline to fail
>with
>a clear message stating the reason, like in the same way when you implement
>a new Codec and forget to override
In the future, you will be able to check and give a hard error if
checkpointing is disabled yet finalization is requested for portable
pipelines:
https://github.com/apache/beam/blob/2be7457a4c0b311c3bd784b3f00b425596adeb06/model/pipeline/src/main/proto/beam_runner_api.proto#L382
On Fri, Jun 14,
In my opinion, for such crucial behavior i would expect the pipeline to
fail with a clear message stating the reason, like in the same way when you
implement a new Codec and forget to override the verifyDeterministic method
(don't recall the right name of it).
Just my 2 cents.
Maximilian Michels
This has come up before: https://issues.apache.org/jira/browse/BEAM-4520
The issue is that checkpoints won't be acknowledged if checkpointing is
disabled in Flink. We throw a WARN when unbounded sources are used without
checkpointing. Not all unbounded sources actually need to finalize
Is there a JIRA for this ? if this solves an issue to multiple users
maybe is worth of integrating the patch.
Would you be up to do this Augustin?
On Fri, Jun 14, 2019 at 10:35 AM Augustin Lafanechere
wrote:
>
> Hello Nicolas,
> I also encountered the same problem.
> RabbitMQIo indeed
Hello Nicolas,
I also encountered the same problem.
RabbitMQIo indeed acknowledges messages on finalizeCheckpoint calls but this
was not clear to me on when this method is called because no message were ack
on pipeline runtime.
I finally decided to implement a patch of the RabbitMqIO to set auto
Hi Nicolas,
what runner do you use? Have you configured checkpoints (if it is one
that needs checkpoints to be configured - e.g. Flink)?
Jan
On 6/13/19 3:47 PM, Nicolas Delsaux wrote:
I'm having big troubles reading data from RabbitMQ.
To understand my troubles, i've simplified my previous
I'm having big troubles reading data from RabbitMQ.
To understand my troubles, i've simplified my previous code to the extreme :
Pipeline pipeline = Pipeline.create(options);
PCollection wat = (PCollection)
pipeline.apply("read_from_rabbit",
RabbitMqIO.read()
