Hi Tobias,
You are right, for this use case where Kafka commits the offset and you
do not use any other stateful sources/operators you will get exactly-once.
Should you have stateful operators that may not hold true anymore. Only
a checkpoint constructs a consistent view of the pipeline.
Hi Tobi,
That makes sense to me. My argument was coming from having
"exactly-once" semantics for a pipeline. In this regard, the stop
functionality does not help. But I think having the option to gracefully
shut down a pipeline is beneficial for other uses cases like the ones
you described.
Good morning Max and thanks for clarifying!
I generated the JAR 2.19.0 in the second test via the default demo code
from Beam. There were no further adjustments from my side, but as I can see
there are some open points in JIRA for 1.9.2, so for now I think that we
can focus on 1.9.1 as a target.
In some sense, stop is different because sources will be stopped first
and then all the downstream operators will stop automatically. However,
in terms of correctness of your program using cancel or stop does not
make a difference because neither approach takes a checkpoint.
Only at the time
The stop functionality has been removed in Beam. It was semantically
identical to using cancel, so we decided to drop support for it.
Both Flink 1.9.2 and 1.10.0 are not supported yet on Beam, probably they will
be part of the 2.21.0 release
1.9.2 should be supported, as it is just a patch
What I found so far is that the "Stop" Button next to the "Cancel" button
is missing when I run my Beam 2.19.0/2.20.0-SNAPSHOT streaming pipeline in
Flink's 1.9.1's web interface. I couldn't figure out yet if it has been
removed by the Flink team on purpose or if that is something "missing" in
the
Copy paste error, sorry:
2.20.0-SNAPSHOT in combination with beam-runners-flink-1.10
or beam-runners-flink-1.10-SNAPSHOT didn't work either for me.
On Thu, Feb 27, 2020 at 11:50 AM Kaymak, Tobias
wrote:
> I can confirm that the pipeline behaves as expected with 2.20.0-SNAPSHOT
> and Flink
I can confirm that the pipeline behaves as expected with 2.20.0-SNAPSHOT
and Flink 1.9.1 - I also tried Flink 1.9.2 but the webinterface didn't show
up (just a blank page - javascript was being loaded though).
I emptied my cache and investigated the log and asked on the Flink mailing
list if this
Since it was merged yesterday you can test with the 2.20.0-SNAPSHOT until
the first candidate is out.
On Wed, Feb 26, 2020 at 4:37 PM Kaymak, Tobias
wrote:
> If I am not running in detached mode (so that my pipeline starts) I am
> unable to Stop it in the webinterface. The only option available
If I am not running in detached mode (so that my pipeline starts) I am
unable to Stop it in the webinterface. The only option available is to
cancel it. Is this expected?
[image: Screenshot 2020-02-26 at 16.34.08.png]
On Wed, Feb 26, 2020 at 4:16 PM Kaymak, Tobias
wrote:
> Hello,
>
> we fixed
Hello,
we fixed the issue and are ready to test :) - is there a RC already
available?
Best,
Tobi
On Wed, Feb 26, 2020 at 12:59 PM Kaymak, Tobias
wrote:
> Hello,
>
> happy to help testing! I am currently fixing a networking issue between
> our dev cluster for integration tests and the Kafka it
Hello,
happy to help testing! I am currently fixing a networking issue between our
dev cluster for integration tests and the Kafka it is consuming from.
After that I would be ready to spin it up and test
Best,
Tobi
On Mon, Feb 24, 2020 at 10:13 PM Maximilian Michels wrote:
> Thank you for
Thank you for reporting / filing / collecting the issues.
There is a fix pending: https://github.com/apache/beam/pull/10950
As for the upgrade issues, the 1.8 and 1.9 upgrade is trivial. I will
check out the Flink 1.10 PR tomorrow.
Cheers,
Max
On 24.02.20 09:26, Ismaël Mejía wrote:
We are
We are cutting the release branch for 2.20.0 next wednesday, so not sure if
these tickets will make it, but hopefully.
For ref,
BEAM-9295 Add Flink 1.10 build target and Make FlinkRunner compatible with
Flink 1.10
BEAM-9299 Upgrade Flink Runner to 1.8.3 and 1.9.2
In any case if you have cycles
Hi Kyle,
thank you for creating the JIRA ticket, I think my best option right now is
to wait for a Beam version that is running on Flink 1.10 then - unless
there is a new Beam release around the corner :)
Best,
Tobi
On Thu, Feb 20, 2020 at 11:52 PM Kyle Weaver wrote:
> Hi Tobi,
>
> This seems
Hi Tobi,
This seems like a bug with Beam 2.19. I filed
https://issues.apache.org/jira/browse/BEAM-9345 to track the issue.
> What puzzles me is that the session cluster should be allowed to have
multiple environments in detached mode - or am I wrong?
It looks like that check is removed in Flink
Hello,
I am trying to upgrade from a Flink session cluster 1.8 to 1.9 and from
Beam 2.16.0 to 2.19.0.
Everything went quite smoothly, the local runner and the local Flink runner
work flawlessly.
However when I:
1. Generate a Beam jar for the FlinkRunner via maven (mvn package
-PFlinkRunner)
17 matches
Mail list logo
