This is somewhat implied in
https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/datastream/kafka/#consumer-offset-committing.
/> Note that Kafka source does //*NOT*//rely on committed offsets for
fault tolerance. Committing offset is only for exposing the progress of
I would suggest updating the documentation to include that statement.
I imagine dynamic partition discovery has no effect on this?
Regards,
Alexis.
Am Do., 21. Juli 2022 um 10:03 Uhr schrieb Chesnay Schepler <
ches...@apache.org>:
> Flink only reads the offsets from Kafka when the job is
Flink only reads the offsets from Kafka when the job is initially
started from a clear slate.
Once checkpoints are involved it only relies on offsets stored in the state.
On 20/07/2022 14:51, Alexis Sarda-Espinosa wrote:
Hello again,
I just performed a test
using
Hello again,
I just performed a test
using OffsetsInitializer.committedOffsets(OffsetResetStrategy.LATEST). I
did a few tests in the following order, and I noticed a few weird things.
Note that our job uses Processing Time windows, so watermarks are
irrelevant.
1. After the job had been running
Hi Yaroslav,
The test I did was just using earliest, I'll test with committed offset
again, thanks.
Regards,
Alexis.
On Thu, 14 Jul 2022, 20:49 Yaroslav Tkachenko, wrote:
> Hi Alexis,
>
> Do you use OffsetsInitializer.committedOffsets() to specify your Kafka
> consumer offsets? In this case,
Hi Alexis,
Do you use OffsetsInitializer.committedOffsets() to specify your Kafka
consumer offsets? In this case, it should get the offsets from Kafka and
not the state.
On Thu, Jul 14, 2022 at 11:18 AM Alexis Sarda-Espinosa <
sarda.espin...@gmail.com> wrote:
> Hello,
>
> Regarding the new
Hello,
Regarding the new Kafka source (configure with a consumer group), I found
out that if I manually change the group's offset with Kafka's admin API
independently of Flink (while the job is running), the Flink source will
ignore that and reset it to whatever it stored internally. Is there any
