The KafkaIO implementation relies on checkpointing to be able to update the last committed offset. This is currently unsupported in the portable Flink runner. BEAM-6868[1] is the associated JIRA. Please vote on it and/or offer to provide an implementation.
1: https://issues.apache.org/jira/browse/BEAM-6868 On Mon, Jul 6, 2020 at 1:42 PM Piotr Filipiuk <piotr.filip...@gmail.com> wrote: > Hi, > > I am trying to run a simple example that uses Python API to ReadFromKafka, > however I am getting the following error when using Flink Runner: > > java.lang.UnsupportedOperationException: The ActiveBundle does not have a > registered bundle checkpoint handler. > > See full log in read_from_kafka_flink.log > > I am using: > Kafka 2.5.0 > Beam 2.22.0 > Flink 1.10 > > When using Direct runner, the pipeline does not fail but does not seem to > be consuming any data (see read_from_kafka.log) even though the updated > offsets are being logged: > > [2020-07-06 13:36:01,342] {worker_handlers.py:398} INFO - severity: INFO > timestamp { > seconds: 1594067761 > nanos: 340000000 > } > message: "Reader-0: reading from test-topic-0 starting at offset 165" > log_location: "org.apache.beam.sdk.io.kafka.KafkaUnboundedSource" > thread: "23" > > I am running both Kafka and Flink locally. I would appreciate your help > understanding and fixing the issue. > > -- > Best regards, > Piotr >
