Hi, 这个是 Apache Kafka consumer 的一个已知问题，参见 FLINK-28060 [1] 和 KAFKA-13840 [2]。
[1] https://issues.apache.org/jira/browse/FLINK-28060 [2] https://issues.apache.org/jira/browse/KAFKA-13840 祝好， 庆盛 > On Jun 27, 2022, at 09:16, RS <tinyshr...@163.com> wrote: > > Hi， > 请教下各位，Flink-1.15.0，消费Kafka发现下面个问题，offset提交失败的情况，有的任务应该是一直提交失败的，数据消费了，但是offset不变，这种情况如何处理？ > > > 现象如下： > 1. 任务没有异常， > 2. 数据能正常消费处理，不影响数据使用 > 3. 任务有配置checkpoint，几分钟一次，理论上执行checkpoint的时候会提交offset > 4. 部分任务的从Kafka的offset提交失败，部分正常 > > > WARN日志如下： > 2022-06-27 01:07:42,725 INFO > org.apache.flink.streaming.api.functions.sink.filesystem.Buckets [] - Subtask > 0 checkpointing for checkpoint with id=11398 (max part counter=1). > 2022-06-27 01:07:42,830 INFO > org.apache.flink.streaming.api.functions.sink.filesystem.Buckets [] - Subtask > 0 received completion notification for checkpoint with id=11398. > 2022-06-27 01:07:43,820 INFO > org.apache.flink.streaming.api.functions.sink.filesystem.Buckets [] - Subtask > 0 checkpointing for checkpoint with id=11476 (max part counter=0). > 2022-06-27 01:07:43,946 INFO > org.apache.flink.streaming.api.functions.sink.filesystem.Buckets [] - Subtask > 0 received completion notification for checkpoint with id=11476. > 2022-06-27 01:07:45,218 INFO > org.apache.flink.streaming.api.functions.sink.filesystem.Buckets [] - Subtask > 0 checkpointing for checkpoint with id=11521 (max part counter=47). > 2022-06-27 01:07:45,290 INFO > org.apache.flink.streaming.api.functions.sink.filesystem.Buckets [] - Subtask > 0 received completion notification for checkpoint with id=11521. > 2022-06-27 01:07:45,521 WARN > org.apache.flink.connector.kafka.source.reader.KafkaSourceReader [] - Failed > to commit consumer offsets for checkpoint 11443 > org.apache.kafka.clients.consumer.RetriableCommitFailedException: Offset > commit failed with a retriable exception. You should retry committing the > latest consumed offsets. > Caused by: org.apache.kafka.common.errors.CoordinatorNotAvailableException: > The coordinator is not available. > 2022-06-27 01:07:45,990 WARN > org.apache.flink.connector.kafka.source.reader.KafkaSourceReader [] - Failed > to commit consumer offsets for checkpoint 11398 > org.apache.kafka.clients.consumer.RetriableCommitFailedException: Offset > commit failed with a retriable exception. You should retry committing the > latest consumed offsets. > Caused by: org.apache.kafka.common.errors.CoordinatorNotAvailableException: > The coordinator is not available. > > > Thanks~
