[jira] [Resolved] (KAFKA-8375) Offset jumps back after commit
[ https://issues.apache.org/jira/browse/KAFKA-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus Dybeck resolved KAFKA-8375. -- Fix Version/s: 2.4.2 Resolution: Later We have not encountered the issue after upgrading both kafka-clients and akka. > Offset jumps back after commit > -- > > Key: KAFKA-8375 > URL: https://issues.apache.org/jira/browse/KAFKA-8375 > Project: Kafka > Issue Type: Bug > Components: offset manager >Affects Versions: 1.1.1 >Reporter: Markus Dybeck >Priority: Major > Fix For: 2.4.2 > > Attachments: partition_lag_metrics.png > > > *Setup* > Kafka: 1.1.1 > Kafka-client: 1.1.1 > Zookeeper: 3.4.11 > Akka streams: 0.20 > *Topic config* > DELETE_RETENTION_MS_CONFIG: "5000" > CLEANUP_POLICY_CONFIG: "compact,delete" > RETENTION_BYTES_CONFIG: 2000L > RETENTION_MS_CONFIG: 3600 > *Consumer config* > AUTO_OFFSET_RESET_CONFIG: "earliest" > *Behavior* > We have 7 Consumers consuming from 7 partitions, and some of the consumers > lag jumped back a bit randomly. No new messages were pushed to the topic > during the time. We didn't see any strange logs during the time, and the > brokers did not restart either. > Either way, if there would be a restart or rebalance going on, we can not > understand why the offset would jump back after it was committed? > We did observe it both with logs and by watching metrics of the lag. Our logs > pointed out that after we committed the offset, around 30-35 seconds later we > consumed an earlier committed message and then the loop begun. The behavior > was the same after a restart of all the consumers. The behavior then stopped > after a while all by itself. > We have no clue going forward, or if these might be an issue with akka. But > is there any known issue that might cause this? > Attaching a screendump with metrics that shows the lag for one partition. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-8375) Offset jumps back after commit
[ https://issues.apache.org/jira/browse/KAFKA-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus Dybeck updated KAFKA-8375: - Description: *Setup* Kafka: 1.1.1 Kafka-client: 1.1.1 Zookeeper: 3.4.11 Akka streams: 0.20 *Topic config* DELETE_RETENTION_MS_CONFIG: "5000" CLEANUP_POLICY_CONFIG: "compact,delete" RETENTION_BYTES_CONFIG: 2000L RETENTION_MS_CONFIG: 3600 *Consumer config* AUTO_OFFSET_RESET_CONFIG: "earliest" *Behavior* We have 7 Consumers consuming from 7 partitions, and some of the consumers lag jumped back a bit randomly. No new messages were pushed to the topic during the time. We didn't see any strange logs during the time, and the brokers did not restart either. Either way, if there would be a restart or rebalance going on, we can not understand why the offset would jump back after it was committed? We did observe it both with logs and by watching metrics of the lag. Our logs pointed out that after we committed the offset, around 30-35 seconds later we consumed an earlier committed message and then the loop begun. The behavior was the same after a restart of all the consumers. The behavior then stopped after a while all by itself. We have no clue going forward, or if these might be an issue with akka. But is there any known issue that might cause this? Attaching a screendump with metrics that shows the lag for one partition. was: *Setup* Kafka: 1.1.1 Kafka-client: 1.1.1 Zookeeper: 3.4.11 Akka streams: 0.20 *Topic config* DELETE_RETENTION_MS_CONFIG: "5000" CLEANUP_POLICY_CONFIG: "compact,delete" RETENTION_BYTES_CONFIG: 2000L RETENTION_MS_CONFIG: 3600 *Behavior* We have 7 Consumers consuming from 7 partitions, and some of the consumers lag jumped back a bit randomly. No new messages were pushed to the topic during the time. We didn't see any strange logs during the time, and the brokers did not restart either. Either way, if there would be a restart or rebalance going on, we can not understand why the offset would jump back after it was committed? We did observe it both with logs and by watching metrics of the lag. Our logs pointed out that after we committed the offset, around 30-35 seconds later we consumed an earlier committed message and then the loop begun. The behavior was the same after a restart of all the consumers. The behavior then stopped after a while all by itself. We have no clue going forward, or if these might be an issue with akka. But is there any known issue that might cause this? Attaching a screendump with metrics that shows the lag for one partition. > Offset jumps back after commit > -- > > Key: KAFKA-8375 > URL: https://issues.apache.org/jira/browse/KAFKA-8375 > Project: Kafka > Issue Type: Bug > Components: offset manager >Affects Versions: 1.1.1 >Reporter: Markus Dybeck >Priority: Major > Attachments: partition_lag_metrics.png > > > *Setup* > Kafka: 1.1.1 > Kafka-client: 1.1.1 > Zookeeper: 3.4.11 > Akka streams: 0.20 > *Topic config* > DELETE_RETENTION_MS_CONFIG: "5000" > CLEANUP_POLICY_CONFIG: "compact,delete" > RETENTION_BYTES_CONFIG: 2000L > RETENTION_MS_CONFIG: 3600 > *Consumer config* > AUTO_OFFSET_RESET_CONFIG: "earliest" > *Behavior* > We have 7 Consumers consuming from 7 partitions, and some of the consumers > lag jumped back a bit randomly. No new messages were pushed to the topic > during the time. We didn't see any strange logs during the time, and the > brokers did not restart either. > Either way, if there would be a restart or rebalance going on, we can not > understand why the offset would jump back after it was committed? > We did observe it both with logs and by watching metrics of the lag. Our logs > pointed out that after we committed the offset, around 30-35 seconds later we > consumed an earlier committed message and then the loop begun. The behavior > was the same after a restart of all the consumers. The behavior then stopped > after a while all by itself. > We have no clue going forward, or if these might be an issue with akka. But > is there any known issue that might cause this? > Attaching a screendump with metrics that shows the lag for one partition. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-8375) Offset jumps back after commit
[ https://issues.apache.org/jira/browse/KAFKA-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus Dybeck updated KAFKA-8375: - Attachment: (was: Skärmavbild 2019-05-16 kl. 08.41.53.png) > Offset jumps back after commit > -- > > Key: KAFKA-8375 > URL: https://issues.apache.org/jira/browse/KAFKA-8375 > Project: Kafka > Issue Type: Bug > Components: offset manager >Affects Versions: 1.1.1 >Reporter: Markus Dybeck >Priority: Major > Attachments: partition_lag_metrics.png > > > *Setup* > Kafka: 1.1.1 > Kafka-client: 1.1.1 > Zookeeper: 3.4.11 > Akka streams: 0.20 > *Topic config* > DELETE_RETENTION_MS_CONFIG: "5000" > CLEANUP_POLICY_CONFIG: "compact,delete" > RETENTION_BYTES_CONFIG: 2000L > RETENTION_MS_CONFIG: 3600 > *Behavior* > We have 7 Consumers consuming from 7 partitions, and some of the consumers > lag jumped back a bit randomly. No new messages were pushed to the topic > during the time. We didn't see any strange logs during the time, and the > brokers did not restart either. > Either way, if there would be a restart or rebalance going on, we can not > understand why the offset would jump back after it was committed? > We did observe it both with logs and by watching metrics of the lag. Our logs > pointed out that after we committed the offset, around 30-35 seconds later we > consumed an earlier committed message and then the loop begun. The behavior > was the same after a restart of all the consumers. The behavior then stopped > after a while all by itself. > We have no clue going forward, or if these might be an issue with akka. But > is there any known issue that might cause this? > Attaching a screendump with metrics that shows the lag for one partition. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-8375) Offset jumps back after commit
[ https://issues.apache.org/jira/browse/KAFKA-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus Dybeck updated KAFKA-8375: - Attachment: partition_lag_metrics.png > Offset jumps back after commit > -- > > Key: KAFKA-8375 > URL: https://issues.apache.org/jira/browse/KAFKA-8375 > Project: Kafka > Issue Type: Bug > Components: offset manager >Affects Versions: 1.1.1 >Reporter: Markus Dybeck >Priority: Major > Attachments: partition_lag_metrics.png > > > *Setup* > Kafka: 1.1.1 > Kafka-client: 1.1.1 > Zookeeper: 3.4.11 > Akka streams: 0.20 > *Topic config* > DELETE_RETENTION_MS_CONFIG: "5000" > CLEANUP_POLICY_CONFIG: "compact,delete" > RETENTION_BYTES_CONFIG: 2000L > RETENTION_MS_CONFIG: 3600 > *Behavior* > We have 7 Consumers consuming from 7 partitions, and some of the consumers > lag jumped back a bit randomly. No new messages were pushed to the topic > during the time. We didn't see any strange logs during the time, and the > brokers did not restart either. > Either way, if there would be a restart or rebalance going on, we can not > understand why the offset would jump back after it was committed? > We did observe it both with logs and by watching metrics of the lag. Our logs > pointed out that after we committed the offset, around 30-35 seconds later we > consumed an earlier committed message and then the loop begun. The behavior > was the same after a restart of all the consumers. The behavior then stopped > after a while all by itself. > We have no clue going forward, or if these might be an issue with akka. But > is there any known issue that might cause this? > Attaching a screendump with metrics that shows the lag for one partition. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-8375) Offset jumps back after commit
Markus Dybeck created KAFKA-8375: Summary: Offset jumps back after commit Key: KAFKA-8375 URL: https://issues.apache.org/jira/browse/KAFKA-8375 Project: Kafka Issue Type: Bug Components: offset manager Affects Versions: 1.1.1 Reporter: Markus Dybeck Attachments: Skärmavbild 2019-05-16 kl. 08.41.53.png *Setup* Kafka: 1.1.1 Kafka-client: 1.1.1 Zookeeper: 3.4.11 Akka streams: 0.20 *Topic config* DELETE_RETENTION_MS_CONFIG: "5000" CLEANUP_POLICY_CONFIG: "compact,delete" RETENTION_BYTES_CONFIG: 2000L RETENTION_MS_CONFIG: 3600 *Behavior* We have 7 Consumers consuming from 7 partitions, and some of the consumers lag jumped back a bit randomly. No new messages were pushed to the topic during the time. We didn't see any strange logs during the time, and the brokers did not restart either. Either way, if there would be a restart or rebalance going on, we can not understand why the offset would jump back after it was committed? We did observe it both with logs and by watching metrics of the lag. Our logs pointed out that after we committed the offset, around 30-35 seconds later we consumed an earlier committed message and then the loop begun. The behavior was the same after a restart of all the consumers. The behavior then stopped after a while all by itself. We have no clue going forward, or if these might be an issue with akka. But is there any known issue that might cause this? Attaching a screendump with metrics that shows the lag for one partition. -- This message was sent by Atlassian JIRA (v7.6.3#76005)