[
https://issues.apache.org/jira/browse/KAFKA-20416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18115919#comment-18115919
]
Fabian Bell commented on KAFKA-20416:
-------------------------------------
[~bbejeck] thank you for your continous investigation. KIP-848 reads quite
nicely but according to
[https://docs.confluent.io/platform/current/streams/developer-guide/streams-rebalance-protocol.html]
we still need to wait for KIP-1071 to use it with Kafka Streams. The current
supported MSK version is only 4.2.x. Thereby, it does not solve our problem.
When can we expect to have the support for Kafka Streams? Would it be possible
to still implement the fix for the current group coordinator? We just had this
case on Friday again, causing 4 of our instances to process the data
incorrectly due to the invalid store data.
> RocksDB loses entries during broker patches.
> --------------------------------------------
>
> Key: KAFKA-20416
> URL: https://issues.apache.org/jira/browse/KAFKA-20416
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 3.9.1
> Environment: Broker: MSK with kafka.m7g.2xlarge instances
> Application: EKS with m7i.2xlarge instances and Bottlerocket OS 1.57.0
> Reporter: Fabian Bell
> Assignee: Bill Bejeck
> Priority: Major
> Attachments: logs
>
>
> h2. Problem:
> We discovered a strange behaviour on our production environment. We use a
> KTable to look up data from a topic we write to.
>
> {code:java}
> builder.table(topicName, Consumed.with(keySerde, valueSerde),
> Materialized.as(storeName)) {code}
>
> When we access the store in the processor, we observed that the store
> returned null values for keys that have non-null entries in the topic that
> backs the KTable after an MSK security patch. We never tombstone an entry in
> our topic nor have a delete retention activated.
> This only happens for some of our instances.
> We see the following stream logs:
>
> {code:java}
> Committing task(s) 0_14 failed.
> Detected the states of tasks [0_14] are corrupted. Will close the task as
> dirty and re-create and bootstrap from scratch.
> Active task(s) got corrupted. Triggering a rebalance.
> End offset for changelog our-topic-14 initialized as 16596290.
> Restoration in progress for 1 partitions. {our-topic-14: position=0,
> end=16596290, totalRestored=0}
> State transition from RUNNING to PARTITIONS_REVOKED
> No followup rebalance was requested, resetting the rebalance schedule.
> partition revocation took 80 ms.
> State transition from PARTITIONS_REVOKED to PARTITIONS_ASSIGNED
> State transition from PARTITIONS_ASSIGNED to RUNNING {code}
>
> This all happens within a few seconds, and the `Restoration in progress ...`
> log is the only one we can see. A full restoration usually takes like 30 min.
> The error message of the commit failure is
> {code:java}
> o.a.k.c.e.TimeoutException: Timeout expired after 60000ms while awaiting
> AddOffsetsToTxn {code}
> We can fix this situation by clearing the state directory and forcing a full
> restoration.
> h2. Context:
> Each instance has its own persistent state directory. The configured state
> directory does not change.
> Processing Guarantee: exactly_once_v2
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
