[
https://issues.apache.org/jira/browse/KAFKA-20416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18117777#comment-18117777
]
Bill Bejeck commented on KAFKA-20416:
-------------------------------------
Hi [~fabianbell] — thanks for the logs. I've looked them over, but we don't
have enough information to confirm the root cause of not restoring from the
source topic as changelog.
The loss only occurs if, after that wipe, the store re-restores to a ceiling of
0 (nothing restored) instead of rebuilding from the source topic as the
changelog. That decision is logged only by StoreChangelogReader, and there are
no StoreChangelogReader lines anywhere in logs.jsonl.
Are there any other logs you can share? Broker side logs as well would be
helpful as well.
And to confirm the error we are running down is that after the rolling security
patch update you have KTable(s) returning "null" for known values in the source
topic?
The confirming line we're looking for is:
"End offset for changelog <source-topic>-<partition> initialized as 0."
"Finished restoring changelog <source-topic>-<partition> to store <store> with
a total number of 0 records"
> RocksDB loses entries during broker patches.
> --------------------------------------------
>
> Key: KAFKA-20416
> URL: https://issues.apache.org/jira/browse/KAFKA-20416
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 3.9.1
> Environment: Broker: MSK with kafka.m7g.2xlarge instances
> Application: EKS with m7i.2xlarge instances and Bottlerocket OS 1.57.0
> Reporter: Fabian Bell
> Assignee: Bill Bejeck
> Priority: Major
> Fix For: 4.3.2, 4.5.0, 4.4.1
>
> Attachments: logs, logs.jsonl
>
>
> h2. Problem:
> We discovered a strange behaviour on our production environment. We use a
> KTable to look up data from a topic we write to.
>
> {code:java}
> builder.table(topicName, Consumed.with(keySerde, valueSerde),
> Materialized.as(storeName)) {code}
>
> When we access the store in the processor, we observed that the store
> returned null values for keys that have non-null entries in the topic that
> backs the KTable after an MSK security patch. We never tombstone an entry in
> our topic nor have a delete retention activated.
> This only happens for some of our instances.
> We see the following stream logs:
>
> {code:java}
> Committing task(s) 0_14 failed.
> Detected the states of tasks [0_14] are corrupted. Will close the task as
> dirty and re-create and bootstrap from scratch.
> Active task(s) got corrupted. Triggering a rebalance.
> End offset for changelog our-topic-14 initialized as 16596290.
> Restoration in progress for 1 partitions. {our-topic-14: position=0,
> end=16596290, totalRestored=0}
> State transition from RUNNING to PARTITIONS_REVOKED
> No followup rebalance was requested, resetting the rebalance schedule.
> partition revocation took 80 ms.
> State transition from PARTITIONS_REVOKED to PARTITIONS_ASSIGNED
> State transition from PARTITIONS_ASSIGNED to RUNNING {code}
>
> This all happens within a few seconds, and the `Restoration in progress ...`
> log is the only one we can see. A full restoration usually takes like 30 min.
> The error message of the commit failure is
> {code:java}
> o.a.k.c.e.TimeoutException: Timeout expired after 60000ms while awaiting
> AddOffsetsToTxn {code}
> We can fix this situation by clearing the state directory and forcing a full
> restoration.
> h2. Context:
> Each instance has its own persistent state directory. The configured state
> directory does not change.
> Processing Guarantee: exactly_once_v2
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
