Jiri Syrovy created KAFKA-21492:
-----------------------------------

             Summary: Segmented store without segments has no committed offset: 
task is wiped as corrupted under EOS on every re-initialization
                 Key: KAFKA-21492
                 URL: https://issues.apache.org/jira/browse/KAFKA-21492
             Project: Kafka
          Issue Type: Bug
          Components: streams
    Affects Versions: 4.3.2
            Reporter: Jiri Syrovy


h3. Problem

Under exactly_once_v2, a task that contains a persistent window or session 
store which holds no segment is treated as corrupted every time the task is 
initialized on an instance that already has its task directory (after a 
restart, a rebalance, or a switch between active and standby). The whole task 
is wiped and restored from the changelogs, including all its other stores. 
Wiping does not help: the restore re-creates the same empty store directory, so 
the next initialization fails the same way. A corrupted active task also 
triggers a rebalance.

A store has no segment when it was never written.
h3. Root cause

* Since KIP-1035 a store keeps its changelog offsets itself. A segmented store 
keeps them in its segments: {{AbstractSegments#commit}} writes into the 
existing segments only, and {{AbstractSegments#committedOffset}} returns 
{{null}} when there are none.
* Opening the store creates its directory ({{AbstractSegments#openExisting}}), 
so the task directory is not empty afterwards.
* {{ProcessorStateManager#initializeStoreOffsets}} throws 
{{TaskCorruptedException}} when {{committedOffset}} is {{null}}, EOS is enabled 
and the task directory was not empty.

A key-value store does not have the problem: it commits the sentinel 
{{OFFSET_UNKNOWN}} when it has no offset, and {{initializeStoreOffsets}} 
accepts that.

h3. Reproducer

With {{TopologyTestDriver}} and exactly_once_v2: a topology with a persistent 
window (or session) store that no processor writes to. Initializing the task on 
a state directory that contains only the store's empty directory throws:

{noformat}
org.apache.kafka.streams.errors.TaskCorruptedException: Tasks [0_0] are 
corrupted and hence need to be re-initialized
{noformat}

The same with at_least_once starts normally. The pull request adds this as 
{{SegmentedStoreWithoutSegmentsTest}}.

h3. Seen in production

A 4-instance application on 4.3.2 with exactly_once_v2, num.standby.replicas=1 
and the classic protocol. One sub-topology has a timestamped window store whose 
changelog is empty on all partitions. After a start with empty state 
directories the group never settled: every probing rebalance (10 minutes apart) 
ended with tasks of that sub-wiped and restored, on standby and on active 
tasks, and input lag grew until the application was rolled back to a 4.1  
release. No process was killed or restarted during this.

{noformat}
WARN o.a.k.s.p.i.ProcessorStateManager - stream-thread [...] standby-task [2_0] 
State store <window store> did not find checkpoint offsets while stores are not 
empty, since under EOS it has the risk of getting uncommitted data in stores we 
have to treat it as a task corruption error and wipe out the local state of 
task 2_0 before re-bootstrapping
{noformat}

h3. Proposed fix

Report the committed offset of a segmented store without segments as 
{{OFFSET_UNKNOWN}}: there is no local data in that state, so nothing can be 
uncommitted. A store that has a segment but no offset still reports none. 
{{GlobalStateManagerImpl}} has to ignore the sentinel when it loads a global 
store's offset. Pull request to follow.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to