Jiri Syrovy created KAFKA-21492:
-----------------------------------
Summary: Segmented store without segments has no committed offset:
task is wiped as corrupted under EOS on every re-initialization
Key: KAFKA-21492
URL: https://issues.apache.org/jira/browse/KAFKA-21492
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 4.3.2
Reporter: Jiri Syrovy
h3. Problem
Under exactly_once_v2, a task that contains a persistent window or session
store which holds no segment is treated as corrupted every time the task is
initialized on an instance that already has its task directory (after a
restart, a rebalance, or a switch between active and standby). The whole task
is wiped and restored from the changelogs, including all its other stores.
Wiping does not help: the restore re-creates the same empty store directory, so
the next initialization fails the same way. A corrupted active task also
triggers a rebalance.
A store has no segment when it was never written.
h3. Root cause
* Since KIP-1035 a store keeps its changelog offsets itself. A segmented store
keeps them in its segments: {{AbstractSegments#commit}} writes into the
existing segments only, and {{AbstractSegments#committedOffset}} returns
{{null}} when there are none.
* Opening the store creates its directory ({{AbstractSegments#openExisting}}),
so the task directory is not empty afterwards.
* {{ProcessorStateManager#initializeStoreOffsets}} throws
{{TaskCorruptedException}} when {{committedOffset}} is {{null}}, EOS is enabled
and the task directory was not empty.
A key-value store does not have the problem: it commits the sentinel
{{OFFSET_UNKNOWN}} when it has no offset, and {{initializeStoreOffsets}}
accepts that.
h3. Reproducer
With {{TopologyTestDriver}} and exactly_once_v2: a topology with a persistent
window (or session) store that no processor writes to. Initializing the task on
a state directory that contains only the store's empty directory throws:
{noformat}
org.apache.kafka.streams.errors.TaskCorruptedException: Tasks [0_0] are
corrupted and hence need to be re-initialized
{noformat}
The same with at_least_once starts normally. The pull request adds this as
{{SegmentedStoreWithoutSegmentsTest}}.
h3. Seen in production
A 4-instance application on 4.3.2 with exactly_once_v2, num.standby.replicas=1
and the classic protocol. One sub-topology has a timestamped window store whose
changelog is empty on all partitions. After a start with empty state
directories the group never settled: every probing rebalance (10 minutes apart)
ended with tasks of that sub-wiped and restored, on standby and on active
tasks, and input lag grew until the application was rolled back to a 4.1
release. No process was killed or restarted during this.
{noformat}
WARN o.a.k.s.p.i.ProcessorStateManager - stream-thread [...] standby-task [2_0]
State store <window store> did not find checkpoint offsets while stores are not
empty, since under EOS it has the risk of getting uncommitted data in stores we
have to treat it as a task corruption error and wipe out the local state of
task 2_0 before re-bootstrapping
{noformat}
h3. Proposed fix
Report the committed offset of a segmented store without segments as
{{OFFSET_UNKNOWN}}: there is no local data in that state, so nothing can be
uncommitted. A store that has a segment but no offset still reports none.
{{GlobalStateManagerImpl}} has to ignore the sentinel when it loads a global
store's offset. Pull request to follow.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)