[
https://issues.apache.org/jira/browse/KAFKA-8755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913523#comment-16913523
]
ASF GitHub Bot commented on KAFKA-8755:
---------------------------------------
cpettitt-confluent commented on pull request #7238: KAFKA-8755: Fix state
restore for standby tasks with optimized topology
URL: https://github.com/apache/kafka/pull/7238
Key changes include:
1. Moves general offset limit updates down to StandbyTask.
2. Updates offsets for StandbyTask at most once per commit and only when
we need and updated offset limit to make progress.
3. Avoids writing an 0 checkpoint when StandbyTask.update is called but
we cannot apply any of the records.
4. Avoids going into a restoring state in the case that the last
checkpoint is greater or equal to the offset limit (consumer committed
offset). This needs special attention please. Code is in
StoreChangelogReader.
5. Does update offset limits initially for StreamTask because it
provides a way to prevent playing to many records from the changelog
(also the input topic with optimized topology).
NOTE: this PR depends on KAFKA-8816, which is under review separately.
Fortunately the changes involved are few. You can focus just on the KAFKA-8755
commit if you prefer.
@guozhangwang @mjsax @cadonna
*More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.*
*Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.*
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Stand-by Task of an Optimized Source Table Does Not Write Anything to its
> State Store
> -------------------------------------------------------------------------------------
>
> Key: KAFKA-8755
> URL: https://issues.apache.org/jira/browse/KAFKA-8755
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 2.4.0
> Reporter: Bruno Cadonna
> Assignee: Chris Pettitt
> Priority: Major
> Labels: newbie
> Attachments: StandbyTaskTest.java
>
>
> With the following topology:
> {code:java}
> builder.table(
> INPUT_TOPIC,
> Consumed.with(Serdes.Integer(), Serdes.Integer()),
> Materialized.<Integer, Integer, KeyValueStore<Bytes, byte[]>>as(stateName)
> )
> {code}
> and with topology optimization turned on, Kafka Streams uses the input topic
> {{INPUT_TOPIC}} as the change log topic for state store {{stateName}}. A
> stand-by task for such a topology should read from {{INPUT_TOPIC}} and should
> write the records to its state store so that the streams client that runs the
> stand-by task can take over the execution of the topology in case of a
> failure with an up-to-date replica of the state.
> Currently, the stand-by task described above reads from the input topic but
> does not write the records to its state store. Thus, after a failure the
> stand-by task cannot provide any up-to-date state store and the streams
> client needs to construct the state from scratch before it can take over the
> execution.
> The described behaviour can be reproduced with the attached test.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
