[
https://issues.apache.org/jira/browse/KAFKA-21172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Chang updated KAFKA-21172:
-------------------------------
Attachment: test_app_upgrade-from-4.0.1-failed-logs.tar.gz
> Flaky StreamsUpgradeTest.test_app_upgrade: window changelogs are deleted by
> retention before restore
> ----------------------------------------------------------------------------------------------------
>
> Key: KAFKA-21172
> URL: https://issues.apache.org/jira/browse/KAFKA-21172
> Project: Kafka
> Issue Type: Bug
> Components: streams, system tests
> Reporter: Eric Chang
> Priority: Minor
> Labels: flaky-test
> Attachments: test_app_upgrade-from-4.0.1-failed-logs.tar.gz
>
>
> In the Streams upgrade system tests, the old smoke-test clients create their
> window changelogs with one day of retention, but the trunk
> {{SmokeTestDriver}} writes records with timestamps two days in the past. The
> broker deletes the changelog data while the old clients run. After the
> upgrade, the trunk clients cannot restore from their checkpoints
> ({{OffsetOutOfRangeException}}, then {{TaskCorruptedException}}) and restore
> the stores again. This happens in every run; when the extra restores take
> longer than 60 s, {{StreamsUpgradeTest.test_app_upgrade}} fails with:
> {noformat}
> TimeoutError: Did expect to read 'SMOKE-TEST-CLIENT-STARTED' from
> ducker@ducker51
> {noformat}
> Why the retention is too short:
> * [apache/kafka#18830|https://github.com/apache/kafka/pull/18830] moved the
> driver timestamps two days into the past and set
> {{windowstore.changelog.additional.retention.ms}} to three days in
> {{SmokeTestClient}}, but not in the old copies in
> {{streams/upgrade-system-tests-XY}}, which use the default of one day.
> * The old client creates the changelogs, e.g.
> {{SmokeTest-KSTREAM-REDUCE-STATE-STORE-0000000020-changelog}} with
> {{retention.ms=86432000}}. Changelog records keep the two-day-old input
> timestamps, so the broker deletes them almost right away.
> * Streams does not change the config of an existing internal topic, so the
> topic keeps this retention after the upgrade.
> Seen with {{from_version}} 4.0.1, 4.1.2 and 3.9.2. The attached logs
> ({{test_app_upgrade-from-4.0.1-failed-logs.tar.gz}}) are from a failed 4.0.1
> run; see {{537/StreamsSmokeTestJobRunnerService-0-*/ducker51/streams.log}}
> and the broker {{server-start-stdout-stderr.log}} files.
> To reproduce (the test often passes, so look for the data loss in the logs):
> {code:bash}
> TC_PATHS="tests/kafkatest/tests/streams/streams_application_upgrade_test.py::StreamsUpgradeTest.test_app_upgrade@'{\"from_version\":\"4.0.1\",\"metadata_quorum\":\"COMBINED_KRAFT\"}'"
> _DUCKTAPE_OPTIONS="--repeat 5" tests/docker/run_tests.sh
> {code}
> {code:bash}
> grep -rl "OffsetOutOfRangeException" --include="streams.log*" results/latest
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
