[
https://issues.apache.org/jira/browse/KAFKA-21172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Chang updated KAFKA-21172:
-------------------------------
Description:
In the Streams upgrade system tests, the old smoke-test clients create their
window changelogs with one day of retention, but the trunk {{SmokeTestDriver}}
writes records with timestamps two days in the past. The broker deletes the
changelog data while the old clients run. After the upgrade, the trunk clients
cannot restore from their checkpoints ({{OffsetOutOfRangeException}}, then
{{TaskCorruptedException}}) and restore the stores again. This happens in every
run; when the extra restores take longer than 60 s,
{{StreamsUpgradeTest.test_app_upgrade}} fails with:
{noformat}
TimeoutError: Did expect to read 'SMOKE-TEST-CLIENT-STARTED' from
ducker@ducker51
{noformat}
Why the retention is too short:
* [apache/kafka#18830|https://github.com/apache/kafka/pull/18830] moved the
driver timestamps two days into the past and set
{{windowstore.changelog.additional.retention.ms}} to three days in
{{SmokeTestClient}}, but not in the old copies in
{{streams/upgrade-system-tests-XY}}, which use the default of one day.
* The old client creates the changelogs, e.g.
{{SmokeTest-KSTREAM-REDUCE-STATE-STORE-0000000020-changelog}} with
{{retention.ms=86432000}}. Changelog records keep the two-day-old input
timestamps, so the broker deletes them almost right away.
* Streams does not change the config of an existing internal topic, so the
topic keeps this retention after the upgrade.
Seen with {{from_version}} 4.0.1, 4.1.2 and 3.9.2. The attached logs
({{test_app_upgrade-from-4.0.1-failed-logs.tar.gz}}) are from a failed 4.0.1
run; see {{537/StreamsSmokeTestJobRunnerService-0-*/ducker51/streams.log}} and
the broker {{server-start-stdout-stderr.log}} files.
To reproduce (the test often passes, so look for the data loss in the logs):
{code:bash}
TC_PATHS="tests/kafkatest/tests/streams/streams_application_upgrade_test.py::StreamsUpgradeTest.test_app_upgrade@'{\"from_version\":\"4.0.1\",\"metadata_quorum\":\"COMBINED_KRAFT\"}'"
_DUCKTAPE_OPTIONS="--repeat 5" tests/docker/run_tests.sh
{code}
{code:bash}
grep -rl "OffsetOutOfRangeException" --include="streams.log*" results/latest
{code}
> Flaky StreamsUpgradeTest.test_app_upgrade: window changelogs are deleted by
> retention before restore
> ----------------------------------------------------------------------------------------------------
>
> Key: KAFKA-21172
> URL: https://issues.apache.org/jira/browse/KAFKA-21172
> Project: Kafka
> Issue Type: Bug
> Components: streams, system tests
> Reporter: Eric Chang
> Priority: Minor
> Labels: flaky-test
>
> In the Streams upgrade system tests, the old smoke-test clients create their
> window changelogs with one day of retention, but the trunk
> {{SmokeTestDriver}} writes records with timestamps two days in the past. The
> broker deletes the changelog data while the old clients run. After the
> upgrade, the trunk clients cannot restore from their checkpoints
> ({{OffsetOutOfRangeException}}, then {{TaskCorruptedException}}) and restore
> the stores again. This happens in every run; when the extra restores take
> longer than 60 s, {{StreamsUpgradeTest.test_app_upgrade}} fails with:
> {noformat}
> TimeoutError: Did expect to read 'SMOKE-TEST-CLIENT-STARTED' from
> ducker@ducker51
> {noformat}
> Why the retention is too short:
> * [apache/kafka#18830|https://github.com/apache/kafka/pull/18830] moved the
> driver timestamps two days into the past and set
> {{windowstore.changelog.additional.retention.ms}} to three days in
> {{SmokeTestClient}}, but not in the old copies in
> {{streams/upgrade-system-tests-XY}}, which use the default of one day.
> * The old client creates the changelogs, e.g.
> {{SmokeTest-KSTREAM-REDUCE-STATE-STORE-0000000020-changelog}} with
> {{retention.ms=86432000}}. Changelog records keep the two-day-old input
> timestamps, so the broker deletes them almost right away.
> * Streams does not change the config of an existing internal topic, so the
> topic keeps this retention after the upgrade.
> Seen with {{from_version}} 4.0.1, 4.1.2 and 3.9.2. The attached logs
> ({{test_app_upgrade-from-4.0.1-failed-logs.tar.gz}}) are from a failed 4.0.1
> run; see {{537/StreamsSmokeTestJobRunnerService-0-*/ducker51/streams.log}}
> and the broker {{server-start-stdout-stderr.log}} files.
> To reproduce (the test often passes, so look for the data loss in the logs):
> {code:bash}
> TC_PATHS="tests/kafkatest/tests/streams/streams_application_upgrade_test.py::StreamsUpgradeTest.test_app_upgrade@'{\"from_version\":\"4.0.1\",\"metadata_quorum\":\"COMBINED_KRAFT\"}'"
> _DUCKTAPE_OPTIONS="--repeat 5" tests/docker/run_tests.sh
> {code}
> {code:bash}
> grep -rl "OffsetOutOfRangeException" --include="streams.log*" results/latest
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
