Alieh Saeedi created KAFKA-21493:
------------------------------------
Summary: Headers-aware source KTable / globalTable store different
headers after a restore from the source topic
Key: KAFKA-21493
URL: https://issues.apache.org/jira/browse/KAFKA-21493
Project: Kafka
Issue Type: Bug
Affects Versions: 4.4.0
Reporter: Alieh Saeedi
A headers-aware store that uses its source topic as the changelog (a
\{{globalTable}}, or a source \{{KTable}} with \{{topology.optimization=all}})
returns different headers depending on whether it was restored:
* live processing: \{{KTableSource}} / the global processor store empty headers
(KIP-1285 limitation), so IQ returns \{{[]}}.
* after a restore (e.g. restart with an empty state dir):
\{{RecordConverters.rawValueToHeadersValue()}} copies \{{record.headers()}}
into the stored value, so IQ returns the source record headers.
For a dedicated changelog the copy is correct, since the changelogger writes
the stored headers as the changelog record headers. For a source topic reused
as the changelog, the source record headers are not what was stored. A source
\{{KTable}} with a dedicated changelog stays \{{[]}} after a restore.
Repro: \{{IQv1HeadersStoreIntegrationTest#shouldQueryHeadersAwareGlobalStore}}
(KAFKA-20259) pins the current behavior: live \{{[]}}, after the restart
\{{[source=dsl-input]}}. Same with \{{Stores.*WithHeaders}} and
\{{dsl.store.format=HEADERS}}.
Options:
# store the source record headers during live processing as well, matching the
restore. Header-aware value serdes (e.g. a schema id in a header) need them to
read the stored bytes.
# drop the headers on restore for source-topic changelogs, matching the current
live behavior.
Option 1 seems the better fit, but it changes what DSL source stores hold, so
it may need to go with KIP-1285 follow-up work. Related: KAFKA-21009,
KAFKA-21374.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)