Alieh Saeedi created KAFKA-21493:
------------------------------------

             Summary: Headers-aware source KTable / globalTable store different 
headers after a restore from the source topic
                 Key: KAFKA-21493
                 URL: https://issues.apache.org/jira/browse/KAFKA-21493
             Project: Kafka
          Issue Type: Bug
    Affects Versions: 4.4.0
            Reporter: Alieh Saeedi


A headers-aware store that uses its source topic as the changelog (a 
\{{globalTable}}, or a source \{{KTable}} with \{{topology.optimization=all}}) 
returns different headers depending on whether it was restored:
* live processing: \{{KTableSource}} / the global processor store empty headers 
(KIP-1285 limitation), so IQ returns \{{[]}}.
* after a restore (e.g. restart with an empty state dir): 
\{{RecordConverters.rawValueToHeadersValue()}} copies \{{record.headers()}} 
into the stored value, so IQ returns the source record headers.

For a dedicated changelog the copy is correct, since the changelogger writes 
the stored headers as the changelog record headers. For a source topic reused 
as the changelog, the source record headers are not what was stored. A source 
\{{KTable}} with a dedicated changelog stays \{{[]}} after a restore.

Repro: \{{IQv1HeadersStoreIntegrationTest#shouldQueryHeadersAwareGlobalStore}} 
(KAFKA-20259) pins the current behavior: live \{{[]}}, after the restart 
\{{[source=dsl-input]}}. Same with \{{Stores.*WithHeaders}} and 
\{{dsl.store.format=HEADERS}}.

Options:
# store the source record headers during live processing as well, matching the 
restore. Header-aware value serdes (e.g. a schema id in a header) need them to 
read the stored bytes.
# drop the headers on restore for source-topic changelogs, matching the current 
live behavior.

Option 1 seems the better fit, but it changes what DSL source stores hold, so 
it may need to go with KIP-1285 follow-up work. Related: KAFKA-21009, 
KAFKA-21374.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to