Hello Alieh, Thanks for the KIP! The proposal looks very interesting.
I have two questions/observations: bloku_1: It makes sense that headers in the changelog topic would be stored as native Kafka record headers. This became clear to me after following the discussion thread, but I couldn't find an explicit mention of it in the KIP document itself. Should we document this explicitly in the KIP to avoid any ambiguity? bloku_2: While I support the decision to move the DSL stores update to a separate KIP, I’m curious about the exclusion of non-timestamped stores. If we are already touching the PAPI (Processor API) for this change, wouldn't it make sense to include non-timestamped stores as well, or is there a specific technical constraint keeping them out of scope for now? Looking forward to your response, Uladzislau Blok On Fri, Jan 16, 2026 at 6:03 PM Alieh Saeedi via dev <[email protected]> wrote: > Updates to KIP > > - > > 1- A varint header_size field is introduced to enable lazy deserialization > when scanning large ranges. > - > > 2- The current serialization/deserialization methods in StateSerdes are > marked as deprecated to keep the class concise. > - > > 3- Note that VersionedKeyValueStoreWithHeaders cannot extend > VersionedKeyValueStore because their methods differ in input and/or output > types. In particular, the VersionedRecord returned by > VersionedKeyValueStore > methods is a final class and therefore cannot be subclassed. > > Thanks, > Alieh > > On Thu, Jan 15, 2026 at 4:46 PM Chia-Ping Tsai <[email protected]> wrote: > > > chia_03: Regarding the header size, using a Varint is consistent with > > Kafka's serialization standards. It avoids the overhead of a large > > fixed-size field while still achieving the efficient skipping capability > we > > want. > > > > chia_04: That makes sense. > > > > Alieh Saeedi via dev <[email protected]> 於 2026年1月15日週四 下午10:59寫道： > > > >> Hi Chia-Ping Tsai, > >> > >> Thanks for the feedback. > >> > >> chia_03: The difficulty with adding a header length is deciding between > a > >> fixed-size field for all records or a configuration allowing users to > >> define a maximum. Alternatively, we could consider using a varint for > the > >> header length to remain flexible and space-efficient. > >> > >> chia_04: > >> It only makes sense to give the second column family its own RocksDB > >> config if its access pattern or data characteristics are materially > >> different. > >> Here we have the same keys, the > >> same or very similar read/write patterns (e.g., same get, put, range > >> queries), > >> and roughly comparable value sizes (CF2 slightly larger per entry). > >> Then from RocksDB’s perspective the two CFs behave very similarly: > >> both are generic key–value blobs, written and read with the same > >> pattern. Most of the important RocksDB options (compaction style, > >> write buffer sizes, block cache, bloom filters, etc.) would be tuned > >> the same way for both. > >> Do you see huge difference between these two? > >> > >> Thanks, > >> Alieh > >> > >> On Thu, Jan 15, 2026 at 3:03 AM Chia-Ping Tsai <[email protected]> > >> wrote: > >> > >> > hi > >> > > >> > chia_03: should we provide a more effective way to load the value > >> without > >> > scanning the header bytes? (e.g., by storing the total size of > headers) > >> > > >> > chia_04: Do we need to allow separate Rocksdb configuration for the > new > >> > column family > >> > > >> > Best, > >> > Chia-Ping > >> > > >> > On 2026/01/09 22:14:18 Alieh Saeedi via dev wrote: > >> > > Hi all, > >> > > > >> > > I’d like to start a discussion on KIP-1271, which proposes allowing > >> Kafka > >> > > Streams state stores to preserve record headers. > >> > > This would let header-based metadata like schema IDs, tracing info, > >> and > >> > > feature flags be stored and restored alongside values. > >> > > The KIP introduces header-aware store types and a small config to > cap > >> the > >> > > size of headers written into state. > >> > > Details are in the KIP: > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1271%3A+Allow+to+Store+Record+Headers+in+State+Stores > >> > > . > >> > > I’d appreciate your feedback and questions on the proposal. > >> > > > >> > > Thanks, > >> > > Alieh > >> > > > >> > > >> > > >
