Updates to KIP -
1- A varint header_size field is introduced to enable lazy deserialization when scanning large ranges. - 2- The current serialization/deserialization methods in StateSerdes are marked as deprecated to keep the class concise. - 3- Note that VersionedKeyValueStoreWithHeaders cannot extend VersionedKeyValueStore because their methods differ in input and/or output types. In particular, the VersionedRecord returned by VersionedKeyValueStore methods is a final class and therefore cannot be subclassed. Thanks, Alieh On Thu, Jan 15, 2026 at 4:46 PM Chia-Ping Tsai <[email protected]> wrote: > chia_03: Regarding the header size, using a Varint is consistent with > Kafka's serialization standards. It avoids the overhead of a large > fixed-size field while still achieving the efficient skipping capability we > want. > > chia_04: That makes sense. > > Alieh Saeedi via dev <[email protected]> 於 2026年1月15日週四 下午10:59寫道： > >> Hi Chia-Ping Tsai, >> >> Thanks for the feedback. >> >> chia_03: The difficulty with adding a header length is deciding between a >> fixed-size field for all records or a configuration allowing users to >> define a maximum. Alternatively, we could consider using a varint for the >> header length to remain flexible and space-efficient. >> >> chia_04: >> It only makes sense to give the second column family its own RocksDB >> config if its access pattern or data characteristics are materially >> different. >> Here we have the same keys, the >> same or very similar read/write patterns (e.g., same get, put, range >> queries), >> and roughly comparable value sizes (CF2 slightly larger per entry). >> Then from RocksDB’s perspective the two CFs behave very similarly: >> both are generic key–value blobs, written and read with the same >> pattern. Most of the important RocksDB options (compaction style, >> write buffer sizes, block cache, bloom filters, etc.) would be tuned >> the same way for both. >> Do you see huge difference between these two? >> >> Thanks, >> Alieh >> >> On Thu, Jan 15, 2026 at 3:03 AM Chia-Ping Tsai <[email protected]> >> wrote: >> >> > hi >> > >> > chia_03: should we provide a more effective way to load the value >> without >> > scanning the header bytes? (e.g., by storing the total size of headers) >> > >> > chia_04: Do we need to allow separate Rocksdb configuration for the new >> > column family >> > >> > Best, >> > Chia-Ping >> > >> > On 2026/01/09 22:14:18 Alieh Saeedi via dev wrote: >> > > Hi all, >> > > >> > > I’d like to start a discussion on KIP-1271, which proposes allowing >> Kafka >> > > Streams state stores to preserve record headers. >> > > This would let header-based metadata like schema IDs, tracing info, >> and >> > > feature flags be stored and restored alongside values. >> > > The KIP introduces header-aware store types and a small config to cap >> the >> > > size of headers written into state. >> > > Details are in the KIP: >> > > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1271%3A+Allow+to+Store+Record+Headers+in+State+Stores >> > > . >> > > I’d appreciate your feedback and questions on the proposal. >> > > >> > > Thanks, >> > > Alieh >> > > >> > >> >
