uneven distribution of events across kafka topic partitions for small number of unique keys

Hi All, We are experiencing some uneven distribution of events across topic partitions for a small set of unique keys: following are the details: 1. topic with 6 partitions 2. 8 unique keys used to produce events onto the topic Used 'key' based partitioning while producing events onto the above

Re: Stream to KTable internals

We want to make further improvement to stream-table joins. It's just not easy and a larger scoped project. -Matthias On 11/18/21 12:09 PM, Chad Preisler wrote: I'm wondering if the kafka architects have plans to redesign/enhance this behavior. Having to guess the idle time isn't the most satis

Re: KTable updates by non-key field

The following link shows an example of how to make a KTable load like a GlobalKTable and fetch the latest record when you have a cyclic relationship. https://gitlab.com/chad.preisler/kakfainmemorycacheexample You must use the zero timestamp extractor to make this work. I've only used this where t

Issues with Kafka with High Volume of Data

Hello, During my Internship at KLA Inc, we were using Kafka as a streaming platform. We were using it to transfer text and image data from a Linux Machine to a Windows machine. The data is in the form of discrete non serialized records that contain text and image data in the form of blobs. We the

AW: KTable updates by non-key field

Hi Chad, thanks for your input. I thought about just doing the key mappings in the invalidation topics, too. But I guess there is no way around holding the primary key values as a list. Although with just the IDs this should not be a problem. Do you have examples how you implemented this stuff