Hi Navneeth, Yes, you are correct. I think there are some opportunities for improvement there, but there are also reasons for it to be serialized in the in-memory store.
Off the top of my head, we need to serialize stored data anyway to send it to the changelog. Also, even though the store is in memory, there is still a cache (which helps reduce the update rate downstream); the cache stores serialized data because it’s the same cache covering all store types, and because it allows us to estimate and bound memory usage. Also, there are times when we need to determine if the current value is the same as the prior value; and comparing serialized forms is more reliable than using the equals() method. Round-tripping through serialization also helps to limit the scope of mutable object bugs, which I have seen in some user code. Finally, it simplifies the overall API and internals to have one kind of store to plug in, rather than to have some byte stores and some object stores. All that aside, I’ve always felt that it could be a significant performance advantage to do what you suggest. If you feel passionate about it and want to do some experiments, I’d be happy to provide some guidance. Just let me know! Thanks, John On Sun, Dec 20, 2020, at 14:27, Navneeth Krishnan wrote: > Hi All, > > I have a question about the inMemoryKeyValue store. I was under the > assumption that in-memory stores would not serialize the objects but when I > looked into the implementation I see InMemoryKeyValueStore uses a > NavigableMap of bytes which indicates the user objects have to be > serialized and stored. > > Am I missing something? Wouldn't this cause more serialization overhead for > storing something in memory? > > In my case I have a punctuator which reads all the entries in the state > store and forwards the data. When there are around 10k entries it takes > about 400ms to complete. I was trying to optimize this problem. I use kryo > serde and the objects are bigger in size (approx 500 bytes). > > Regards, > Navneeth >
