navina commented on PR #9544:
URL: https://github.com/apache/pinot/pull/9544#issuecomment-1271355633
@vvivekiyer I had intentionally tried to stay away from using `Generics` in
this new interface as it make the code hard to maintain. Moreover, I wanted to
make sure that the `StreamConsumer` contract did not include deserializing the
incoming payload (today, it is not really clear
> LinkedIn uses MessageBatch<IndexedRecord> for it's kafka client. Keeping
StreamMessage generic will avoid unnecessary type conversion (which could
involve serializing and deserializing) for such users.
I am curious why Linkedin does this? isn't is expensive to deserialize every
record until when the deserialized payload is actually needed? The consumer's
contract should not involve deserializing the payload. Can you please explain
why this is useful?
I also don't understand the point about "unnecessary type conversion" here.
Can you please elaborate?
Using generics forces the segment manager implementation to deal with raw
usage of parameterized classes (due to type erasures) and make the code hard to
read and maintain.
Besides,`StreamMessage` is meant to abstract the entire incoming payload.
Using only the type of the record's "value" in this generic class seems
prohibitive.
Soliciting feedback from @npawar / @Jackie-Jiang / @kishoreg here.
Thanks!
> Also fixed a NPE in StreamDataDecoderImpl.java where metadata header is
null.
Thank you for fixing this! I recently noticed this in my testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
