sijie commented on pull request #9448: URL: https://github.com/apache/pulsar/pull/9448#issuecomment-773132055
@eolivelli Unfortunately, I disagree with you on this. This approach creates a very bad example for other people to follow. First of all, it basically creates a "connector" class per schema type. This is a very bad practice. I would discourage a connector implementation going down this route. It is impossible to maintain. Secondly, it solves a very narrow scoped problem by introducing a specific type of source connector in which the key is a string and the value is AVRO. All the key schema information is dropped. Key schema information is important to a lot of streaming use cases. You can't solve the problem with this approach. Following your approach, you will end up creating N*N connector classes (where N is the number of schema types). I would encourage people not introducing a code change that is specialized for their own needs. The connector implementation should be beneficial to broader users. > I would like not to enter the details of how Kafka and Confluent serialize data Unfortunately, the Kafka AVRO is a confluent open-source thing not a Kafka community thing. We are always in the game of touching serialization details when converting a message from X format to Y format. You either do the conversation at a high level using abstraction or at a low level by realizing the serialization details. The approach you proposed also has a bad performance because it will churn a lot of object allocations. Realizing the serialization details can save a lot of memory copy and serialization/deserialization. > we allowing Sources to push GenericRecords, This is a good initiative. But it should be isolated from this KafkaSource change. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
