codelipenghui commented on issue #11962: URL: https://github.com/apache/pulsar/issues/11962#issuecomment-915711069
> If the user only had Kafka clients for now, they can choose entry.format=kafka for high performance. If one day they switched to Pulsar clients, they can configure message converters to consume old data. If the user already had Kafka clients and Pulsar clients for the same topics, they must choose entry.format=pulsar now without PIP 94. In future, when they eliminate all Kafka clients, even including KoP itself, they'll find there's no way to consume old data. @BewareMyPower Yes, I know this case. Use `entry.format=pulsar`, the message publishing side burden the broker. With PIP-94 consume from Pulsar client burden the broker, In my opinion, it is difficult for users to choose, if you have only one subscription to read the data from the topic by the Pulsar client, you might choose broker converter, but if you have 100 subscriptions to read the data from the topic by Pulsar client, you might want to choose the `entry.format=pulsar`. And If we are try to introduce the data format processor at the client-side again, users will have 3 options, `entry.format` for KoP, messageConverter for the broker, data format processor for the client-side. This will make it very complicated to use. In my opinion, 1. if users want to achieve high performance, they should avoid the broker-side data format conversion. 2. If users do not care about the performance, they can use `entry.format=pulsar`, no need to add more dependencies at the client-side. The proposal looks like an `intermediate` approach for users who want to get high performance. I agree PIP-94 can bring performance improvements to certain scenarios, but considering the complexity and the broker resource consumption that PIP-94 might bring to Pulsar, we need to think about it carefully. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
