BewareMyPower commented on issue #11962: URL: https://github.com/apache/pulsar/issues/11962#issuecomment-915269780
@codelipenghui The GC problem for KoP is caused by the heap memory usage, not the conversion, the converter only affects CPU theoretically because all memory usages of messages are from direct memory. > I think we need to clarify the purpose of this proposal, we are trying to find a way to support efficient data conversion between different formats such as Kafka, Pulsar. It's not. Introducing data format in KoP is to avoid the data conversion, right. Because we tends to push users to use pure Kafka client when using KoP. However, if they want to turn to Pulsar client later, it could be impossible because there's no way for Pulsar consumer to consume these messages produced by Kafka producer. Actually, it's right the conversion should be performed at client side. But upgrading the client might not be always easy. @eolivelli was right that this is not always possible because downstream users may not be aware of the format of the data. > If users tend to convert the data at the broker side, the existing KoP implementation already supports this requirement. No. It requires configuring the `entry.format=pulsar` and then the interact among Kafka clients could have a bad performance. In conclusion, supporting message converter at both broker level and client level is necessary. They have both PROs and CONs. - Broker support: All Pulsar clients are supported but the conversion overhead might affect the whole broker. - Client support: Only the latest Pulsar client (e.g. 2.9.0) is supported. Users must upgrade their client version. But the conversion overhead won't affect the whole broker. It's a priority comparison about **availability** vs. **performance**. I chose the **availability** so I tried to implement the Message converter at broker side first. The message converter here can also be applied to Pulsar Client. I'm not sure if it's proper to contain two tasks in a PIP. If yes, I can add the client side support in this proposal. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
