jerrypeng opened a new pull request #10315: URL: https://github.com/apache/pulsar/pull/10315
### Motivation Currently, the internal PulsarSource used in the Pulsar Function framework used to read from topics extend the PushSource. This is done to support the use case in which multiple consumers have to be created to read from multiple topics with different consumer settings. While this approach is necessary for reading from multiple topics with consumers with different settings, it is inefficient for reading a single topic or list of topics with the same consumer settings (not address in this PR). The inefficiency is due to the fact that for PushSource all messages are push onto an intermediate queue and then pop off to be processed. This intermediate queueing is unnecessary for reading from one topic. By removing having to enqueue and dequeue everyone message from this intermediate queue performance can improve around 20% ### Modifications Create separate code paths for when reading from one topic vs multiple topics. Two different PulsarSource implementations are introduced. SingleConsumerPulsarSource, which is used when only one consumer is needed to be created i.e. reading from one topic. MultiConsumerPulsarSource, which is used when multiple consumers are needed to be created i.e. reading from multiple topics with different consumer settings. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
