GitHub user fabrizziocht edited a discussion: Is RabbitMQ source connectors safe?
I was exploring the rabbitMQ source connector (https://github.com/apache/pulsar/blob/v4.0.1/pulsar-io/rabbitmq/src/main/java/org/apache/pulsar/io/rabbitmq/RabbitMQSource.java) that implements the class PushSource (it extends AbstractPushSource class), I can see that each time a message is collected by the RabbitMQConsumer class using the handleDelivery method it invokes the RabbitMQSource consume: https://github.com/apache/pulsar/blob/v4.0.1/pulsar-io/rabbitmq/src/main/java/org/apache/pulsar/io/rabbitmq/RabbitMQSource.java#L99 Then it stores only in memory in a LinkedBlockingQueue and the ack is sent to RabbitMQ: https://github.com/apache/pulsar/blob/master/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/AbstractPushSource.java#L76 What happens if the pulsar IO service is restarted? I can see that the data stored in the LinkedBlockingQueue will be lost, because the message was already acked and removed in rabbitMQ and also was not still written in the pulsar topic. Is it a limitation of the rabbitmq source connector? or all connectors that implements the PushSource has the problem? Exists another interface/class that can guarantee that the message was written first in the pulsar topic before send the ack to rabbitMQ? GitHub link: https://github.com/apache/pulsar/discussions/23677 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
