merlimat commented on a change in pull request #1507: Using Pulsar as a message 
queue (cookbook)
URL: https://github.com/apache/incubator-pulsar/pull/1507#discussion_r180496422
 
 

 ##########
 File path: site/docs/latest/cookbooks/message-queue.md
 ##########
 @@ -0,0 +1,89 @@
+---
+title: Using Pulsar as a message queue
+lead: Although Pulsar is typically known as a real-time messaging system, it's 
also an excellent choice for a queuing system
+tags: [clients, java, python, message queue, cookbook]
+---
+
+Message queues are essential components of many large-scale data 
architectures. If every single work object that passes through your system 
absolutely *must* be processed in spite of the slowness or downright failure of 
this or that system component, there's a good chance that you'll need a message 
queue to step in and ensure that unprocessed data is retained---with correct 
ordering---until the required actions are taken.
+
+Pulsar is a great choice for a message queue because:
+
+* it was built with [persistent message 
storage](../../getting-started/ConceptsAndArchitecture#persistent-storage) in 
mind
+* it offers automatic load balancing across {% popover consumers %} for 
messages on a topic (or custom load balancing if you wish)
+
+{% include admonition.html type="success" content="You can use the same Pulsar 
installation to act as a real-time message bus and as a message queue if you 
wish (or just one or the other). You can set aside some topics for real-time 
purposes and other topics for message queue purposes (or use specific 
namespaces for either purpose if you wish)." %}
+
+## Client configuration changes
+
+To use a Pulsar {% popover topic %} as a message queue, you should distribute 
the receiver load on that topic across several {% popover consumers %}. Each 
consumer must:
+
+* Have a [shared 
subscription](../../getting-started/ConceptsAndArchitecture#shared) and use the 
same subscription name as the other consumers (otherwise the subscription is 
not shared)
+* Have the receiver queue size set to zero (or very low). Each Pulsar {% 
popover consumer %} has a **receiver queue** that determines how many messages 
the consumer will attempt to fetch at a time. A receiver queue of 1000 (the 
default), for example, means that the consumer will attempt to process 1000 
messages from the topic's backlog upon connection. Setting the receiver queue 
to zero essentially means ensuring that each consumer is only doing one thing 
at a time.
 
 Review comment:
   I don't think we should recommend to always reduce the receiver queue size. 
A reduced receiver queue size can limit the throughput a particular consumer 
can receive.
   
   In particular, the `0` receiver queue size, should be only used when having 
a super-tight control on the dispatching is really necessary, because it's very 
chatty (client <->broker) and it doesn't work when consuming from a partitioned 
topic.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Reply via email to