sijie closed pull request #1507: Using Pulsar as a message queue (cookbook)
URL: https://github.com/apache/incubator-pulsar/pull/1507
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/site/_data/sidebar.yaml b/site/_data/sidebar.yaml
index a4eeb0ba5c..b9be90b762 100644
--- a/site/_data/sidebar.yaml
+++ b/site/_data/sidebar.yaml
@@ -140,6 +140,8 @@ groups:
endpoint: RetentionExpiry
- title: Encryption
endpoint: Encryption
+ - title: Using Pulsar as a message queue
+ endpoint: message-queue
- title: Developing Pulsar
dir: project
diff --git a/site/docs/latest/cookbooks/Encryption.md
b/site/docs/latest/cookbooks/Encryption.md
index 5ef80c1aa4..942ce1f7b5 100644
--- a/site/docs/latest/cookbooks/Encryption.md
+++ b/site/docs/latest/cookbooks/Encryption.md
@@ -5,6 +5,7 @@ tags:
- crypto
- encryption
- clients
+- cookbook
---
<!--
diff --git a/site/docs/latest/cookbooks/PartitionedTopics.md
b/site/docs/latest/cookbooks/PartitionedTopics.md
index aa6004d903..deb495419e 100644
--- a/site/docs/latest/cookbooks/PartitionedTopics.md
+++ b/site/docs/latest/cookbooks/PartitionedTopics.md
@@ -6,6 +6,7 @@ tags:
- partitioning
- admin
- clients
+- cookbook
---
<!--
diff --git a/site/docs/latest/cookbooks/RetentionExpiry.md
b/site/docs/latest/cookbooks/RetentionExpiry.md
index 68448c031c..8191e56684 100644
--- a/site/docs/latest/cookbooks/RetentionExpiry.md
+++ b/site/docs/latest/cookbooks/RetentionExpiry.md
@@ -1,7 +1,7 @@
---
title: Message retention and expiry
lead: Manage how long messages remain stored in your Pulsar instance
-tags: [admin, expiry, retention, backlog]
+tags: [admin, expiry, retention, backlog, cookbook]
---
<!--
diff --git a/site/docs/latest/cookbooks/message-queue.md
b/site/docs/latest/cookbooks/message-queue.md
new file mode 100644
index 0000000000..dd2a77d0ae
--- /dev/null
+++ b/site/docs/latest/cookbooks/message-queue.md
@@ -0,0 +1,113 @@
+---
+title: Using Pulsar as a message queue
+lead: Although Pulsar is typically known as a real-time messaging system, it's
also an excellent choice for a queuing system
+tags: [clients, java, python, message queue, cookbook]
+---
+
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+
+Message queues are essential components of many large-scale data
architectures. If every single work object that passes through your system
absolutely *must* be processed in spite of the slowness or downright failure of
this or that system component, there's a good chance that you'll need a message
queue to step in and ensure that unprocessed data is retained---with correct
ordering---until the required actions are taken.
+
+Pulsar is a great choice for a message queue because:
+
+* it was built with [persistent message
storage](../../getting-started/ConceptsAndArchitecture#persistent-storage) in
mind
+* it offers automatic load balancing across {% popover consumers %} for
messages on a topic (or custom load balancing if you wish)
+
+{% include admonition.html type="success" content="You can use the same Pulsar
installation to act as a real-time message bus and as a message queue if you
wish (or just one or the other). You can set aside some topics for real-time
purposes and other topics for message queue purposes (or use specific
namespaces for either purpose if you wish)." %}
+
+## Client configuration changes
+
+To use a Pulsar {% popover topic %} as a message queue, you should distribute
the receiver load on that topic across several {% popover consumers %} (the
optimal number of consumers will depend on the load). Each consumer must:
+
+* Establish a [shared
subscription](../../getting-started/ConceptsAndArchitecture#shared) and use the
same subscription name as the other consumers (otherwise the subscription is
not shared and the consumers can't act as a processing ensemble)
+* If you'd like to have tight control over message dispatching across
consumers, set the consumers' **receiver queue** size very low (potentially
even to 0 if necessary). Each Pulsar {% popover consumer %} has a receiver
queue that determines how many messages the consumer will attempt to fetch at a
time. A receiver queue of 1000 (the default), for example, means that the
consumer will attempt to process 1000 messages from the topic's backlog upon
connection. Setting the receiver queue to zero essentially means ensuring that
each consumer is only doing one thing at a time.
+
+ The downside to restricting the receiver queue size of consumers is that
that limits the potential throughput of those consumers and cannot be used with
{% popover partitioned topics %}. Whether the performance/control trade-off is
worthwhile will depend on your use case.
+
+### Java clients {#java}
+
+Here's an example Java consumer configuration that uses a shared subscription:
+
+```java
+import org.apache.pulsar.client.api.Consumer;
+import org.apache.pulsar.client.api.PulsarClient;
+import org.apache.pulsar.client.api.SubscriptionType;
+
+String SERVICE_URL = "pulsar://localhost:6650";
+String TOPIC = "persistent://sample/standalone/ns1/mq-topic-1";
+String subscription = "sub-1";
+
+PulsarClient client = PulsarClient.builder()
+ .serviceUrl(SERVICE_URL)
+ .build();
+
+Consumer consumer = client.newConsumer()
+ .topic(TOPIC)
+ .subscriptionName(subscription)
+ .subscriptionType(SubscriptionType.Shared)
+ // If you'd like to restrict the receiver queue size
+ .receiverQueueSize(10)
+ .subscribe();
+```
+
+### Python clients {#python}
+
+Here's an example Python consumer configuration that uses a shared
subscription:
+
+```python
+from pulsar import Client, ConsumerType
+
+SERVICE_URL = "pulsar://localhost:6650"
+TOPIC = "persistent://sample/standalone/ns1/mq-topic-1"
+SUBSCRIPTION = "sub-1"
+
+client = Client(SERVICE_URL)
+consumer = client.subscribe(
+ TOPIC,
+ SUBSCRIPTION,
+ # If you'd like to restrict the receiver queue size
+ receiver_queue_size=10,
+ consumer_type=ConsumerType.Shared)
+```
+
+### C++ clients {#cpp}
+
+Here's an example C++ consumer configuration that uses a shared subscription:
+
+```cpp
+#include <pulsar/Client.h>
+
+std::string serviceUrl = "pulsar://localhost:6650";
+std::string topic = "persistent://sample/standalone/ns1/mq-topic-1";
+std::string subscription = "sub-1";
+
+Client client(serviceUrl);
+
+ConsumerConfiguration consumerConfig;
+consumerConfig.setConsumerType(ConsumerType.ConsumerShared);
+// If you'd like to restrict the receiver queue size
+consumerConfig.setReceiverQueueSize(10);
+
+Consumer consumer;
+
+Result result =
client.subscribe("persistent://sample/standalone/ns1/my-topic", subscription,
consumerConfig, consumer);
+```
\ No newline at end of file
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
