sijie commented on a change in pull request #4168: [Issue 4167] [pulsar-broker]
Support to get the number of current messages
URL: https://github.com/apache/pulsar/pull/4168#discussion_r279597913
##########
File path:
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java
##########
@@ -1522,4 +1525,38 @@ protected MessageId internalGetLastMessageId(boolean
authoritative) {
return messageId;
}
+
+ protected long internalGetMessageCount(boolean authoritative) {
+ validateAdminOperationOnTopic(authoritative);
+
+ if (!(getTopicReference(topicName) instanceof PersistentTopic)) {
+ log.error("[{}] Not supported operation of non-persistent topic
{}", clientAppId(), topicName);
+ throw new RestException(Status.METHOD_NOT_ALLOWED,
+ "GetMessageCount on a non-persistent topic is not allowed");
+ }
+
+ Reader<byte[]> reader = null;
+ final AtomicLong count = new AtomicLong();
+
+ try {
+ reader = pulsar().getClient().newReader()
Review comment:
I don't think it is a good idea to implement in this way in the admin
restful service. it is a very expensive operation, which waste the bandwidth.
I think a better approach would be: (what we used to do it at Twitter):
- add some sort of metadata into the message, indicating its position (aka
how many of messages have been persistent) in the ledger.
- when a ledger is closed (due to rollover) or recovered (due to failover),
propagate the number of messages to the metadata in managed ledger.
so the calculation of the number of messages between any two messages:
1) get the first message in the partition. it will tell you where its
position in the ledger.
2) get the last message in the partition. it will tell you where its
position in the ledger.
3) get the metadata from the managed ledger. it will tell you how many
messages stored per ledger.
so you can calculate the total number of messages using the following
formula:
`total-number-of-message-of-first-ledger -
position-of-first-message-in-first-ledger +
sum(number-of-message-in-other-ledgers) +
position-of-last-message-in-last-ledger`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
