If you have TTL, the messages will be expired.
You mean "cursor was lost", how do you verify this?
to list subscriptions or not able to consume the message?
If it is the latter, I think it should be related to the message TTL.
And how about the "brokerDeleteInactiveTopicsMode" in your broker settings?
If you are using "delete_when_subscriptions_caught_up", after all the
message
been expired, the topic will be deleted automatically by default.
Penghui
On Sat, Mar 12, 2022 at 5:28 AM Tecno Brain <cerebrotecnolog...@gmail.com>
wrote:
> We do have a backlog quota and messageTTL
>
> Our namespace is configured as follows:
>
> Backlog quota:
>
>
> *"message_age BacklogQuotaImpl(limit=-1, limitSize=-1,
> limitTime=180000, policy=consumer_backlog_eviction)"*
> Retention:
>
>
>
>
> *{ "retentionTimeInMinutes" : 0, "retentionSizeInMB" : 0}*
>
> Message TTL:
>
> *172800*
>
> All topics are partitioned.
>
> *{
> "partitions" : 3
> }*
>
>
> On Thu, Mar 10, 2022 at 11:24 PM PengHui Li <codelipeng...@gmail.com>
> wrote:
>
>> Have you changed the backlog quota policy or enabled the message TTL?
>> Pulsar will not remove any cursors or skip messages by default.
>>
>> Penghui
>> On Mar 11, 2022, 1:25 PM +0800, Tecno Brain <cerebrotecnolog...@gmail.com>,
>> wrote:
>>
>> Hello,
>> I have an application using Pulsar just as JMS (we get single messages,
>> acknowledge them when we are done processing it)
>> The entire system, composed of several types of apps, uses about 40
>> different topics.
>>
>> Yesterday, an application that subscribes to about 20 queues, suddenly
>> was inundated with thousands of messages from two of the queues but I could
>> not track those messages to an application producing them. We found that
>> the messages were *duplicates*.
>> So it seems that the cursor to these two topics was lost, and messages
>> from 3 hours earlier were consumed again. I found the following paragraph :
>>
>> "Each subscription stores a cursor. The cursor is the current offset in
>> the log. Subscriptions store their cursor in BookKeeper in ledgers. This
>> makes cursor tracking scalable just like topics."
>> (
>> https://jack-vanlightly.com/blog/2018/10/2/understanding-how-apache-pulsar-works
>> )
>>
>> My guess is that the cursor was lost.
>> How could I verify that this was the case? I can't find anything relevant
>> in the logs.
>>
>> The only message I found occurring around the same time is
>>
>> New ensemble:
>> [pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local:3181,
>> pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181,
>> pulsar-bookie-3.pulsar-bookie.pulsar.svc.cluster.local:3181] is not
>> adhering to Placement Policy
>>
>> Any pointers are appreciated.
>>
>>
>>
>>
