I wonder what would I have seen in the logs if someone had done something
like this:
bin/pulsar-admin persistent reset-cursor --time 3h
persistent://tenant/namespace/topic
On Thu, Mar 17, 2022 at 1:54 PM Tecno Brain <cerebrotecnolog...@gmail.com>
wrote:
> Hi Penghui,
> No, we are not seeing messages "disappear" because of TTL.
> What we observed is that messages from 3 out of 20 topics were
> reprocessed.
> We initially thought that the messages were written again into the topic
> by our own applications, but we did not find any evidence of that
> happening. Our applications logs are pretty good and we would have found
> some evidence.
> We couldn't find the reason.
> Our hypothesis was that the cursor was lost and I was trying to find a
> way to verify that hypothesis through the Pulsar logs...looking if we had
> lost a broker or a bookie.
> Initially, I thought that perhaps those 3 topics belonged to the same
> "bundle" and whenever the broker changed, the cursor was lost.
> But Pulsar stores the cursor in the bookies, not the broker....so, a
> broker change shouldn't affect the cursor for the subscription (a shared
> subscription)
> We haven't observed the same issue again.
>
>
>
> On Sat, Mar 12, 2022 at 7:21 AM PengHui Li <peng...@apache.org> wrote:
>
>> If you have TTL, the messages will be expired.
>> You mean "cursor was lost", how do you verify this?
>> to list subscriptions or not able to consume the message?
>> If it is the latter, I think it should be related to the message TTL.
>>
>> And how about the "brokerDeleteInactiveTopicsMode" in your broker
>> settings?
>> If you are using "delete_when_subscriptions_caught_up", after all the
>> message
>> been expired, the topic will be deleted automatically by default.
>>
>> Penghui
>>
>> On Sat, Mar 12, 2022 at 5:28 AM Tecno Brain <cerebrotecnolog...@gmail.com>
>> wrote:
>>
>>> We do have a backlog quota and messageTTL
>>>
>>> Our namespace is configured as follows:
>>>
>>> Backlog quota:
>>>
>>>
>>> *"message_age BacklogQuotaImpl(limit=-1, limitSize=-1,
>>> limitTime=180000, policy=consumer_backlog_eviction)"*
>>> Retention:
>>>
>>>
>>>
>>>
>>> *{ "retentionTimeInMinutes" : 0, "retentionSizeInMB" : 0}*
>>>
>>> Message TTL:
>>>
>>> *172800*
>>>
>>> All topics are partitioned.
>>>
>>> *{
>>> "partitions" : 3
>>> }*
>>>
>>>
>>> On Thu, Mar 10, 2022 at 11:24 PM PengHui Li <codelipeng...@gmail.com>
>>> wrote:
>>>
>>>> Have you changed the backlog quota policy or enabled the message TTL?
>>>> Pulsar will not remove any cursors or skip messages by default.
>>>>
>>>> Penghui
>>>> On Mar 11, 2022, 1:25 PM +0800, Tecno Brain <
>>>> cerebrotecnolog...@gmail.com>, wrote:
>>>>
>>>> Hello,
>>>> I have an application using Pulsar just as JMS (we get single
>>>> messages, acknowledge them when we are done processing it)
>>>> The entire system, composed of several types of apps, uses about 40
>>>> different topics.
>>>>
>>>> Yesterday, an application that subscribes to about 20 queues, suddenly
>>>> was inundated with thousands of messages from two of the queues but I could
>>>> not track those messages to an application producing them. We found that
>>>> the messages were *duplicates*.
>>>> So it seems that the cursor to these two topics was lost, and
>>>> messages from 3 hours earlier were consumed again. I found the following
>>>> paragraph :
>>>>
>>>> "Each subscription stores a cursor. The cursor is the current offset
>>>> in the log. Subscriptions store their cursor in BookKeeper in ledgers. This
>>>> makes cursor tracking scalable just like topics."
>>>> (
>>>> https://jack-vanlightly.com/blog/2018/10/2/understanding-how-apache-pulsar-works
>>>> )
>>>>
>>>> My guess is that the cursor was lost.
>>>> How could I verify that this was the case? I can't find anything
>>>> relevant in the logs.
>>>>
>>>> The only message I found occurring around the same time is
>>>>
>>>> New ensemble:
>>>> [pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local:3181,
>>>> pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181,
>>>> pulsar-bookie-3.pulsar-bookie.pulsar.svc.cluster.local:3181] is not
>>>> adhering to Placement Policy
>>>>
>>>> Any pointers are appreciated.
>>>>
>>>>
>>>>
>>>>
