I found this in the logs:
* Failed to mark delete while trimming data ledgers: Reset cursor in
progress - unable to mark delete position 18377:-1*
but although the messages indicate different topics, the time matches when
we started to see messages replayed.
What causes that message? What are the consequences of it?
On Fri, Mar 18, 2022 at 1:31 PM Tecno Brain <cerebrotecnolog...@gmail.com>
wrote:
> I wonder what would I have seen in the logs if someone had done something
> like this:
>
> bin/pulsar-admin persistent reset-cursor --time 3h
> persistent://tenant/namespace/topic
>
>
> On Thu, Mar 17, 2022 at 1:54 PM Tecno Brain <cerebrotecnolog...@gmail.com>
> wrote:
>
>> Hi Penghui,
>> No, we are not seeing messages "disappear" because of TTL.
>> What we observed is that messages from 3 out of 20 topics were
>> reprocessed.
>> We initially thought that the messages were written again into the
>> topic by our own applications, but we did not find any evidence of that
>> happening. Our applications logs are pretty good and we would have found
>> some evidence.
>> We couldn't find the reason.
>> Our hypothesis was that the cursor was lost and I was trying to find a
>> way to verify that hypothesis through the Pulsar logs...looking if we had
>> lost a broker or a bookie.
>> Initially, I thought that perhaps those 3 topics belonged to the same
>> "bundle" and whenever the broker changed, the cursor was lost.
>> But Pulsar stores the cursor in the bookies, not the broker....so, a
>> broker change shouldn't affect the cursor for the subscription (a shared
>> subscription)
>> We haven't observed the same issue again.
>>
>>
>>
>> On Sat, Mar 12, 2022 at 7:21 AM PengHui Li <peng...@apache.org> wrote:
>>
>>> If you have TTL, the messages will be expired.
>>> You mean "cursor was lost", how do you verify this?
>>> to list subscriptions or not able to consume the message?
>>> If it is the latter, I think it should be related to the message TTL.
>>>
>>> And how about the "brokerDeleteInactiveTopicsMode" in your broker
>>> settings?
>>> If you are using "delete_when_subscriptions_caught_up", after all the
>>> message
>>> been expired, the topic will be deleted automatically by default.
>>>
>>> Penghui
>>>
>>> On Sat, Mar 12, 2022 at 5:28 AM Tecno Brain <
>>> cerebrotecnolog...@gmail.com> wrote:
>>>
>>>> We do have a backlog quota and messageTTL
>>>>
>>>> Our namespace is configured as follows:
>>>>
>>>> Backlog quota:
>>>>
>>>>
>>>> *"message_age BacklogQuotaImpl(limit=-1, limitSize=-1,
>>>> limitTime=180000, policy=consumer_backlog_eviction)"*
>>>> Retention:
>>>>
>>>>
>>>>
>>>>
>>>> *{ "retentionTimeInMinutes" : 0, "retentionSizeInMB" : 0}*
>>>>
>>>> Message TTL:
>>>>
>>>> *172800*
>>>>
>>>> All topics are partitioned.
>>>>
>>>> *{
>>>> "partitions" : 3
>>>> }*
>>>>
>>>>
>>>> On Thu, Mar 10, 2022 at 11:24 PM PengHui Li <codelipeng...@gmail.com>
>>>> wrote:
>>>>
>>>>> Have you changed the backlog quota policy or enabled the message TTL?
>>>>> Pulsar will not remove any cursors or skip messages by default.
>>>>>
>>>>> Penghui
>>>>> On Mar 11, 2022, 1:25 PM +0800, Tecno Brain <
>>>>> cerebrotecnolog...@gmail.com>, wrote:
>>>>>
>>>>> Hello,
>>>>> I have an application using Pulsar just as JMS (we get single
>>>>> messages, acknowledge them when we are done processing it)
>>>>> The entire system, composed of several types of apps, uses about 40
>>>>> different topics.
>>>>>
>>>>> Yesterday, an application that subscribes to about 20 queues,
>>>>> suddenly was inundated with thousands of messages from two of the queues
>>>>> but I could not track those messages to an application producing them. We
>>>>> found that the messages were *duplicates*.
>>>>> So it seems that the cursor to these two topics was lost, and
>>>>> messages from 3 hours earlier were consumed again. I found the following
>>>>> paragraph :
>>>>>
>>>>> "Each subscription stores a cursor. The cursor is the current offset
>>>>> in the log. Subscriptions store their cursor in BookKeeper in ledgers.
>>>>> This
>>>>> makes cursor tracking scalable just like topics."
>>>>> (
>>>>> https://jack-vanlightly.com/blog/2018/10/2/understanding-how-apache-pulsar-works
>>>>> )
>>>>>
>>>>> My guess is that the cursor was lost.
>>>>> How could I verify that this was the case? I can't find anything
>>>>> relevant in the logs.
>>>>>
>>>>> The only message I found occurring around the same time is
>>>>>
>>>>> New ensemble:
>>>>> [pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local:3181,
>>>>> pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181,
>>>>> pulsar-bookie-3.pulsar-bookie.pulsar.svc.cluster.local:3181] is not
>>>>> adhering to Placement Policy
>>>>>
>>>>> Any pointers are appreciated.
>>>>>
>>>>>
>>>>>
>>>>>
