> Ideally we want the mark delete to keep pace with the dispatch rate (more or less). When that does not happen, we end up with the a large set of duplicate deliveries on topic reload.
That's not exact. If you delay the ack for a particular message, that information is being stored. After crash/restart, only that message will be re-delivered. > So one way to specify the problem would be to set a size N (configurable, with a sensible default) that would be the operating gap between mark delete and the read cursor. Anything that falls beyond N is an ack hole. The broker needs to remember message by message (or better said, entry by entry in case of batches). > This would be difficult to explain, without users grasping what happens under the covers. I would rather we use the concept and terminology of the "number of unacked messages". But that is a bad metric to prevent issues in the broker. Even if you don't ack 1M messages, if they are consecutives, they won't pose any issue. We just need to store 1 range info: Acked messages: [0, 100) [100000, 2000000) This info is very compact. The limit on the *number* of ack-holes is to limit the number of ranges we need to store. If (per Rajan's test) we can store 10K disjointed ranges with 154KB size, I think that's a pretty good trade off. You can acknowledge out-of-order and the broker will safely store that information, up to a certain limit. The 10K limit (for ranges) should be high enough to not be a concern under regular consumer operations, but preventing a misbehaving consumer to cause problem in the system. The "alert" I was mentioning earlier, should be based on this number. If that goes high, it should be good to check with user on how he's using acknowledgements. On Thu, Sep 7, 2017 at 8:07 AM Joe F <j...@apache.org> wrote: > I don't think we should attempt to store "regular" ack holes. What I mean > there is that all shared(queue) consumers will have ack holes in the > regular order of business. So if we store this unconditionally, we will > end up doing this for all queues. > > Ideally we want the mark delete to keep pace with the dispatch rate (more > or less). When that does not happen, we end up with the a large set of > duplicate deliveries on topic reload. > > So one way to specify the problem would be to set a size N (configurable, > with a sensible default) that would be the operating gap between mark > delete and the read cursor. Anything that falls beyond N is an ack hole. > > The other comment I have is that the "ack hole" is a concept that is not > easy for end users to understand. Holes could vary in size. There could be > 10K messages in a single ack hole, or 1000 messages in 1000 ack holes (one > in each. So in one case you could have 10,000 unacked messages and have no > issues, but in the other case you could have 1,001 messages and have > issues. This would be difficult to explain, without users grasping what > happens under the covers. I would rather we use the concept and > terminology of the "number of unacked messages". > > This is an area where we have and continue to face significant user > education and support cases, hence my emphasis on user friendliness. > > Joe > > On Wed, Sep 6, 2017 at 5:02 PM, <g...@git.apache.org> wrote: > >> merlimat commented on issue #742: Avoid huge backlog on topic reloading: >> due to large gap between markDelete-offset and read-position of cursor. >> URL: >> https://github.com/apache/incubator-pulsar/issues/742#issuecomment-327644681 >> >> >> > where large ack-holes can create backlog >> >> large *number* of ack-holes >> >> > So, I think having option to restrict distance between markDelete >> and readPosition is something useful for broker. >> >> It's not related at all with the distance. I think the better metric >> is indeed the number of "holes" >> >> ---------------------------------------------------------------- >> This is an automated message from the Apache Git Service. >> To respond to the message, please log on GitHub and use the >> URL above to go to the specific comment. >> >> For queries about this service, please contact Infrastructure at: >> us...@infra.apache.org >> >> >> With regards, >> Apache Git Services >> > > -- Matteo Merli <mme...@apache.org>
