On Thu, Sep 7, 2017 at 11:10 AM Joe F <j...@apache.org> wrote: > Sure. Setting N=0 (or 1), and you will get this behavior. In a large (1M > topics) system, I could, say set N = 1000, and leave normal queue > consumption out of this mechanism, (to reduce the ZK writes). So this > approach is flexible enough to handle both situations. >
To be precise, these are BookKeeper writes that would be happening anyway (when the cursor is updated). The only difference is that the size of that BK entry increases when there are "holes", up to 154Kb for 10K holes. Also these BK writes for cursor updates are already throttled (by default 1 per sec) > Ack holes are not an useful user metric. They are useful as a system > limit - only so many holes can be stored. and as a system metric we can use > that. > Absolutely agree that are not useful to user, though they are the best indicator for triggering a system level alert, so that someone can take a look and explain to user :) > But to me as an user, letting 1M messages go unacked is a problem - even > though Pulsar can store it super-efficient. I see the core user problem as > the unexpected duplicate delivery of a large set of messages on certain > broker failures. As distinct from how Pulsar can store a large set of > holes, and to what limit. > Even if we can store a large set of messages super-efficient, it does not > resolve the end user pain experienced today - which is a large set of old > messages getting delivered on broker failure. Why did you dump 1M messages > on me all of a sudden after a week? -- that is the question we have run > into many times. So I would prefer that once a certain set limit is > crossed, delivery is stopped/alert generated and the the user fixes the > problem. In essence, deal with unacked messages as a first class user > resource limit, like backlog quota. > We could also do that once you reach the max number of "holes" the delivery stops. -- Matteo Merli <mme...@apache.org>
