Hi - My thoughts here may be completely useless but I wonder if clients can add an optional argument to the broker call when pulling events. The argument would be the amount of delay. Any messages younger than the delay are not returned by the broker.
Regards, Dave > On Feb 19, 2019, at 11:47 AM, Ezequiel Lovelle <ezequiellove...@gmail.com> > wrote: > >> The recent changes made to support DLQ caused major problems with garbage > collection > > If garbage collection is a big concern maybe we could add some config > parameter on the broker to disable the usage of this feature and return > BrokerMetadataException in this situation, giving the power to the > administrator whether to offer this feature or not. > >> is it acceptable to do it at broker side? > > I think this is the big question that needs to be answered. > >> can we just have a separated dispatcher for fixed delayed subscription? > > I will try to do a completely new approach, simpler, and more isolated > from broker logic. Maybe some way to plug to the broker some logic without > interfering with its core? > > In our business fixed delay at consumer level regardless of any producer > configuration is a big win due to easy implementation and usage. > > -- > *Ezequiel Lovelle* > > > On Wed, 13 Feb 2019 at 23:25, Sijie Guo <guosi...@gmail.com> wrote: > >> Agreed that dispatcher is a performance sensitive piece of code. Feel bad >> to hear that DLQ causes GC. Are there any issues tracking those items you >> guys identified with DLQ changes? >> >>> How is this different from a subscription running behind? >> >> As far as I understand form the discussion at #3155, I don't think there is >> a fundamental difference from a backlogged subscriber. >> The discussion point will mainly be - if a delayed subscription can be >> implemented with a simpler approach at broker side without changing other >> dispatcher logic, >> is it acceptable to do it at broker side? So we don't have to reimplement >> the same mechanism at different language clients. I think that's the same >> tradeoff we were discussing for generic delayed messages. >> >> My thought would be - can we just have a separated dispatcher for fixed >> delayed subscription? The logic can be ISOLATED from other normal >> dispatchers. if users don't enable delayed subscription, they will not >> exercise that dispatcher. This can be a good direction to explore for >> future changes that are related to dispatchers. >> >> - Sijie >> >> >> On Thu, Feb 14, 2019 at 8:43 AM Joe F <joefranc...@gmail.com> wrote: >> >>> Delayed subscription is simpler, and probably worth doing in the broker >> IF >>> done right. >>> >>> How is this different from a subscription running behind? Why does >>> supporting that require this complex a change in the dispatcher, when we >>> already support backlogged subscribers? >>> >>> I am extremely wary of changes in the dispatcher. The recent changes made >>> to support DLQ caused major problems with garbage collection, broker >>> failure and service interruptions for us. Even though we ARE NOT using >> the >>> DLQ feature. Not a pleasant experience. >>> >>> This is a very performance sensitive piece of code, and it should be >>> treated as such. >>> >>> Joe >>> >>> >>> >>> On Wed, Feb 13, 2019 at 3:58 PM Sijie Guo <guosi...@gmail.com> wrote: >>> >>>> Hi all, >>>> >>>> I am going to wrap up the discussion regarding delayed delivery use >>> cases. >>>> >>>> For arbitrary delayed delivery, there are a few +1s to doing PIP-26 in >>>> functions. I am assuming that we will go down this path, unless there >> are >>>> other proposals. >>>> >>>> However there is a use case Lovelle pointed out about "Fixed Delayed >>>> Message". More specifically it is >>>> https://github.com/apache/pulsar/pull/3155 >>>> (The caption in #3155 is a bit misleading). IMO it is a "delayed >>>> subscription", basically all messages in the subscription is delayed to >>>> dispatch in a given time interval. The consensus of this feature is not >>> yet >>>> achieved. Basically, there will be two approaches for this: >>>> >>>> a) DONT treat "fixed delayed message" as a different case. Just use the >>>> same approach as in PIP-26. >>>> b) treat "fixed delayed message" as a different case, e.g. we can >> better >>>> call it "delayed subscription" or whatever can distinguish it from >>> general >>>> arbitrary delayed delivery. Use the approach proposed/discussed in >> #3155. >>>> >>>> I would like the community to discuss this and also come to an >> agreement. >>>> So Lovelle can move forward with the approach agreed by the community. >>>> >>>> Thanks, >>>> Sijie >>>> >>>> On Tue, Jan 29, 2019 at 6:30 AM Ezequiel Lovelle < >>>> ezequiellove...@gmail.com> >>>> wrote: >>>> >>>>> "I agree, but that is *not what #3155 tries to achieve." >>>>> >>>>> This typo made this phrase nonsense, sorry! >>>>> >>>>> On Mon, 28 Jan 2019, 16:44 Ezequiel Lovelle < >> ezequiellove...@gmail.com >>>>> wrote: >>>>> >>>>>>> What exactly is the delayed delivery use case? >>>>>> >>>>>> This is helpful on systems relaying on pulsar for persistent >>> guarantees >>>>>> and using it for synchronization or some sort of checks, but on >> such >>>>>> systems is common to have some overhead committing data on >> persistent >>>>>> storage maybe due to buffered mechanism or distributing the data >>> across >>>>>> the network before being available. >>>>>> >>>>>> Surely would be more use cases I don't came across right now. >>>>>> >>>>>>> Random insertion and deletion is not what FIFO queues like Pulsar >>> are >>>>>> designed for. >>>>>> >>>>>> I agree, but that is now what #3155 tries to achieve. #3155 is >> just a >>>>>> fixed delay for all message in a consumer, that's the reason that >> the >>>>>> implementation of #3155 is quite trivial. >>>>>> >>>>>> +1 from me for doing PIP-26 in functions. >>>>>> >>>>>> -- >>>>>> *Ezequiel Lovelle* >>>>>> >>>>>> >>>>>> On Sat, 26 Jan 2019 at 09:57, Yuva raj <uvar...@gmail.com> wrote: >>>>>> >>>>>>> Considering the way pulsar is built +1 for doing PIP-26 in >>> functions. >>>> I >>>>> am >>>>>>> more of thinking in a way like publish it pulsar we will make it >>>>> available >>>>>>> in a different queuing system if you need priority and delay >>> messages >>>>>>> support. Pulsar functions would go enough for this kind of use >>> cases. >>>>>>> >>>>>>> On Fri, 25 Jan 2019 at 22:29, Ivan Kelly <iv...@apache.org> >> wrote: >>>>>>> >>>>>>>>> Correct. PIP-26 can be implemented in Functions. I believe the >>>> last >>>>>>>>> discussion in PIP-26 thread kind of agree on functions >> approach. >>>>>>>>> If the community is okay with PIP-26 in functions, I think >> that >>> is >>>>>>>> probably >>>>>>>>> a good approach to start. >>>>>>>> >>>>>>>> +1 for doing it in functions. >>>>>>>> >>>>>>>> -Ivan >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Thanks* >>>>>>> >>>>>>> *Yuvaraj L* >>>>>>> >>>>>> >>>>> >>>> >>> >>
