Hi Lari, I recommend creating a new implementation rather than directly replacing the existing one. This approach aligns with how we’ve handled several proposals in the past and allows us to maintain stability while introducing improvements
- PIP-192: New Pulsar Broker Load Balancer - PIP-195: New bucket based delayed message tracker Once the new implementation proves to be stable, we can switch the default implementation of the Key_Shared subscription to the new ‘draining hashes’ solution. On Sat, Sep 14, 2024 at 8:40 AM Enrico Olivelli <eolive...@gmail.com> wrote: > Awesome proposal, no questions from my side > > +1 > > Enrico > > Il giorno sab 14 set 2024 alle ore 16:21 Lari Hotari <lhot...@apache.org> > ha scritto: > > > Dear Pulsar Community, > > > > I'd like to propose a new improvement for Pulsar's Key_Shared > > subscription mode, outlined in PIP-379. This proposal aims to address > > several issues with the current implementation and introduce a more > > efficient mechanism for managing message ordering. > > > > Problem: > > The current Key_Shared implementation faces challenges including: > > 1. Complex management of "recently joined consumers" > > 2. Incomplete fulfillment of ordering guarantees > > 3. Unnecessary message blocking > > 4. Poor observability > > > > PIP-379 introduces a "draining hashes" concept to efficiently manage > > message ordering by tracking affected hashes when consumer assignments > > change. The high-level solution is drafted in the PIP document. > > > > Benefits: > > 1. Improved message ordering guarantees > > 2. Reduced unnecessary message blocking > > 3. Better scalability and performance > > 4. Enhanced observability > > > > This proposal would replace the existing "recently joined consumers" > > mechanism, addressing its limitations while providing a more robust > > solution. > > > > The full proposal can be found at: > > https://github.com/apache/pulsar/pull/23309 > > The direct link to the rendered version of the markdown file is: > > https://github.com/lhotari/pulsar/blob/lh-pip-379/pip/pip-379.md > > > > I welcome your feedback and discussion on this proposal. Please share > > your thoughts, concerns, or suggestions. > > > > -Lari > > >
