Hi Cong, I'd like to add this PR to be reviewed and merged: https://github.com/apache/pulsar/pull/24488 (PIP-420: Supports users implement external schemas)
Thanks, Yunze On Fri, Aug 22, 2025 at 7:03 PM Lari Hotari <lhot...@apache.org> wrote: > > Hi all, > > Thank you, Cong, for driving the 4.1.0 release process. > > I'd like to propose considering the inclusion of PIP-430: Pulsar Broker Cache > Improvements before cutting the branch-4.1. Here's my reasoning: > > Why PIP-430 would be valuable for 4.1: > - Significant performance improvements: PIP-430 addresses fundamental > inefficiencies in the current broker cache eviction mechanisms that impact > CPU utilization, memory pressure, and cache hit rates > - Better resource utilization: The current size-based eviction doesn't > guarantee removal of the oldest entries globally, and timestamp-based > eviction has high CPU overhead when iterating through all managed ledgers. > This overhead is significant when the broker is serving a very large number > of active topics. > - Enhanced support for diverse workloads: The new "expected read count" > strategy better handles mixed read patterns (tailing reads, catch-up reads, > Key_Shared subscriptions) that are common in production > > Technical impact: > - Reduces BookKeeper and tiered storage load through improved cache hit rates > - Addresses performance issues with brokers handling large numbers of topics > - Provides a foundation for future cache optimizations > > The PIP-430 implementation is very close to the finish line. The PR > https://github.com/apache/pulsar/pull/24623 is waiting for reviews and after > it has been merged, it would be ready for release. The first implementation > PR of PIP-430 is already included in the master branch. In PIP-430, there are > configuration options for choosing previous caching strategies if someone is > concerned about regressions. However, extensive testing shows that the > PIP-430 caching strategy performs better than other strategies in all use > cases. > > Another PR that would need to be reviewed and merged before cutting > branch-4.1 is the implementation PR for "PIP-428: Change TopicPoliciesService > interface to fix consistency issues" at > https://github.com/apache/pulsar/pull/24427. This addresses severe > consistency issues in topic policy updates. > > There's yet another reason to wait for PIP-430 to be included: to have a > valuable feature for Pulsar 4.1. There aren't very many actual changes > between the master branch and branch-4.0 at the moment since most PIPs have > also been backported to Pulsar 4.0.x. Many of the PIPs are minor improvements > to existing features. Having a good reason for upgrading to Pulsar 4.1 would > be useful. PIP-430 is such a feature. > > In summary: before we can proceed to cut branch-4.1, these PRs would need to > be reviewed and merged: > * https://github.com/apache/pulsar/pull/24623 (PIP-430 implementation, part 2) > * https://github.com/apache/pulsar/pull/24427 (PIP-428 implementation) > > I'm looking forward to your reviews! > > -Lari > > On 2025/08/20 14:47:44 Cong Zhao wrote: > > Hi community, > > > > It's been a long time since the 4.0 version was released. > > > > - 482 commits: > > > > https://github.com/apache/pulsar/pulls?q=is%3Apr+milestone%3A4.1.0+is%3Amerged+ > > > > - 17 PIPs approved: > > > > https://github.com/apache/pulsar/pulls?q=is%3Apr+milestone%3A4.1.0+is%3Amerged++label%3APIP+ > > > > We'd better prepare for the 4.1.0 release. I would like to cut branch-4.1 > > in the next week. Please leave any ideas or concerns. > > > > Regards, > > Cong Zhao > >
