> If you want to work on cloud-events > integration, please do. I still don't see how's that related to Pulsar > 3.0 discussion.
Would a change to SchemaInfo break the client? Devin G. Bost On Tue, Oct 11, 2022 at 11:58 AM Matteo Merli <matteo.me...@gmail.com> wrote: > > Companies want to be cloud agnostic since it reduces their risk, and > the idea of multi-cloud integration is very attractive to companies that > want the best of both worlds or need to integrate data across platforms. If > Pulsar is not part of this wave, I'm concerned we will be left behind. > > That sounds a lot of marketish. If you want to work on cloud-events > integration, please do. I still don't see how's that related to Pulsar > 3.0 discussion. > > -- > Matteo Merli > <matteo.me...@gmail.com> > > On Tue, Oct 11, 2022 at 9:54 AM Devin Bost <devin.b...@gmail.com> wrote: > > > > How about the gap > > <https://lists.apache.org/thread/jfwv1wkbk4nsd9lhpnh5o2r0pgq39qz8> > between > > SchemaInfo and interoperability with CloudEvents? We're seeing a > > convergence between cloud providers on CloudEvents. I just learned that > SAP > > Event Mesh now is working on integration with Azure EventGrid via > > CloudEvents. GCP is already there. AWS has an adapter from EventBridge. > > There is work taking place on solutions for discovery, like a search > engine > > for event sources. Kafka has a binding for CloudEvents. Pretty soon, I > > suspect we're going to see increasing interest in the integration of > events > > across organizational and technological boundaries, and technologies will > > emerge with direct support for events that conform to the CloudEvents > > spec. Companies want to be cloud agnostic since it reduces their risk, > and > > the idea of multi-cloud integration is very attractive to companies that > > want the best of both worlds or need to integrate data across platforms. > If > > Pulsar is not part of this wave, I'm concerned we will be left behind. > > > > Devin G. Bost > > > > > > On Mon, Oct 10, 2022 at 5:06 PM Joe F <joefranc...@gmail.com> wrote: > > > > > I would prefer that we avoid using the term “breaking changes”, which > is > > > too vague to convey any specific meaning. So let me try to bring some > > > clarity. > > > > > > > > > There have been many changes to implementations, APIs and data storage > > > formats in Pulsar (and book keeper also). I have deployed many of these > > > changes to production. And I know that Matteo and Rajan (and others > too, > > > about whom I’m not up to date on) have implemented and deployed many > such > > > changes. But none of those changes ever required taking the system > > > offline. NONE. > > > > > > > > > Pulsar was developed as a 24x7x365 system, and rolling upgrades and > > > rollbacks were a given. Like “this is water”, there was no special > callout > > > needed for declaring this reality. No change, including enhancements to > > > wire protocols, broke client compatibility. Existing clients > continued to > > > work; they may not be able to use all the new features. Use of new > features > > > would require the app to be rebuilt anyway. (Checksums, e2e > encryption are > > > examples) > > > > > > > > > We have even succeeded in getting Pulsar adopted for some use cases, > just > > > because the complexity of upgrading from K’s old clients to new ones > were > > > costly enough to allow consideration of an alternative like Pulsar. > The > > > business cost of forcing a client upgrade can be significant, to the > point > > > of this being unviable for business. That just cannot be hand-waved > over > > > > > > > > > There have also been changes in storage formats(the ZK metadata change > from > > > text to binary is an example). But through all such changes, > compatibility > > > and upgradeability has been a given. There has never been a situation > where > > > a live Pulsar upgrade was not possible, and a coordinated client > upgrade > > > was mandatory. > > > > > > > > > So the question should not be about whether “signifcant” changes > should > > > be made or not. Changes can be made and released in a way that breaks > > > *business*, or they can be made in a way that lets businesses sail > > > smoothly through that change. So the question is about how such > changes > > > gets rolled out. > > > > > > > > > And to that question, my strong opinion is that any change that does > not > > > allow a live/rolling upgrade or rollback, or anything that forces a > client > > > to upgrade just to continue functioning, is a non-starter. All > changes > > > can be made in a compatible, phased manner, and in a way that does not > > > penalise older versions ( older versions doing worse on new releases > is > > > also not an acceptable way of making changes) Changes can be made in a > > > manner that make l A/B testing possible by the user, with limited > risk, and > > > then choosing to a not go back. It has all been done in Pulsar before. > > > > > > > > > Would that be harder than just breaking stuff? Yes. But that is far > more > > > preferable than forcing users to take a hit. > > > > > > > > > -joe > > > > > > On Sat, Oct 8, 2022 at 1:25 PM Rajan Dhabalia <rdhaba...@apache.org> > > > wrote: > > > > > > > I would say first we should gather a list of changes which we want to > > > > target and find out which improvements really need major version > release. > > > > We can take the Pulsar-1.0 to Pulsar-2.0 upgrade example to avoid > major > > > > interruption and impact on existing systems and still achieve our > goal. > > > So, > > > > the first step is discovery of such features and then we can discuss > how > > > to > > > > introduce them in Pulsar with minimum impact on existing systems. > > > > > > > > Thanks, > > > > Rajan > > > > > > > > On Sat, Oct 8, 2022 at 1:05 PM Devin Bost <devin.b...@gmail.com> > wrote: > > > > > > > > > I'm noticing some pushback on the idea of pre-emptively proposing > any > > > > kind > > > > > of breaking upgrade that would necessitate cutting a 3.0 release. > > > > > I do understand the concern about introducing a breaking change... > For > > > a > > > > > distributed messaging application like Pulsar, if clients needed > to be > > > > > simultaneously upgraded with brokers, that could be extremely > difficult > > > > or > > > > > infeasible for companies to coordinate without treating it like a > > > > migration > > > > > to a new technology. > > > > > > > > > > At the same time, do we want to be completely closed to the > possibility > > > > > that a breaking change could be required at some point in the > future? > > > If > > > > a > > > > > circumstance like that appears, those are the kinds of situations > that > > > > can > > > > > lead to a fork. Are there certain kinds of breaking changes that > are > > > more > > > > > acceptable than others? > > > > > > > > > > Also, if the forward looking plan is to never introduce breaking > > > changes, > > > > > when *would* we ever cut a Pulsar 3.x release? Do we have any > criteria > > > > on > > > > > what kinds of changes would necessitate cutting a new major > release but > > > > > would still be considered acceptable by the community? > > > > > > > > > > -- > > > > > Devin Bost > > > > > Sent from mobile > > > > > Cell: 801-400-4602 > > > > > > > > > > On Sat, Oct 8, 2022, 2:14 PM Rajan Dhabalia <rdhaba...@apache.org> > > > > wrote: > > > > > > > > > > > This sounds like the current state of Apache Pulsar has a lot of > > > issues > > > > > and > > > > > > it requires fundamental design changes to make it promising > which is > > > > > > definitely not true and I disagree with it. And I would be > careful > > > > > > comparing with Kafka as I still don't think the Kafka release has > > > > > anything > > > > > > to do with Pulsar's improvement. I would still recommend to list > down > > > > all > > > > > > the changes at one place so we can bring everyone on the same > page. > > > > > discuss > > > > > > as a community and we make sure existing usecases continue using > > > Pulsar > > > > > and > > > > > > not try to find Pulsar alternatives with incorrect disruption > > > > impression > > > > > > and efforts they might have to put to upgrade or maintain pulsar. > > > > > > > > > > > > Thanks, > > > > > > Rajan > > > > > > > > > > > > On Fri, Oct 7, 2022 at 7:49 PM Lari Hotari <lhot...@apache.org> > > > wrote: > > > > > > > > > > > > > We could all have our own favorite names for this work. :) > > > > > > > > > > > > > > There's advice that you should disrupt yourself before someone > > > > disrupts > > > > > > > you. > > > > > > > Shouldn't we follow that advice for Apache Pulsar? We can > disrupt > > > > > Pulsar > > > > > > > together with our Apache hats on. The catch is that since we > are > > > > doing > > > > > > > this, we will be able to learn and improve Pulsar so that we > stay > > > > ahead > > > > > > of > > > > > > > competition. Pulsar was long ways ahead of competition for so > many > > > > > years, > > > > > > > but Kafka is finally catching up. Did Kafka surpass Pulsar in > some > > > > > > aspects > > > > > > > with the recent 3.3 release, where Kraft became GA? That's a > > > question > > > > > > that > > > > > > > many might be asking. Why wouldn't we rev up Pulsar's engine > and > > > show > > > > > the > > > > > > > tail lights to Kafka? > > > > > > > > > > > > > > We don't have to have deadlines or any restrictions like that > right > > > > > now. > > > > > > > The sky's the limit. > > > > > > > Linus Torvalds has written a book called "Just for fun". I got > my > > > > copy > > > > > of > > > > > > > this book signed by Linus himself in year 2000 at an event > that the > > > > > book > > > > > > > publisher had organized in Finland. > > > > > > > > > > > > > > What if we did this "just for fun"? The intention could also > be to > > > > beat > > > > > > > Kafka, but that could be a boring goal for many. What if we > could > > > > > unleash > > > > > > > some talent that is among us and hasn't had a chance to show > its > > > full > > > > > > > potential? Opensource is about joy. It is about welcoming > everyone > > > to > > > > > > join. > > > > > > > Opensource should be egoless, although we must all admit that > we > > > > don't > > > > > > > succeed in that aspect. We must fight our biases. > > > > > > > > > > > > > > Jarek Potiuk explains the importance of being welcoming for > success > > > > at > > > > > > > Apache, in a 3-minute YouTube interview: > > > > > > > https://www.youtube.com/watch?v=Dx5kQnVFo7E > > > > > > > This interview is about Jarek's blog post "Success at Apache: > > > > Welcoming > > > > > > > communities strengthens the Apache way": > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://news.apache.org/foundation/entry/success-at-apache-welcoming-communities > > > > > > > I was pleased to meet Jarek at ApacheCon among so many other > > > > welcoming > > > > > > > personalities of the Apache community and the Apache Pulsar > > > > community. > > > > > > > > > > > > > > Goals have to be ambitious. What if we set the bar really high? > > > > > > > Apache Pulsar with 10 million topics in a cluster? > > > > > > > Why not go up to 100 million topics? > > > > > > > Just for fun. :) > > > > > > > > > > > > > > -Lari > > > > > > > > > > > > > > On 2022/10/07 22:53:59 Matteo Merli wrote: > > > > > > > > I actually disagree with the term "Pulsar Next Gen", because > I > > > > > haven't > > > > > > > > seen any proposal for which that would make sense to me to be > > > > called > > > > > > > > so. > > > > > > > > > > > > > > > > Rajan: That's the whole point of breaking it down. If you > > > > accumulate > > > > > > > > many "big" changes it introduces a lot of risk for > instabilities > > > > and > > > > > > > > incompatibilities. Breaking it down in multiple steps helps > to > > > see > > > > > the > > > > > > > > incremental changes and introduced them in a phased manner. > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Matteo Merli > > > > > > > > <matteo.me...@gmail.com> > > > > > > > > > > > > > > > > On Fri, Oct 7, 2022 at 3:37 PM Rajan Dhabalia < > > > > rdhaba...@apache.org> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > Can we get the list of changes at one place which we are > > > planning > > > > > to > > > > > > > get as > > > > > > > > > part of 3.0. One thing I would like to see as a part of a > major > > > > > > > release, it > > > > > > > > > CAN NOT impact existing usecases and users in any way > which can > > > > > force > > > > > > > them > > > > > > > > > to upgrade the client library. Applications using < 3.0 > version > > > > > > should > > > > > > > > > continue getting all the client and server side > enhancements > > > and > > > > > bug > > > > > > > fixes. > > > > > > > > > Failing to provide bug-fixes and features to client < 3.0 > means > > > > we > > > > > > are > > > > > > > > > forcing them to upgrade client version by putting efforts > to > > > > handle > > > > > > all > > > > > > > > > incompatibility. and that's something we should definitely > > > > prevent > > > > > > > because > > > > > > > > > Apache Pulsar is used by many large scale business > usecases and > > > > we > > > > > > > should > > > > > > > > > accommodate and motivate them to continue using Apache > Pulsar. > > > > > > > > > I understand as a Pulsar community we should always try to > > > > progress > > > > > > and > > > > > > > > > build better but not at the cost of losing or reducing the > > > Apache > > > > > > > Pulsar > > > > > > > > > community. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Rajan > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Oct 7, 2022 at 12:41 PM Lari Hotari < > > > lhot...@apache.org> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Thank you, Matteo. I agree that features should be > delivered > > > > > > > continuously > > > > > > > > > > when that is possible. In this case, that might not > apply. > > > > > > > > > > > > > > > > > > > > I also agree that calling this Pulsar 3.0 isn't > necessarily > > > > > aligned > > > > > > > with > > > > > > > > > > PIP-175 since an LTS release is when the major version is > > > > bumped. > > > > > > > I'm fine > > > > > > > > > > in calling this "Pulsar Next Gen" or something that > calls out > > > > > that > > > > > > > this is > > > > > > > > > > planning for making a major leap in Pulsar. > > > > > > > > > > > > > > > > > > > > There are several unresolved issues with PIP-45 and the > > > Pulsar > > > > > Load > > > > > > > > > > balancer. The previously referred email threads contain > a lot > > > > of > > > > > > > context to > > > > > > > > > > this. Resolving the issues efficiently will most likely > > > result > > > > in > > > > > > > breaking > > > > > > > > > > changes, which will be the reason why it deserves a major > > > > version > > > > > > > upgrade. > > > > > > > > > > > > > > > > > > > > We have discussed it before that it's crucial to have a > path > > > to > > > > > > > migrate > > > > > > > > > > users when there are breaking changes. This should be > covered > > > > in > > > > > > any > > > > > > > of the > > > > > > > > > > solutions that are introduced. Optimally, users of Pulsar > > > would > > > > > be > > > > > > > able to > > > > > > > > > > upgrade seamlessly to Pulsar Next Gen / Pulsar 3.0, but > > > rolling > > > > > > back > > > > > > > might > > > > > > > > > > not be directly supported. > > > > > > > > > > > > > > > > > > > > I am welcoming everyone to join this planning for the > Apache > > > > > Pulsar > > > > > > > Next > > > > > > > > > > Gen architecture. Please check the first email in this > thread > > > > for > > > > > > > details > > > > > > > > > > of context, and start participating and contributing > today. > > > The > > > > > > best > > > > > > > way to > > > > > > > > > > contribute is to participate in the email threads, since > they > > > > > > contain > > > > > > > > > > details with better context. > > > > > > > > > > > > > > > > > > > > -Lari > > > > > > > > > > > > > > > > > > > > On 2022/10/07 18:03:00 Matteo Merli wrote: > > > > > > > > > > > Given the past experiences and the discussions that > already > > > > > > > happened > > > > > > > > > > > around "PIP-175: Extend time based release process", > the > > > idea > > > > > is > > > > > > to > > > > > > > > > > > detach the 3.0 from "big-features" items or > "incompatible > > > > > > changes". > > > > > > > > > > > > > > > > > > > > > > The changes are going to get included as they are > ready, > > > > within > > > > > > > > > > > feature releases, and in a fully compatible way. We > don't > > > > need > > > > > to > > > > > > > > > > > group them together and create unnecessary risk for the > > > > release > > > > > > > > > > > schedule and the users. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > Matteo Merli > > > > > > > > > > > <matteo.me...@gmail.com> > > > > > > > > > > > > > > > > > > > > > > On Fri, Oct 7, 2022 at 10:47 AM Lari Hotari < > > > > > lhot...@apache.org> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > > > Greeting from ApacheCon North America 2022 from New > > > > Orleans! > > > > > > > > > > > > We had a great conference with a dedicated Pulsar > track. > > > > > Thanks > > > > > > > to all > > > > > > > > > > presenters and everyone who attended. The talks weren't > > > > recorded, > > > > > > > but the > > > > > > > > > > slides will be later on posted on the conference website > [1]. > > > > > > > > > > > > > > > > > > > > > > > > At ApacheCon there were several presentations about > "the > > > > > Apache > > > > > > > way" > > > > > > > > > > and what that means in practice. Based on that, we all > know > > > > that > > > > > no > > > > > > > person > > > > > > > > > > is nominated as the CTO of Apache Pulsar who decides on > > > Pulsar > > > > > 3.0 > > > > > > > and when > > > > > > > > > > that happens. It's us, the community, that serve that > role > > > > > > together. > > > > > > > We > > > > > > > > > > come together as individuals with the Apache hat on. > Everyone > > > > is > > > > > > > equal in > > > > > > > > > > the community, regardless of whether they are > contributors, > > > > > > > committers or > > > > > > > > > > PMC members. > > > > > > > > > > > > We welcome everyone to participate. The small detail > > > about > > > > > > voting > > > > > > > > > > shouldn't stop anyone from participating in any aspects > of > > > the > > > > > > > planning for > > > > > > > > > > the roadmap. > > > > > > > > > > > > > > > > > > > > > > > > I'll like to get the discussions going for Pulsar > 3.0. We > > > > > don't > > > > > > > need a > > > > > > > > > > separate decision to start planning that. Please correct > me > > > if > > > > > I'm > > > > > > > wrong or > > > > > > > > > > if you have a different opinion. > > > > > > > > > > > > > > > > > > > > > > > > There are a few previous discussion threads that are > > > > related > > > > > to > > > > > > > Pulsar > > > > > > > > > > 3.0 planning. > > > > > > > > > > > > If you are interested in getting involved with Apache > > > > Pulsar > > > > > > 3.0 > > > > > > > > > > planning, I think that it makes sense for you to read > these > > > > > threads > > > > > > > > > > carefully and reply to them. Please also suggest what you > > > think > > > > > > > makes sense. > > > > > > > > > > > > > > > > > > > > > > > > PIP-45 related: > > > > > > > > > > > > > > https://lists.apache.org/thread/tvco1orf0hsyt59pjtfbwoq0vf6hfrcj > > > > > > > > > > > > Pulsar Load balancer / namespace bundle related: > > > > > > > > > > > > > > > > > > https://lists.apache.org/thread/roohoc9h2gthvmd7t81do4hfjs2gphpk > > > > > > > > > > > > renaming topics: > > > > > > > > > > > > > > > > > > https://lists.apache.org/thread/vrr75rrh4trqlp14objh3snlfvmzdrp2 > > > > > > > > > > > > backpressure: > > > > > > > > > > > > > > https://lists.apache.org/thread/v7xy57qfzbhopoqbm75s6ng8xlhbr2q6 > > > > > > > > > > > > > > > > > > > > > > > > Long list of Metadata inconsistency issues by Zac > > > Bentley: > > > > > > > > > > > > https://github.com/apache/pulsar/issues/12555 > > > > > > > > > > > > That would be a good starting point to understanding > the > > > > data > > > > > > > > > > inconsistency issues related to current PIP-45 design. > > > Perhaps > > > > > > those > > > > > > > could > > > > > > > > > > be addressed already before Pulsar 3.0? > > > > > > > > > > > > > > > > > > > > > > > > I'm looking forward to everyone's participation in > the > > > > Apache > > > > > > > Pulsar > > > > > > > > > > 3.0 planning discussions. > > > > > > > > > > > > > > > > > > > > > > > > Best Regards, > > > > > > > > > > > > > > > > > > > > > > > > -Lari > > > > > > > > > > > > > > > > > > > > > > > > 1 - https://www.apachecon.com/acna2022/schedule.html > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
