On Thu, Sep 2, 2021 at 9:51 AM Enrico Olivelli <eolive...@gmail.com> wrote:
> As Sijie said it is probably too early to bring it in, as the project is > still very new. > On the other hand, as far as it is a single person project, I am not sure > how many people will want to use it in production > and we want to bring Pulsar to a wider audience. > Reactive Streams is the future of streaming applications on the JVM. It won't be a single person project. There's already Pulsar community members that are looking forward to contributing. Here's one example: https://github.com/apache/pulsar/issues/10437#issuecomment-869928272 I have also another person that has contacted me and would like to start contributing. I have understood that it is a different process to accept an existing project with multiple contributors to Apache. Sijie and Enrico, please elaborate more about the concerns you have about the project being new and that I'm the author of it. I'd like to understand what problem it causes that the project is new. Isn't that a benefit? Let's discuss more about your concerns to gain more understanding. -Lari > > some answers to other comments inline below > > Enrico > > Il giorno gio 2 set 2021 alle ore 08:26 Rajan Dhabalia < > rdhaba...@apache.org> > ha scritto: > > > *My two cents about developing and contributing adapters/connectors for > > Apache Pulsar. Apache Pulsar was open-sourced with Spark and Storm > adapters > > initially and then such adapter families evolved with multiple other > > adapters, connectors and IO connectors (sink/source) into apache pulsar > git > > repo. Such adapters help pulsar for its adaption and user growth. > However, > > after adding more adapters in the pulsar repo, we realized two major > > issues: One of the issues, has made Pulsar git repo bulky which increases > > build time and sometimes it’s difficult to build and implement > integration > > tests. Another issue is the maintenance of such adapters with new > releases > > of the dependent systems and bug fixes for multiple versions. Therefore, > we > > eventually created a separate repository to manage adapters separately > into > > pulsar-adapters gitrepo. So, one of the takeaways from past experience, > if > > we absolutely need an adapter then we can use a pulsar-adapter gitrepo > and > > we should evaluate the feasibility of maintaining such adapters with > their > > different versions and their interest in the community. Thanks,Rajan* > > > > On Wed, Sep 1, 2021 at 10:59 PM Lari Hotari <l...@hotari.net> wrote: > > > > > Replies inline below: > > > > > > On Thu, Sep 2, 2021 at 3:37 AM Sijie Guo <guosi...@gmail.com> wrote: > > > > > > > Lari, > > > > > > > > Thank you for bringing this up! > > > > > > > > In general, I would love to see this being accepted to the project. > > > > > > > > > > Thanks! > > > > > > > > > > > > > > However, in the past, there were other language clients contributed > to > > > the > > > > project. They were not accepted because the PMC doesn't have enough > > > > bandwidth to staff it. > > > > I think that the community is still growing and also the PMC, if we have > enough people that are willing to support this > then I feel we can accept it. > In particular Lari is already part of the Community as committer and he > will take care of the project. > > > > > > > > > > > > > > > > https://github.com/apache/pulsar/wiki/PIP-77%3A-Contribute-Supernova-to-Apache-Pulsar > > > > is one of the examples. > > > > > > > > > > I don't think that Reactive Streams is comparable to this Supernova > > > example. That is an example of a very marginal language. > > > > > > Reactive Streams support is a mainstream technology that originates > from > > > Netflix. It's not about a single programming language. > > > The Reactive Streams is an interoperability specification and there are > > > multiple implementations for the JVM. > > > > > > For example, this Reaction Pulsar Adapter library uses Project Reactor > as > > > the implementation, but the library can be used together with Akka > > Streams, > > > RxJava 3, Vert.x, SmallRye Mutiny (RedHat/Quarkus) and others. > > > > > > This library is not about a single programming language, it's about all > > > languages and platforms on the JVM that support Reactive Streams. > > > It could be an extension directly to the Pulsar Java Client. I just > > thought > > > that maintaining the project would be easier when it's a separate > > > repository. > > > If that is causing the staffing issue that you referred to being the > > > showstopper issue, I could also restructure the project so that it > could > > be > > > added as modules to apache/pulsar repository. > > > > > > There's a short summary in > > > > > > > > > https://www.reactive.foundation/post/why-reactive-principles-ensure-system-scalability-with-josh-long > > > about the value proposition of Reactive Streams and why it matters. > > > The recent Netflix Techblog explicitly mentions Reactive Streams in the > > > blog post about Netflix's device management platform: > > > > > > > > > https://netflixtechblog.com/towards-a-reliable-device-management-platform-4f86230ca623 > > > "Back-Pressure Support - Because the processing workload varies > > > significantly over time, the solution must first and foremost scale > with > > > the message load by providing back-pressure support as defined in the > > > Reactive Streams specification — in other words, the solution should be > > > able to switch between push and pull-based back-pressure models > depending > > > on the downstream being able to cope with the message production rate > or > > > not." > > > > > > > > > > Also, the project is still relatively new and there is only one > > > > contributor. > > > > > > > > > > Everything has a beginning. :) > > > I was assuming that this would be optimal that there is only one > > > contributor and that other contributions could contaminate the IPR and > > make > > > it harder to move to the Apache Pulsar project later. > > > As mentioned above, this is more like an extension to the Pulsar Java > > > Client and I could restructure the project so that it could be added as > > > modules to the apache/pulsar repository if the separate repository > > > causes the barrier. > > > > > > > > > > Hence, my recommendation is to grow the contributors for this project > > so > > > > that there are enough contributors to work on that project. Once > there > > > are > > > > sufficient resources and contributors, the PMC can revisit this > > proposal > > > > and see if the PMC is able to staff it. > > > > > > > > > > Did the PMC already vote on this? :) I hope we could have the > discussion > > > before voting. > > > Adding support for Reactive Streams will add a lot of value to Apache > > > Pulsar. > > > > > > I'm talking about the value of Reactive Streams for Apache Pulsar > > > application developers TODAY AT 11:05PT in > > > my SpringOne talk "Reactive Applications with Apache Pulsar and Spring > > > Boot" > > > > > > > > > https://springone.io/2021/sessions/reactive-applications-with-apache-pulsar-and-spring-boot > > > It is free to attend, https://springone.io to register. There's also > an > > > AMA > > > session immediately after the session Q&A. > > > Please join. Come and see! > > > > > > I'm looking forward to hearing more voices from the Apache Pulsar > > community > > > in this email discussion. > > > > > > BR, Lari > > > > > > > > > > > > > > - Sijie > > > > > > > > > > > > On Wed, Sep 1, 2021 at 4:31 AM Lari Hotari <lhot...@apache.org> > wrote: > > > > > > > > > Dear Apache Pulsar community members, > > > > > > > > > > I have developed a Reactive Streams adapter for the Apache Pulsar > > Java > > > > > Client. It uses Project Reactor as the Reactive Streams > > implementation. > > > > > > > > > > The repository is here: https://github.com/lhotari/reactive-pulsar > > > > > and library is available in Maven Central: > > > > > > > > > > > > > > > https://search.maven.org/search?q=g:com.github.lhotari%20a:reactive-pulsar* > > > > > > > > > > I would like to contribute this library to the Apache Pulsar > project. > > > > > I would suggest that the library is committed to a separate > > repository, > > > > for > > > > > example https://github.com/apache/pulsar-reactive . > > > > > > > > > > # Motivation > > > > > > > > > > There's a need to "go reactive from end-to-end" when building > modern > > > > > reactive applications with platforms such as Spring Reactive. > > > > > > > > > > There are ways to adapt the Apache Pulsar Java client async API > calls > > > to > > > > > Reactive Streams with a few lines of code. However, a lot will be > > > missing > > > > > and achieving the complete solution will require much more effort. > > > > > > > > > > The primary goal of this library is to provide a Reactive Streams > > > adapter > > > > > that wraps the Apache Pulsar Java client with Reactive APIs > > > > > for sending, reading and consuming messages. > > > > > The secondary goal is to provide a Spring Boot starter for easily > > > > adopting > > > > > the power of Apache Pulsar for Spring Boot applications. > > > > > > > > > > It is currently out of scope for this library to build a reactive > > > client > > > > > all the way from the binary protocol handling. > > > > > This library builds upon the existing Apache Pulsar Java client and > > > uses > > > > > the async API for operations. > > > > > > > > > > # Features > > > > > > > > > > - Simple and intuitive reactive APIs for sending, reading and > > consuming > > > > > messages. > > > > > - Non-blocking backpressure support for the provided reactive APIs > > > > > - Pulsar client resource lifecycle management that integrates with > > the > > > > > Reactive programming model > > > > > - Additional producer caching that caches producers. > > > > > - Useful for API backends since the Pulsar client producer > resource > > > can > > > > > be shared and cached across multiple REST calls. > > > > > - Support for in-order parallel (concurrent) processing at the > > > > application > > > > > level > > > > > - I/O intensive message processing can leverage Project Reactor's > > > > > capabilities to efficiently > > > > > handle thousands of simultaneous I/O calls to external REST > APIs > > > and > > > > > resources that have Reactive clients/drivers. > > > > > - In-order parallel processing support can retain key-order at > the > > > > level > > > > > of a single consumer. > > > > > > > > > > I'm presenting the library and it's features tomorrow on September > > 2nd > > > in > > > > > the SpringOne virtual conference, which is free to attend. > > > > > The session details and schedule are at > > > > > > > > > > > > > > > > > > > > https://springone.io/2021/sessions/reactive-applications-with-apache-pulsar-and-spring-boot > > > > > . > > > > > > > > > > # Licenses > > > > > > > > > > The Reactive Pulsar Adapter library is under the Apache Licence > > Version > > > > 2.0 > > > > > . The main dependencies are reactor-core > > > > > (https://github.com/reactor/reactor-core) and jctools-core ( > > > > > https://github.com/JCTools/JCTools) which are also > > > > > ASL 2.0 licensed. > > > > > There's also a dependency to Caffeine ( > > > > > https://github.com/ben-manes/caffeine) > > > > > and the Spring Boot Starter > > > > > depends on Spring Boot and Spring Framework. These are all ASL 2.0 > > > > licensed > > > > > libraries. > > > > > The additional Spring integration testing support library depends > on > > > > > Testcontainers, which is also ASL 2.0 licensed. > > > > > > > > > > > > > > > I'm looking forward to the discussion about this. > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > Lari Hotari > > > > > > > > > > > > > > >
