I know we have folks spread across different time zones, would a 9AM PST / 4PM UTC discussion work? Yes, we will post the recording afterwards but that makes it hard to ask live questions :-)
Anthony > On Sep 21, 2021, at 12:14 AM, Alberto Gomez <alberto.go...@est.tech> wrote: > > I think this is a great initiative. Not only are you giving a reminder about > the need to implement new features in a modular and pluggable way, but you > are also providing a real example with an already implemented not in the best > way feature, to show the steps to follow in the right direction for this and > future features. > > I would be interested in attending to a live walkthrough over the details of > the changes. > > Thanks, > > Alberto > ________________________________ > From: Jacob Barrett <jabarr...@vmware.com> > Sent: Tuesday, September 21, 2021 3:04 AM > To: dev@geode.apache.org <dev@geode.apache.org> > Subject: Re: [DISCUSS] Modularizing new WAN TX Batching (and Modularization > Efforts in General) > > Unfortunately I can’t do meetings that early. The earliest I could do that > day is 9am PDT. > >> On Sep 20, 2021, at 3:34 PM, Anthony Baker <bak...@vmware.com> wrote: >> >> The approach makes sense to me. The idea of a stable core + extensibility >> is a common and successful pattern in many OSS projects. >> >> @Jake, you want to discuss at the next Geode Community meeting in Oct? >> >> Anthony >> >> >>> On Sep 20, 2021, at 1:48 PM, Jacob Barrett <jabarr...@vmware.com> wrote: >>> >>> Devs, >>> >>> We need to be doing a better job with implementing new features in a >>> modular and plugable way. We have had discussions in the past but we >>> haven’t been the best at sticking to it. Most recently we began work on a >>> modified version of WAN that supports transactional batching. Rather than >>> implement it in a plugable model we modified the existing implementation >>> with a new property that changes the internal behavior of the >>> implementation. This is a clear smell that what we have should be plugable >>> and modularized. I am not suggesting that we run out and define clear >>> public SPIs for everything or come up with some complicated plan to >>> re-architect everything into modules but rather that we take small steps. I >>> will argue that when we are adding functionality to a core service that is >>> the point we should consider steps to break it up into clear module >>> components. Think to yourself, what would it take for me to implement this >>> new functionality as its own module, meaning its own jar and Gradle >>> sub-project. As you begin to develop the solution you may find you need >>> some clean interfaces for it to extend from the core, that might be the >>> start of an internal SPI. You may find that some concrete classes you would >>> have normally modified just need to be extended with a few protected >>> methods to implement alternative logic. >>> >>> I think we should focus efforts on extracting an interface to plug in >>> different WAN gateway implementations so that existing implementations >>> aren’t modified with new behavior. With proper interface extraction we can >>> more easily unit test around WAN gateways. By keeping implementations small >>> we can more easily test them in isolation. Making them all plugable allows >>> distributions of Geode to deliver specific implementations they would like >>> to support without impacting the existing implementations. It also frees >>> Geode to release new experimental or beta implementations of WAN gateways >>> without impacting the existing implementations rather than delaying >>> releases waiting for modified WAN gateways to be production ready and fully >>> tested. >>> >>> In looking at the geode-wan module one might notice that it was already >>> designed to be plugable. Unfortunately it isn’t that easy. This SPI was >>> originally there to provide a way for Geode to be donated initially without >>> WAN support. It turns out that most of the core to WAN is actually still in >>> geode-core and only some of the “secret sauce” was moved into geode-wan. >>> The bulk of the geode-core code for WAN is actually in support of the >>> region queues for WAN and AEQ, so moving it into geod-wan would have cut >>> off AEQ. While it would be possible to utilize this SPI for providing >>> alternative gateway implementations it is very high level,so much of the >>> alternative implementations would be duplications, and it doesn’t allow for >>> both implementations to sit side by side at runtime. I would actually >>> suggest we eliminate this public SPI in favor of just the geode-wan core >>> module that it is and eventually migrate the region queue code into its own >>> module as well, but these are for another day. >>> >>> Looking closer at the WAN gateways themselves there is mostly a pluggable >>> interface already there with the existing interfaces. I spent a little time >>> pulling apart an internal SPI and it was quite easy. With a small >>> modification to gfsh and cache xml to specify that alternative >>> implementation by name is all that needs to be done immediately to >>> configure an alternative. Without extracting too many of the common >>> implementation out into its own module just a few of the classes in >>> geode-core can be modified to provide empty implementation of key >>> overridable plug-in points for the TX batching implementation. The result >>> is that the TX batching module can be added to the classpath and be >>> configured as a gateway sender without injecting any of the TX batching >>> specific code into geode-core or geode-wan. Deeper details can be found at >>> the end of this discussion. >>> >>> This solution isn’t perfect but it is a step in the right direction. The >>> more we continue to extract and extend the closer we will get to a true >>> pluggable architecture. The key to being successful will be to keep the >>> changes to the core components small and the interfaces between modules >>> internal. If we all make an effort towards this, both in our code and in >>> our reviews of others code, we can keep the efforts small and manageable. >>> >>> I would love feedback and thoughts on this suggestion. >>> >>> -Jake >>> >>
