I definitely agree that we can use guidance for any consideration or implementation of OSGi. I am definitely no OSGi expert, but at least for the past couple of years, I have spent a reasonable amount of time exploring the use of OSGi within/around the existing NiFi framework. Here are some observations (hopefully as apple-to-apple as possible), for what they're worth:
- Both NiFi and OSGi are modular application frameworks, defining a plugin component model realized by the NAR (NiFi Archive) format and the OSGi bundle format, respectively. - Both frameworks are able to find, load and run components at application startup. OSGi can additionally support dynamic loading and unloading at any point during runtime. - Both provide module independence via classloader isolation. - Both implement a service layer architecture, allowing modules to interact outside of their isolation container. OSGi has an extremely mature service registration, discovery and binding architecture, given this was one of its primary target areas. NiFi's service architecture approach is limited and immature in comparison; several opportunities for growth can be easily recognized. - The NAR format allows for a single parentage, effectively required (but limiting) for service discovery. OSGi does not define any restriction between modules and services, since any registered service can be discovered by any module (many-to-many). - The NAR format, being very similar to WAR, includes all dependencies inside of the archive. Multiple copies of the same base library (for example, any Apache Commons jar) are loaded into each isolated classloader space; NAR and WAR are very analogous in this way. In contrast, OSGi bundles do not traditionally include dependencies as part of the bundle archive. Instead with OSGi, only a single copy of any common library is maintained for the entire application framework. i.e. one and only one copy of any commons jar is ever loaded. Additionally, OSGi provides version conflict resolution, somewhat similar to maven conflict resolution. OSGi benefits from reduced permgen usage loading due to these features. - NiFi requires all NAR modules to be available locally at startup. OSGi containers generally support dynamic loading of modules and services from remote repositories (OBR or OSGi Bundle Repository). OSGi bundles can be easily added or updated from an OBR, including all transitive dependencies. Note that usage of an OBR for dependency resolution, however, can be an impediment in environments without network connectivity to such external repositories. Manual download and local install would still likely be required on any restricted network environment. (Nexus can act as an OBR, though.) - Both provide module lifecycle support. However, OSGi lifecycle support is geared more towards module or service loading/unloading; whereas, NAR lifecycle support in NiFi is geared to processor level lifecycle events. Limited lifecycle control is provided at the NAR level itself. - OSGi defines dependencies and service interfaces via bundle declarations in the MANIFEST.MF file. NAR modules are self-contained and require no META-INF information. NARs are arguably simpler to build, though OSGi related maven plugins can help. In summary, the NiFi framework has a lot of features that OSGi provides, particularly around application startup, module loading, classloader isolation, service discovery, etc. OSGi gets "right" many of the concepts that NiFi has grown into. While I don't believe any near-term release of NiFi could utilize OSGi, it is something that could likely be considered for a major (definitely non-backwards compatible) release in the future. It won't come without significant investment in time, as many parts in the "guts" of NiFi will be affected. Allowing OSGi compatible NARs (and the processors contained) to be loaded/unloaded at anytime during runtime needs to be thought through carefully. Obviously, NiFi should refrain from "reinventing the wheel" and "not invented here" in places where a well recognized platform is already available. Therefore it's possible that OSGi could be used for many of the problems which NiFi addresses today. Most of the NiFi concepts mentioned above can be implemented on top of OSGi. This would free NiFi to instead focus on its true value proposition, namely data flow management. Food for thought. Hope this was useful. Adam On Sun, Dec 14, 2014 at 6:08 PM, Joe Witt <[email protected]> wrote: > > I think ultimately with Nar or OSGi you can achieve the same ends (most > likely). Where the differences come in would be how easily a developer can > build, access, and reason over a bundle and the associated overhead/baggage > that comes with it at runtime. > > Nars (the nifi kind) are purpose built for NiFi and so the overhead and > class loading model is very well understood and battle tested. But the > truth is our plugin reflects our novice (copy and paste) level of knowledge > of the maven internals specifically as it relates to building plugins. > > Nars did, and perhaps at the moment still do, have the right tradeoff. But > OSGi by design should be the right choice and so we need to gain > familiarity or better yet do a good job of solving our other priorities and > attracting folks with expertise here who can help us get it right (though > Benson's notes are encouraging too). > > I'm glad to hear folks aren't balking at the notion of deploying nar > bundles to maven repos. That means we can prioritize other issues higher > (like the docs and having a build process that is more common and > approachable). > > Thanks > Joe > > On Sun, Dec 14, 2014 at 7:34 PM, Mark Payne <[email protected]> wrote: > > > > I have very little experience with OSGi (I toyed with it at home when we > > discussed the introduction of NARs several years ago but that's about > it). > > But I have quite a lot of knowledge about NARs, so I thought I'd point > out > > the key features/design points. OSGi may or may not cover all of these > > concepts, but if we are thinking about OSGi as a possible alternative > (or a > > parallel), I think it's good to go ahead and call out these concepts. > > NARs map directly to a Java ClassLoader. Whatever your NAR depends on in > > its pom is what will be loaded into the Java ClassLoader. This makes > > understanding ClassLoading issues much easier, I think. > > A NAR can declare a "Parent NAR." That means that the parent of its > > ClassLoader will be the ClassLoader of the parent NAR. > > NARs can also provide a 'native' folder that houses native libraries. > This > > provides native library isolation as well as class loading isolation. > > You are able to define a Controller Service interface in a 'service-api' > > nar and then have that NAR be the parent of say my-processor-nar and > > my-service-nar. The Processor can then call methods in the Controller > > Service via the interface. Even though the implementation lives in > > my-service-nar, we can cross that NAR boundary safely since we only > > interact with the interface (which is defined in the common parent). > > > > > Date: Sun, 14 Dec 2014 18:44:05 -0500 > > > Subject: Re: nar plugin > > > From: [email protected] > > > To: [email protected] > > > > > > I wouldn't reject Maven Central off-hand. > > > > > > And I wouldn't reject keeping your own maven plugin and packaging, even > > if > > > it's a nearly complete fork of the maven-war-plugin. > > > > > > I think that Maven's handling of 'packaging' and such is less than > delux, > > > and leads in this direction. You'd like to be able to take advantage of > > the > > > underlying function of the war plugin (or the assembly plugin) and > still > > > get DRY by avoiding having your devs need to copy a complex assembly > > > descriptor or do something funny-looking with the war plugin, but I > don't > > > know a way. So if you want a nice, clean, pom for 'build me a NiFi > > plugin', > > > you end up, I suspect, with .. a maven plugin. > > > > > > I've recently stuck my toes into the OSGi waters at work; I have a > theory > > > of how to have an OSGi system while still cleaving closely to the Maven > > > dependency universe so as to avoid ending up adrift in OBR. The theory > is > > > neither completely worked out nor proven in production, but I could > write > > > it up such as it is. I'm still digesting some ideas from the Felix user > > > list. > > > > > > Basically, I think of OSGi as a generalization of 'war', and then I am > > > trying to work out a low-labor way to keep track of all the > dependencies. > > > > > > > > > On Sun, Dec 14, 2014 at 6:30 PM, Tony Kurc <[email protected]> wrote: > > > > > > > there seem to be plenty of wars which include dependencies in > > > > repository.apache.org > > > > > > > > On Sun, Dec 14, 2014 at 5:59 PM, Sean Busbey <[email protected]> > > wrote: > > > > > > > > > > On Dec 14, 2014 4:25 PM, "Joe Witt" <[email protected]> wrote: > > > > > > > > > > > > OSGi was the most obvious choice at the time but we stayed away > > due to > > > > > the > > > > > > tooling/cost/benefit. > > > > > > > > > > > > > > > > Might be worth revisiting. I know the specter of OSGi is raised > every > > > > time > > > > > classpath woes surface in Hadoop-land. > > > > > > > > > > > Our simple goal was for those building extensions to easily > produce > > > > > bundles > > > > > > of their code and associated dependencies. > > > > > > > > > > > > Building uber jars was generally considered non-desirable because > > we > > > > did > > > > > > not want to change the structure of the extension or dependent > > jars. > > > > > > > > > > > > > > > > +1 for continuing to avoid uber jars. > > > > > > > > > > > The assembly plugin might in fact be totally sufficient for our > > > > purposes. > > > > > > It requires some extra steps for the client but those are > probably > > > > > > completely automated by maven archetypes. > > > > > > > > > > > > > > > > Reworking things so that nar becomes a kind of assembly is an > > interesting > > > > > idea. Do we have docs on the expectations the runtime has for a > nar? > > > > > > > > > > > In any event we recognize the current nar concept is problematic > > > > because > > > > > > nobody is going to want us publishing nar artifact bundles to > maven > > > > > repos. > > > > > > So we're looking at what we should do. > > > > > > > > > > > > > > > > I don't see why publishing nars would be a problem. Other projects > > > > publish > > > > > tarballs (source and binary) and things go well. > > > > > > > > > > > > > >
