Sounds like a great proposal. Im excited to see what you come up with. james
14.03.2017, 19:56, "Otto Fowler" <ottobackwa...@gmail.com>: > RE: questions - > > - The archetype will produce NARs so having it with the VFSClassloader > doesn’t work, unless > I do an archetype without nars, but that would be so short lived I wouldn’t > bother. > - My goal is to do it in such a way that I could propose / submit the > changes made back to nifi ( worse case is work alike ). > It is not out of the questions that between metron-nar ( or whatever I call > it ) and nifi-nar, we come up with > a 3rd thing that both switch to, that is generic and cross project. To > that end, in the work I’m doing I am keeping dependencies > to metron or nifi to zero. > > I am still doing some poc work. But the nar work ( that I will be putting > into jira ) will be: > > * nar-utils ( the class loaders and expanders ) to a top level but not > dependent metron library > * refactor nar -utils to be independent of the nifi-properties and more > generically configurable > * replace the nifi file utils and file / uri class loaders with VFS > FileSystemManager based implementation. ( this will work for them too, and > I will be posting it back to them ) > * fork the nifi maven plugin and make it configurable as to output name > extension and metadata names ( XXX-ID instead of Nar-ID ) - this too I > will propose as a PR back > > I feel the right thing to do, when landing the parsers as separate, is not > what I have done so far. Each parser should be a product of the archetype > themselves. > That is how I intend to proceed. > > On March 14, 2017 at 17:42:28, Kyle Richardson (kylerichards...@gmail.com) > wrote: > > I like the direction of using NAR. The key benefits I see (there are > others, but I picked by favorites): > > 1. Move away from the uber jars in Storm. Granted, this should be possible > with modifications from the new Stellar classloader. My hope is that this > would allow us to stop shading the parser project packages. > 2. Allow parser development to be somewhat independent of the core code > base. I do think we'll have to address versioning and backwards > compatibility. > 3. "Drop in" extensions to Metron. Future expansion beyond parsers. > > Questions: > - Is there a way to do 1 using the VFS Classloader and land the archetype > as an MVP? If so, we could avoid shading as part of the archetype and maybe > iterate from there on it. Do you think this would still be too much change > to the archetype after putting it out there? > - How would we adapt NAR from Nifi without frankensteining it? I'm all for > code reuse but, ideally, we'd like Nifi and Metron's versions of NAR to end > up converging in the end. > > On Fri, Mar 10, 2017 at 3:43 PM, Casey Stella <ceste...@gmail.com> wrote: > >> I would definitely agree that moving forward we should consider something >> like Nar for Stellar. I'm not seeing the need for parsers exactly. >> >> I don't want to squash the forward thinking aspect here; we should be > > broad >> and think about the end, ideal state. I just want to make sure we think >> through something that we can iterate on as an initial state that still >> solves your problem, MVP style. >> >> On Fri, Mar 10, 2017 at 3:39 PM, Otto Fowler <ottobackwa...@gmail.com> >> wrote: >> >> > "The Apache NiFi NAR ‘system’ allows for the packaging and loading of >> java >> > resources with classloader isolation. >> > Although technically it is the Service Provider api that makes the >> > ‘plugins’ part of the system, you can view them >> > together, and thus look at the NAR features as a system to create, >> > package, load, and execute plugins in a java system >> > while maintaining classloader isolation and dependency separation. >> > >> > While the NiFi problem case ( many plugins possibly executing in the > > same >> > vm ) is not universal, the functionality provided >> > by NAR is commonly needed, and is indeed functionality that I am >> currently >> > looking at implementing in the Apache Metron project.” >> > >> > This is how I put it to Joe. >> > >> > I think what you are proposing would work. I think what I have done up >> > until now will pretty much work. What I have been thinking about and >> > considering >> > is the difference between getting ‘something that works’, and maybe >> > something better. >> > >> > So if you look at nar there is the ‘packaging’ part, and the class >> loading >> > part. >> > We are already doing almost the same thing with the assembly of the >> > .tag.gz. The Nar is a next step to this which adds more metadata and > > the >> > dependency repo. >> > More of a refinement than a change. >> > >> > As far as class loading, the Nar is a more refined system for deploying >> > and consuming jars and dependencies, and setting up classloader >> instances. >> > It has more >> > functionality than we need at the moment in storm, but in other > > services >> > where multiple parsers or plugin types may need to be loaded, it would >> make >> > more sense. >> > Rest may be that case. Stellar may be that case too, if anyone ever >> > writes a stellar function with different dependencies than the > > platform. >> > >> > >> > >> > On March 10, 2017 at 14:32:00, Casey Stella (ceste...@gmail.com) wrote: >> > >> > So, my question is whether we really need nar here. We have a >> classloading >> > mechanism that will allow us to deploy just the parser logic just added >> > into master for stellar, should we be considering another one? >> > >> > I would understand using nar if we needed to have multiple nars around >> > that >> > needed isolation from one another, but in the parser topology, we get >> that >> > isolation naturally. It seems to me that, at least for a MVP, we should >> > use the existing classloader that we just added. That being said, I > > might >> > be missing something, so let me know your thoughts. >> > >> > Casey >> > >> > On Fri, Mar 10, 2017 at 2:18 PM, Matt Foley <ma...@apache.org> wrote: >> > >> > > I like the approach. I think Nar constitutes a production-quality >> > > existing solution meeting highly similar needs to Metron’s. >> > > >> > > Just a ‘btw’ regarding Joe’s input that I transmitted: >> > > - Joe made clear that he was only giving his personal opinion, since > > of >> > > course no individual can speak for the community. >> > > - Joe also felt that if Metron succeeded in re-using the Nar system >> > > without having to change it too much, that that would be a good >> > supporting >> > > argument for later proposing that it become a separate child project. >> > > - Whereas if we or they tried to break it out as a separate project >> now, >> > > we would have to do all the community-building work around it, as > > well >> > as >> > > the technical work of adapting it for a different environment from >> NiFi. >> > > - So he recommended to copy and appropriate it for now. >> > > - Which I also agree with. >> > > >> > > Thanks, >> > > --Matt >> > > >> > > On 3/10/17, 7:42 AM, "Otto Fowler" <ottobackwa...@gmail.com> wrote: >> > > >> > > As previously discussed here, I have been working on side loading of >> > > parsers. The goals of this work are: >> > > * Make it possible of developers to create, maintain and deploy > > parsers >> > > outside of the Metron code tree and not have to fork >> > > * Create maven archetype support for developers of parsers >> > > * Introduce a parser ‘lifecycle’ to support multiple instances and >> > > configurations, states of being installed, under configuration, and >> > > deployed >> > > etc. >> > > >> > > I would like to have some discussion based on where I am after > > rebasing >> > > onto METRON-671 which revamps deployment to be totally ambari based. >> > > >> > > >> > > Packaging and Loading and Extensions >> > > >> > > I have mentioned previously, and we have discussed on list wanting to >> > > move >> > > away from uber’s for somethings to using custom class loaders ( from >> > > hdfs >> > > possibly ). >> > > We also want the REST api to work with 3rd parter parsers >> > > We would like to reduce the size of having so many ubers in the build >> > > We would benefit from tooling around this, maven building, archetypes >> > > etc >> > > We could benefit from explicitly required metadata and information >> > > We want a generic extension methodology >> > > We want to be able to upgrade parsers/extensions in some way >> > > >> > > >> > > I have also mentioned that this would look or work a lot like NiFi’s >> > > NAR >> > > system. >> > > >> > > Now I’m going to put it differently: >> > > >> > > I propose that we adapt and introduce the NAR system for Metron >> > > Extensions, >> > > starting with parsers, with that adoption >> > > extended to allow for VFS Classloading from hdfs as we are now doing >> > > with >> > > Stellar. And that this is done as a follow on to >> > > the base mvp side loading work. >> > > >> > > This provides a solution to the above issues, and would afford us a >> > > great >> > > amount of flexibility going forward. >> > > >> > > https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#nars >> > > https://cwiki.apache.org/confluence/display/NIFI/Maven+ >> > > Projects+for+Extensions >> > > >> > > The functional concept would be: >> > > * the archetype and all the parser projects produce nars ( either >> > > including the configuration and patterns or splitting between runtime >> > > ( nar >> > > ) and static ( tar.gz ) >> > > * these are not shaded, but have a ‘repo’ of dependencies for >> > > non-metron >> > > jars. Metron jars are provided and loaded through the classloading. >> > > * possibly the adaption of the Service Provider api/pattern for >> > > parsers and >> > > discovery >> > > * the nar repository/working directory structures would be >> > > implemented in >> > > /usr/metron/version/telemetry ( although discussion on having > > multiple >> > > extension directories vs. one extension dir are welcome ) >> > > * the storm process only references metron-parsers-common >> > > * the parser bolt uses the nar class loading system to load the > > parser >> > > * the rest api uses the nar class loading system to load the parser >> > > * etc etc >> > > * a new version of a parser is deployed as a nar, when the service >> > > restarts, the new nar is unpacked and replaces the old version in the >> > > working system ( we could change the restart requirement ……) >> > > >> > > The nar system gives us something that is: >> > > * production quality >> > > * small enough to grok and extend as opposed to some other solutions >> > > * comes with a highly accessible sister project >> > > * maven plugin tooling to build >> > > * reference archetype for packaging >> > > and other things >> > > >> > > There are a few ways we could approach using Nar: >> > > >> > > 1. ‘fork’ and appropriate the components and ‘metronize’ them >> > > * the maven plugin >> > > * the nar-utils package >> > > 2. Ask for and participate in an effort to pull NAR out into it’s own >> > > project, >> > > * make it more generic >> > > * usable by more than one project >> > > * goal to replace NiFi’s use of nar too >> > > 3. Create our own generic version as a fork >> > > * use it in metron >> > > * submit to NiFi as a proposal >> > > >> > > MattF was nice enough to float this by Joe Witt, who is at > > Hortonworks >> > > and >> > > is the NiFi lead. >> > > He agrees with the idea of making nar usable for multiple projects, > > but >> > > does not see them >> > > having time to do it soon, and suggests we go with 1. >> > > >> > > I will openly admit to ignorance about apache subproject or other >> > > things >> > > that might come into play here. >> > > >> > > I think that 1 is the best option in the near term, and if we can >> > > socialize >> > > and see that there is a demand we can move forward with 2/3 down the >> > > road. >> > > >> > > ?? >> > > Discuss >> > > Profit >> > > >> > > >> > > >> > > >> > >> > ------------------- Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org
