Just so I'm clear Otto, when you discuss the UBER jar, this is a basic parser jar with the system libraries from Metron. And based on METRON-744, we would now be able to load parser jars that are created with the Maven archetypes developed in METRON-258 by loading them into a specific location in HDFS and simply restarting the topology? METRON-744 addresses Stellar functions, so it sounds like we would just need to have METRON-258 expand this capability to allow more than just custom classloading for Stellar functions. I like the sound of this. Have I missed anything?
On Tue, Feb 28, 2017 at 10:36 AM, Otto Fowler <[email protected]> wrote: > Casey and I had some discussion on this today > <http://airmail.calendar/2017-02-28%2012:00:00%20EST>, and this what we > roughed out with regards to custom class loaders and parsers and like to > throw out there as a way forward. > > * See how METRON-744 goes, getting the class loading base framework going. > * Look at METRON-258, which is the initial side loading work for parsers ( > which itself is dependent on the METRON-671 ). > * We change the bolt’s parser loading to use the new class loading > mechanism, but with the same jars, as well as other areas where it would > makes sense to do so. > > * Evolve METRON-258’s ‘archive’ deployment to a more custom deployment > similar to NiFi’s nars > ** This would include changing the parser’s dependencies on ‘metron’ jars > to be provided > ** This would include a /repo deployment for the jars the parser depends > on ( outside of system jars ) > ** This would include archetype changes > ** Expand the class loading to support loading the repos > ** This would include deployment of parsers to hdfs > > The idea here being, there is only one UBER jar submitted to storm for > parsers. It contains the metron system frameworks. By configuration, it > creates a custom class loader for the parser/repo parser/lib directories > and loads and runs the parser then. > > > On February 28, 2017 at 09:28:31, Casey Stella ([email protected]) wrote: > > I started tinkering with the idea of the classloader to see about how hard > it would be and if it would even be feasible and realized it pretty much > writes itself for a MVP, so I submitted a PR (sans testing plan, which I'll > get to today): METRON-744 ( > https://github.com/apache/incubator-metron/pull/468) > > This would essentially conform to the second step above. > > On Mon, Feb 27, 2017 at 2:52 PM, Matt Foley <[email protected]> wrote: > > > Couple thoughts: > > > > 1. I see the Accumulo class loader allows multiple clients with > > potentially conflicting loads, via the “context” mechanism. That’s good. > > NiFi also used a multi-classloader mechanism to support potentially > > conflicting side-loads of their Processor bundles (“nars”), but I don’t > > think they supported re-loading (altho it’s been a few months since I > > looked at it). > > > > 2. I like the idea of loading from a configured location in HDFS. This > > gives a far smaller scope of filesystem to be watched and/or searched, > and > > of course obviates the deploy-to-many-servers problem. Altho it costs > > another upload/maintenance tool for the admin to fiddle with. > > > > Thanks, > > --Matt > > > > On 2/27/17, 11:22 AM, "Casey Stella" <[email protected]> wrote: > > > > Hi All, > > > > The benefit of Stellar is that adding new functionality is as simple as > > providing a Jar. This enables people who want to integrate with > > Metron to > > easy add enrichments or other functionality. The snag currently with > > this > > is that we provide a single jar, so all stellar functions that we have > > available must be dependencies of the main jar that drives the topology > > plus what local directories we can configure via the storm configs. > > This > > makes the process of adding 3rd party jars not as easy as it could be. > > > > What I'm proposing is the following and I'd like to get some community > > feedback on it: > > > > - Split the stellar lang into its own project which does not shade > > its > > dependencies from metron-common > > - this makes creating your own stellar functions easier as you > > only > > need depend on a small project > > - Adjust the the following to additionally load classes from a > > location > > in HDFS /apps/metron/stellar using something like accumulo ( > > https://accumulo.apache.org/blog/2014/05/03/accumulo- > > classloader.html) > > - Profiler topology > > - Parser topology > > - Enrichment topology > > - Enrichment Flat file loader > > - Enrichment MR loader > > - Make the classloader reload upon new files > > - This would necessitate a new Stellar FunctionResolver > > > > I'd like to propose starting with the first two and attempting the > > third > > after we get something stable with the first 2. > > > > What this will give us is the following workflow to enable new stellar > > functions: > > > > - Build your function depending on stellar-lang into a Jar > > - Drop the new jar onto HDFS in /apps/metron/stellar > > - Restart the topology in question (after the 3rd bullet point, > > this is > > no longer required) > > > > Thoughts? > > > > > > > > >
