I started tinkering with the idea of the classloader to see about how hard it would be and if it would even be feasible and realized it pretty much writes itself for a MVP, so I submitted a PR (sans testing plan, which I'll get to today): METRON-744 ( https://github.com/apache/incubator-metron/pull/468)
This would essentially conform to the second step above. On Mon, Feb 27, 2017 at 2:52 PM, Matt Foley <[email protected]> wrote: > Couple thoughts: > > 1. I see the Accumulo class loader allows multiple clients with > potentially conflicting loads, via the “context” mechanism. That’s good. > NiFi also used a multi-classloader mechanism to support potentially > conflicting side-loads of their Processor bundles (“nars”), but I don’t > think they supported re-loading (altho it’s been a few months since I > looked at it). > > 2. I like the idea of loading from a configured location in HDFS. This > gives a far smaller scope of filesystem to be watched and/or searched, and > of course obviates the deploy-to-many-servers problem. Altho it costs > another upload/maintenance tool for the admin to fiddle with. > > Thanks, > --Matt > > On 2/27/17, 11:22 AM, "Casey Stella" <[email protected]> wrote: > > Hi All, > > The benefit of Stellar is that adding new functionality is as simple as > providing a Jar. This enables people who want to integrate with > Metron to > easy add enrichments or other functionality. The snag currently with > this > is that we provide a single jar, so all stellar functions that we have > available must be dependencies of the main jar that drives the topology > plus what local directories we can configure via the storm configs. > This > makes the process of adding 3rd party jars not as easy as it could be. > > What I'm proposing is the following and I'd like to get some community > feedback on it: > > - Split the stellar lang into its own project which does not shade > its > dependencies from metron-common > - this makes creating your own stellar functions easier as you > only > need depend on a small project > - Adjust the the following to additionally load classes from a > location > in HDFS /apps/metron/stellar using something like accumulo ( > https://accumulo.apache.org/blog/2014/05/03/accumulo- > classloader.html) > - Profiler topology > - Parser topology > - Enrichment topology > - Enrichment Flat file loader > - Enrichment MR loader > - Make the classloader reload upon new files > - This would necessitate a new Stellar FunctionResolver > > I'd like to propose starting with the first two and attempting the > third > after we get something stable with the first 2. > > What this will give us is the following workflow to enable new stellar > functions: > > - Build your function depending on stellar-lang into a Jar > - Drop the new jar onto HDFS in /apps/metron/stellar > - Restart the topology in question (after the 3rd bullet point, > this is > no longer required) > > Thoughts? > > > >
