Couple thoughts: 1. I see the Accumulo class loader allows multiple clients with potentially conflicting loads, via the “context” mechanism. That’s good. NiFi also used a multi-classloader mechanism to support potentially conflicting side-loads of their Processor bundles (“nars”), but I don’t think they supported re-loading (altho it’s been a few months since I looked at it).
2. I like the idea of loading from a configured location in HDFS. This gives a far smaller scope of filesystem to be watched and/or searched, and of course obviates the deploy-to-many-servers problem. Altho it costs another upload/maintenance tool for the admin to fiddle with. Thanks, --Matt On 2/27/17, 11:22 AM, "Casey Stella" <[email protected]> wrote: Hi All, The benefit of Stellar is that adding new functionality is as simple as providing a Jar. This enables people who want to integrate with Metron to easy add enrichments or other functionality. The snag currently with this is that we provide a single jar, so all stellar functions that we have available must be dependencies of the main jar that drives the topology plus what local directories we can configure via the storm configs. This makes the process of adding 3rd party jars not as easy as it could be. What I'm proposing is the following and I'd like to get some community feedback on it: - Split the stellar lang into its own project which does not shade its dependencies from metron-common - this makes creating your own stellar functions easier as you only need depend on a small project - Adjust the the following to additionally load classes from a location in HDFS /apps/metron/stellar using something like accumulo ( https://accumulo.apache.org/blog/2014/05/03/accumulo-classloader.html) - Profiler topology - Parser topology - Enrichment topology - Enrichment Flat file loader - Enrichment MR loader - Make the classloader reload upon new files - This would necessitate a new Stellar FunctionResolver I'd like to propose starting with the first two and attempting the third after we get something stable with the first 2. What this will give us is the following workflow to enable new stellar functions: - Build your function depending on stellar-lang into a Jar - Drop the new jar onto HDFS in /apps/metron/stellar - Restart the topology in question (after the 3rd bullet point, this is no longer required) Thoughts?
