Hi All,
The benefit of Stellar is that adding new functionality is as simple as
providing a Jar. This enables people who want to integrate with Metron to
easy add enrichments or other functionality. The snag currently with this
is that we provide a single jar, so all stellar functions that we have
available must be dependencies of the main jar that drives the topology
plus what local directories we can configure via the storm configs. This
makes the process of adding 3rd party jars not as easy as it could be.
What I'm proposing is the following and I'd like to get some community
feedback on it:
- Split the stellar lang into its own project which does not shade its
dependencies from metron-common
- this makes creating your own stellar functions easier as you only
need depend on a small project
- Adjust the the following to additionally load classes from a location
in HDFS /apps/metron/stellar using something like accumulo (
https://accumulo.apache.org/blog/2014/05/03/accumulo-classloader.html)
- Profiler topology
- Parser topology
- Enrichment topology
- Enrichment Flat file loader
- Enrichment MR loader
- Make the classloader reload upon new files
- This would necessitate a new Stellar FunctionResolver
I'd like to propose starting with the first two and attempting the third
after we get something stable with the first 2.
What this will give us is the following workflow to enable new stellar
functions:
- Build your function depending on stellar-lang into a Jar
- Drop the new jar onto HDFS in /apps/metron/stellar
- Restart the topology in question (after the 3rd bullet point, this is
no longer required)
Thoughts?
