I don't believe Storm does, we would have to modify our bolts to add an
additional classloader to the current classloader chain. Shouldn't be too
much work, but a ton of reward.
Accumulo had a similar requirement- distributing and synchronizing jars to
distributed components. They introduced a vfs classloader to their
On Mon, Sep 19, 2016 at 10:17 AM, Casey Stella <ceste...@gmail.com> wrote:
> I really would like this functionality. Just a brief aside on what this
> buys us. As it stands now, we have a few main extension points:
> - Custom Java parsers are found via fully qualified classname
> - Custom Stellar functions are found via annotation of the class and
> being dropped on the classpath
> - Custom Enrichment adapters (e.g. the geo enrichment) are found via
> incorporating changes into the enrichment topology flux file
> Right now the only way to add such things is to compile your code along
> with ours and have them placed in our uber jars that we submit to Storm.
> What would be better is for us to expose the interfaces and have developers
> use them to create their custom functionality, build just their extension
> points and the dependencies and drop the jar file in a directory to be
> picked up the next time the topology starts.
> I like the HDFS idea because it means that we do not have to ensure a 3rd
> party jar directory is sync'd across the storm supervisors. My question is
> whether Storm supports pulling external dependencies via HDFS. Does anyone
> Thanks for bringing this up, Mike.
> On Mon, Sep 19, 2016 at 10:08 AM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
> > As part of https://issues.apache.org/jira/browse/METRON-356 it is now
> > possible to add hbase and hadoop conf to the Storm topology classpath. It
> > is also desirable to expand this functionality to sideloading jars for
> > Storm topologies. That way, users can add additional dependencies without
> > having to recompile/repackage existing jars. One suggestion is to
> > HDFS to store custom jars and add them to the topology.classpath. I want
> > open this discussion to the community.
> > Best,
> > Mike