Yes, it's already possible.
1) You can build topology dynamically from your java code.
2) You can upload a jar with all dependencies of your bolts to Nimbus.
This can be done with StormSubmitter.submitJar(Map conf, String localJar)
3) You can submit your topology with NimbusClient. Check
StormSubmitter.submitTopology(...) for example code.
4) For querying topology check DRPC API.
Best regards,
Alexei Osipov.
On 18.04.2014 17:48, Joël Kuiper wrote:
Hey,
So I’m just philosophising a bit here: is it possible to create and submit
topologies at runtime?
The thought here is to have a set of triples that define which operations are
possible on the data at hand.
A user can then request a certain end-result, and by using logic programming
(or SPARQL) the graph for getting that result from the input is constructed
dynamically from the triples.
Generating the Storm topology would then involve building/retrieving the
appropriate bolts and hooking them up in the manner described by the query.
Then the topology could be submitted, the job processed and the result
returned; after which the topology could be discarded.
It seems like a very nice way of creating (efficient) transformations on data
to many output formats.
The concrete application here would be bioinformatics/epidemiology where a lot
of different data formats, sources and tools exists; but how to wire them is
often ad-hoc.
Using logic programming to find the paths and getting storm to automatically
build and run the topology would seem nice (albeit a bit academic)!
Any thoughts on this?
