Hey, So I’m just philosophising a bit here: is it possible to create and submit topologies at runtime?
The thought here is to have a set of triples that define which operations are possible on the data at hand. A user can then request a certain end-result, and by using logic programming (or SPARQL) the graph for getting that result from the input is constructed dynamically from the triples. Generating the Storm topology would then involve building/retrieving the appropriate bolts and hooking them up in the manner described by the query. Then the topology could be submitted, the job processed and the result returned; after which the topology could be discarded. It seems like a very nice way of creating (efficient) transformations on data to many output formats. The concrete application here would be bioinformatics/epidemiology where a lot of different data formats, sources and tools exists; but how to wire them is often ad-hoc. Using logic programming to find the paths and getting storm to automatically build and run the topology would seem nice (albeit a bit academic)! Any thoughts on this?
signature.asc
Description: Message signed with OpenPGP using GPGMail
