Nathan Leung created STORM-561:
----------------------------------
Summary: Add ability to create topologies dynamically
Key: STORM-561
URL: https://issues.apache.org/jira/browse/STORM-561
Project: Apache Storm
Issue Type: Improvement
Reporter: Nathan Leung
Assignee: Nathan Leung
It would be nice if a storm topology could be built dynamically, instead of
requiring a recompile to change parameters (e.g. number of workers, number of
tasks, layout, etc).
I would propose the following data structures for building core storm
topologies. I haven't done a design for trident yet but the intention would be
to add trident support when core storm support is complete (or in parallel if
there are other people working on it):
{code}
// fields value and arguments are mutually exclusive
class Argument {
Class argumentType; // type used to lookup arguments in method/constructor
Class implementationType; // type used to create this argument
String value; // String used to construct this argument
List<Argument> arguments; // arguments used to build this argument
}
class Dependency {
String upstreamComponent; // name of upstream component
String grouping;
List<Argument> arguments; // arguments for the grouping
}
class StormSpout {
String name;
Class class; // class of this spout
int numTasks;
int numExecutors;
}
class StormBolt {
String name;
Class class;
int numTasks;
int numExecutors;
List<Dependency> dependencies;
}
class StormTopologyRepresentation {
String name;
List<StormSpout> spouts;
List<StormBolt> bolts;
Map config;
int numWorkers;
}
{code}
Topology creation will be built on top of the data structures above. The
benefits:
* Dependency free. Code to unmarshal from json, xml, etc, can be kept in
extensions, or as examples, and users can write a different unmarshaller if
they want to use a different text representation.
* support arbitrary spout and bolts types
* support of all groupings, streams, via reflections
* ability to specify configuration map via config file
* reification of spout / bolt / dependency arguments
** recursive argument reification for complex objects
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
