Hi All,
I have written a Siddhi bolt that you can use to run Siddhi using Storm in
a distributed setup.
You can create a SiddhiBolt(s) given any Siddhi query like following.
SiddhiBolt siddhiBolt = new SiddhiBolt(
new String[]{ "define stream PlayStream1 ( sid string, ts long, x
double, y double, z double, a double, v double);"},
new String[]{ "from PlayStream1#window.timeBatch(1sec) select sid,
avg(v) as avgV insert into AvgRunPlay;" },
new String[]{"AvgRunPlay"});
Then those bolts can be used within Storm topology like any other bolt.
However, the name of components and streams used in CEP queries should
match.
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("PlayStream1", new FootballDataSpout(), 1);
builder.setBolt("AvgRunPlay", siddhiBolt1,
1).shuffleGrouping("PlayStream1");
builder.setBolt("FastRunPlay", siddhiBolt2,1).shuffleGrouping("AvgRunPlay");
builder.setBolt("LeafEacho", new EchoBolt(),
1).shuffleGrouping("FastRunPlay");
I have done a quick performance test and got about 140K TPS in local
cluster. We need to test using distributed setup. Lasantha will integrate
this with CEP code base.
Some potential TODO are
1) Write two new bolts for Siddhi that support reliable processing and
transaction processing using Storm constructs. (for cases where we need
high reliability while processing)
2) Integrate this with out data agent so we can send events into Storm
setup as well.
3) Extend the Siddhi language to support distributed processing, so above
topology can be written is Siddhi language itself.
If performance confirmed to be in the same range, given stability of Storm,
I think we can go with Storm for planned Siddhi distributed processing.
Thanks
Srinath
Code for the bolt can be found in
https://svn.wso2.org/repos/wso2/people/srinath/projects/siddhiStormIntegration/src/org/wso2/siddhi/storm/SiddhiBolt.java
.
Code can be found from
https://svn.wso2.org/repos/wso2/people/srinath/projects/siddhiStormIntegration
--
============================
Srinath Perera, Ph.D.
Director, Research, WSO2 Inc.
Visiting Faculty, University of Moratuwa
Member, Apache Software Foundation
Research Scientist, Lanka Software Foundation
Blog: http://srinathsview.blogspot.com/
Photos: http://www.flickr.com/photos/hemapani/
Phone: 0772360902
_______________________________________________
Architecture mailing list
[email protected]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
