Hi All, I have thought about Paul's idea at the product council to unify CEP and BAM Languages. Look like it can work.
1. Users can write all the queries using Siddhi language 2. If the windows defined in queries are large (e.g. say more than 15 minutes for batch windows or more than 15 minutes slides for sliding windows), the system will automatically generate Hive scripts and run the scripts in MapReduce. 3. If not queries get executed via CEP 4. Incoming data can marge with data stored (e.g. Database or a flat file), via event tables. We will have to do some work to make it work seamlessly. 5. If you combine smaller and larger windows, system should work using CEP and MapReduce side by side. As far as I can tell, anything can be done with Hive script can be done with Siddhi language. BAM CEP Retrieve All from S1 Retrive Some from S1[condition] Projection from .. select Sort Have to implement via sort window Group By via partitions or via group by transform transform function TBD Join Join with right windows Union ? Map/Reduce parition + queries => map send results to one stream, process => reduce There are few builtin functions missing like sin() .. that we can easily add. Pros === One language Cleaner model for both batch and realtime analytics Cons ==== This does not work for "data copied as flat files". Such files need to be replayed, which may be expensive. Thoughts please. Would that work? Thanks Srinath -- ============================ Srinath Perera, Ph.D. Director, Research, WSO2 Inc. Visiting Faculty, University of Moratuwa Member, Apache Software Foundation Research Scientist, Lanka Software Foundation Blog: http://srinathsview.blogspot.com/ Photos: http://www.flickr.com/photos/hemapani/ Phone: 0772360902
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
