Hi,
On Mon, Jan 20, 2014 at 5:36 PM, Maninda Edirisooriya <[email protected]>wrote: > Already BAM can create streams with REST API. To facilitate further for a > person configuring BAM manually without a toolbox, we should provide a UI > to create streams. > CEP already has UI to create streams [1], I think it's for registry stream defintion store. I'm not sure whether it'll support cassandra stream definition store at the moment. We need to test and make it compatible with Cassandra stream definition store as well for BAM. (and better if possible to facilitate edit and delete streams) And that UI > should support defining indexes when required as current implementation of > defining indexes is not nice. > The event streams are common for BAM and CEP hence we should be careful when adding indexes, etc to that common UI where it's not relevant to CEP usecase. Here we have to load the attributes dynamically depending on what stream definition store (cassandra, registry, etc), so that we can add the relevant configuration like indexes, etc. [1] http://docs.wso2.org/display/CEP300/Working+with+Event+Streams Thanks, Sinthuja. > > As REST API fulfils light weight requirements of BAM, we can remove the > Thrift based stream creation feature from the data publisher side. In high > performance requirement of BAM, creating a stream from publisher side is > not valid. They should deploy a toolbox at the beginning instead. Anyway > this will make it backward incompatible for older BAM versions. > > And BTW what about keeping all the toolboxes in a WSO2 Store as discussed > earlier in the thread? > > > * Maninda Edirisooriya* > Software Engineer > > *WSO2, Inc.*lean.enterprise.middleware. > > *Blog* : http://maninda.blogspot.com/ > *Phone* : +94 777603226 > > > On Mon, Jan 20, 2014 at 4:58 PM, Paul Fremantle <[email protected]> wrote: > >> I'd like to add some requirements in if possible. >> >> 1) I'd like some really easy dynamic stuff. For example, if I'm using >> REST, can I just send a JSON? We could create a stream definition >> dynamically with the name based on some part of the URL and compare to >> previous definitions for a version number. This isn't going to be >> efficient, but in many cases I just want to get on with things. >> >> I think we have to be aware we have three potentially different usecases >> here: >> a) High performance BAM and CEP - need the stream defined in advance for >> efficiency >> b) Ad hoc BAM and logging - maybe we don't want to be so prescriptive / >> statically defined >> c) Lightweight CEP / prototyping - being able to use a stream like a >> dynamic language and do duck typing would be useful. >> >> 2) I'd like to support both a Registry model but also being able to point >> to a stream definition JSON via a URL. This would make this nice and >> "webby"/ REST-friendly. >> >> 3) We need to make it super easy to define a stream, edit stream >> definitions, find them, etc. >> >> Paul >> >> >> On 20 January 2014 11:10, Srinath Perera <[email protected]> wrote: >> >>> Hi All, >>> >>> I am reviving this thread as now we have support to add remove streams >>> though CEP UI (and BAM?) and via toolboxes, but we still define the event >>> streams with the data publisher each time. (this thread discusses about >>> stop doing that). >>> >>> Typical CEP/ BAM usecase would go like following >>> >>> 1) first you identify and define streams >>> 2) then you write queries >>> 3) then you write/configure the agents and point to streams they should >>> publish to. >>> >>> However, we redefine streams from agents every time we publish. This has >>> several complications. >>> >>> 1) If an agent choose to define stream different from queries, >>> everything get messed up. (problem was in samples in BAM). >>> 2) If there are multiple event receivers, we need a way to sync them >>> with db everytime we update a stream. >>> >>> I would like to propose we switch to a model where we write queries and >>> streams and deploy them as a toolbox. Then, just mention the stream name in >>> the event publisher to keep things simple. >>> >>> --Srinath >>> >>> >>> >>> On Sat, Jan 19, 2013 at 5:41 PM, Tharindu Mathew <[email protected]>wrote: >>> >>>> A store discover already supported event streams seems to be a good >>>> idea. It would be good idea which we can implement through a UI or a store. >>>> >>>> But, what do you mean by defining event streams everywhere? >>>> >>>> Clients can define any stream they want, but it is only defined at the >>>> stream definition store. If there is an error, it is shown in the stream >>>> definition store and returned to the client (I hope it is returned at >>>> least, otherwise it is a bug). >>>> >>>> This model was first proposed to get out of the eventing hell that we >>>> put ourselves into. Right now, if you want to publish something, you just >>>> define it and publish. You don't have to switch between multiple servers >>>> just to define and publish some events, which was extremely annoying. >>>> >>>> >>>> >>>> >>>> On Sat, Jan 19, 2013 at 2:28 PM, Amila Suriarachchi <[email protected]>wrote: >>>> >>>>> >>>>> >>>>> On Fri, Jan 18, 2013 at 7:54 PM, Sriskandarajah Suhothayan < >>>>> [email protected]> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Fri, Jan 18, 2013 at 6:46 PM, Sanjiva Weerawarana < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> +1 for the stream definition store idea (a new use of the Store). >>>>>>> >>>>>>> +1 >>>>>> >>>>>>> Suho can we change our client API to take the streamID as a param? >>>>>>> That way a user can look it up in the store and use it directly. >>>>>>> >>>>>>> Currently we do have server side APIs to get StreamDefintons from >>>>>> StreamId or StreamName & Version. >>>>>> >>>>> >>>>> I think we need to separate out the event definition from runtime >>>>> event publishing. >>>>> >>>>> If we look at how brokers used in CEP, first users need to define >>>>> broker at the broker Manager and use the broker id at the CEP bucket >>>>> level. >>>>> For event streams we can have default set of streams (which BAM publishers >>>>> and other default Agents use) and let users to define at run time. At the >>>>> event publishing side they can use the stream id. >>>>> >>>>> thanks, >>>>> Amila. >>>>> >>>>> >>>>>> >>>>>> Sanjiva. >>>>>>> >>>>>>> >>>>>>> On Fri, Jan 18, 2013 at 6:33 PM, Sriskandarajah Suhothayan < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Jan 18, 2013 at 4:23 PM, Srinath Perera >>>>>>>> <[email protected]>wrote: >>>>>>>> >>>>>>>>> Hi Suho, >>>>>>>>> >>>>>>>>> Yes, users can define them. But they do not always have to, and >>>>>>>>> we should recommend that users go and do that once rather than doing >>>>>>>>> it >>>>>>>>> every time. >>>>>>>>> >>>>>>>> Yes this is possible even now. First time when we define stream we >>>>>>>> get a streamId and then we can simply reuse that (even after restart). >>>>>>>> The issue is, our clients are not capable to store the streamId >>>>>>>> returned by the server, and therefore to get the already defined >>>>>>>> streamId >>>>>>>> corresponding to the stream definition we are redefining the stream. >>>>>>>> This is because our streamId is not simply regeneratable like >>>>>>>> <streamName>-<version> but we are using <streamName>-<version>-UUID. >>>>>>>> >>>>>>>> Suho >>>>>>>> >>>>>>>>> >>>>>>>>> --Srinath >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Jan 17, 2013 at 10:22 PM, Sriskandarajah Suhothayan < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, Jan 18, 2013 at 2:00 AM, Srinath Perera <[email protected] >>>>>>>>>> > wrote: >>>>>>>>>> >>>>>>>>>>> Currently we handle CEP streams by defining it everywhere we >>>>>>>>>>> publish >>>>>>>>>>> events, and ignoring the duplicates. However, if content of >>>>>>>>>>> events >>>>>>>>>>> change, then we throw an exception. >>>>>>>>>>> >>>>>>>>>>> I have concerns that we are creating a data stream definition >>>>>>>>>>> hell by >>>>>>>>>>> defining it every where. Also, when something changes, you would >>>>>>>>>>> have >>>>>>>>>>> to change things everywhere. >>>>>>>>>>> >>>>>>>>>>> Maybe we should just put that via registry in one place >>>>>>>>>>> >>>>>>>>>>> e.g. Go to registry and add the definitions. When you publish, >>>>>>>>>>> just >>>>>>>>>>> use event stream name only, and do not redefine events. >>>>>>>>>>> >>>>>>>>>>> +1 to make registry based Stream Definition Store. >>>>>>>>>> but we still need to allow the clients to define streams. >>>>>>>>>> We have the ability to delete stream so I believe that is good >>>>>>>>>> enough to. >>>>>>>>>> The disadvantage in Cassandra based Stream Definition Store is >>>>>>>>>> that it dont have a UI and hence we cannot tweak the stream >>>>>>>>>> Definition. >>>>>>>>>> IMHO having UI for the Stream Definition Store will be a solution. >>>>>>>>>> >>>>>>>>>> Suho >>>>>>>>>> >>>>>>>>>> --Srinath >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> ============================ >>>>>>>>>>> Srinath Perera, Ph.D. >>>>>>>>>>> Senior Software Architect, WSO2 Inc. >>>>>>>>>>> Visiting Faculty, University of Moratuwa >>>>>>>>>>> Member, Apache Software Foundation >>>>>>>>>>> Research Scientist, Lanka Software Foundation >>>>>>>>>>> Blog: http://srinathsview.blogspot.com/ >>>>>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>>>>> Phone: 0772360902 >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Architecture mailing list >>>>>>>>>>> [email protected] >>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> *S. Suhothayan* >>>>>>>>>> Software Engineer, >>>>>>>>>> Data Technologies Team, >>>>>>>>>> *WSO2, Inc. * >>>>>>>>>> *http://wso2.com <http://wso2.com/> <http://wso2.com/>* >>>>>>>>>> *lean.enterprise.middleware.* >>>>>>>>>> >>>>>>>>>> *email: **[email protected]* <[email protected]> >>>>>>>>>> * cell: (+94) 779 756 757 blog: * >>>>>>>>>> *http://suhothayan.blogspot.com/*<http://suhothayan.blogspot.com/> >>>>>>>>>> *twitter: >>>>>>>>>> **http://twitter.com/suhothayan*<http://twitter.com/suhothayan> >>>>>>>>>> * linked-in: **http://lk.linkedin.com/in/suhothayan >>>>>>>>>> <http://lk.linkedin.com/in/suhothayan>* >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> ============================ >>>>>>>>> Srinath Perera, Ph.D. >>>>>>>>> http://www.cs.indiana.edu/~hperera/ >>>>>>>>> http://srinathsview.blogspot.com/ >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> *S. Suhothayan* >>>>>>>> Software Engineer, >>>>>>>> Data Technologies Team, >>>>>>>> *WSO2, Inc. * >>>>>>>> *http://wso2.com <http://wso2.com/> <http://wso2.com/>* >>>>>>>> *lean.enterprise.middleware.* >>>>>>>> >>>>>>>> *email: **[email protected]* <[email protected]> >>>>>>>> * cell: (+94) 779 756 757 blog: >>>>>>>> **http://suhothayan.blogspot.com/*<http://suhothayan.blogspot.com/> >>>>>>>> *twitter: >>>>>>>> **http://twitter.com/suhothayan*<http://twitter.com/suhothayan> >>>>>>>> * linked-in: **http://lk.linkedin.com/in/suhothayan >>>>>>>> <http://lk.linkedin.com/in/suhothayan>* >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Architecture mailing list >>>>>>>> [email protected] >>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Sanjiva Weerawarana, Ph.D. >>>>>>> Founder, Chairman & CEO; WSO2, Inc.; http://wso2.com/ >>>>>>> email: [email protected]; phone: +94 11 763 9614; cell: +94 77 787 >>>>>>> 6880 | +1 650 265 8311 >>>>>>> blog: http://sanjiva.weerawarana.org/ >>>>>>> >>>>>>> Lean . Enterprise . Middleware >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Architecture mailing list >>>>>>> [email protected] >>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> *S. Suhothayan* >>>>>> Software Engineer, >>>>>> Data Technologies Team, >>>>>> *WSO2, Inc. * >>>>>> *http://wso2.com <http://wso2.com/> <http://wso2.com/>* >>>>>> *lean.enterprise.middleware.* >>>>>> >>>>>> *email: **[email protected]* <[email protected]> >>>>>> * cell: (+94) 779 756 757 blog: >>>>>> **http://suhothayan.blogspot.com/*<http://suhothayan.blogspot.com/> >>>>>> *twitter: **http://twitter.com/suhothayan*<http://twitter.com/suhothayan> >>>>>> * linked-in: **http://lk.linkedin.com/in/suhothayan >>>>>> <http://lk.linkedin.com/in/suhothayan>* >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> [email protected] >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *Amila Suriarachchi* >>>>> >>>>> Software Architect >>>>> WSO2 Inc. ; http://wso2.com >>>>> lean . enterprise . middleware >>>>> >>>>> phone : +94 71 3082805 >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> [email protected] >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> Regards, >>>> >>>> Tharindu Mathew >>>> >>>> Associate Technical Lead, WSO2 BAM >>>> Member - Data Mgmt. Committee >>>> >>>> blog: http://tharindumathew.com/ >>>> M: +94777759908 >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> [email protected] >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> ============================ >>> Srinath Perera, Ph.D. >>> http://people.apache.org/~hemapani/ >>> http://srinathsview.blogspot.com/ >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> Paul Fremantle >> CTO and Co-Founder, WSO2 >> OASIS WS-RX TC Co-chair, Apache Member >> >> UK: +44 207 096 0336 >> US: +1 646 595 7614 >> >> blog: http://pzf.fremantle.org >> twitter.com/pzfreo >> [email protected] >> >> wso2.com Lean Enterprise Middleware >> >> Disclaimer: This communication may contain privileged or other >> confidential information and is intended exclusively for the addressee/s. >> If you are not the intended recipient/s, or believe that you may have >> received this communication in error, please reply to the sender indicating >> that fact and delete the copy you received and in addition, you should not >> print, copy, retransmit, disseminate, or otherwise use the information >> contained in this communication. Internet communications cannot be >> guaranteed to be timely, secure, error or virus-free. The sender does not >> accept liability for any errors or omissions. >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- *Sinthuja Rajendran* Software Engineer <http://wso2.com/> WSO2, Inc.:http://wso2.com Blog: http://sinthu-rajan.blogspot.com/ Mobile: +94774273955
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
