Hi Sanjiva, The deleting works by giving the row key of the record, so it will not conflict with anything else. The scenario is, where after a batch job is run, lets say, some of the result is processed, needs to be sent as an notification to someone. For example, after the mediation stats are processed, we can check if a specific service is overloaded with requests or anything like that, and we send out that information out to a stream using this feature. And we can define a flow where we get event from that stream, and send out an email/sms notification.
The coordination across the Hadoop cluster works where, the result will only be written by a single operation, similar to records being written to a database. And the task the processes these data also is running in a fail-over aware manner, where if the task goes down, it will be be started in another node. Cheers, Anjana. On Thu, Oct 24, 2013 at 1:56 PM, Sanjiva Weerawarana <[email protected]>wrote: > Anjana given that Cassandra is not transactional how does deleting work > when someone else may be writing at the same time? > > I'm a bit unclear why its critical to send events out of Hive itself. Can > you elaborate the scenario please? How do you coordinate that across > potentially a large Hadoop cluster? > > Sanjiva. > > > On Tue, Oct 22, 2013 at 3:00 PM, Anjana Fernando <[email protected]> wrote: > >> Hi Srinath, >> >> Yeah, the data is always cleaned up when the task is run. Basically, the >> task reads all the data in the column family, send each event to the target >> stream, and in the same time, deletes it from the data store. >> >> The notification is totally customizable by the user, what this simply >> does is send any arbitrary data to a stream, at the point when possibly >> some "insert" statement is executed from the Hive script, which can be at >> the end of the script or anywhere. After the event comes to a stream, the >> user can do anything with it, either run an CEP query against it, or >> directly passthrough it to some transport like email or sms. >> >> Cheers, >> Anjana. >> >> >> On Tue, Oct 22, 2013 at 2:30 PM, Srinath Perera <[email protected]> wrote: >> >>> Hi Anjana, >>> >>> Basically, we are polling the cassandra location. I think it is OK. But >>> we need to make sure we clean up these tasks when we detected that job has >>> finished. >>> >>> What does the notification says? does it says job has finished or can >>> user give an condition when to send the notification? We eventually need >>> that. >>> >>> --Srinath >>> >>> >>> On Tue, Oct 22, 2013 at 2:03 PM, Anjana Fernando <[email protected]>wrote: >>> >>>> Hi, >>>> >>>> For BAM notification, the approach we have at the moment is, using CEP, >>>> which we do ship by default with BAM now. But, there is another limitation, >>>> where we cannot trigger any notifications from Hive scripts, which is what >>>> is used mostly. >>>> >>>> So the requirement is, somehow, we should be able to send messages from >>>> Hive to a stream to send out notifications, that is, when messages comes to >>>> a stream, we can use (CEPs) message builder/formatters to send out >>>> email/sms etc.. So I've implemented a simple mechanism to do this, where >>>> when Hive wants to send out a message to a stream, it will write a data row >>>> to a pre-defined Cassandra CF ("bam_notification_messages"), where it will >>>> have a column with the name "streamId", and other columns (maps to >>>> "payload" section of a stream). And then, in the BAM server, there is a >>>> scheduled task running, where it polls the data in that CF (5 second >>>> intervals), to get the existing rows, and reads the streamId and other >>>> columns to generate an event to be send to the target stream, and processed >>>> rows will be deleted. So with this approach, effectively, we can now send >>>> events to a specific stream from Hive. >>>> >>>> I've tested this feature in BAM. And hope this approach is fine for the >>>> requirement. >>>> >>>> Cheers, >>>> Anjana. >>>> >>>> -- >>>> *Anjana Fernando* >>>> Technical Lead >>>> WSO2 Inc. | http://wso2.com >>>> lean . enterprise . middleware >>>> >>> >>> >>> >>> -- >>> ============================ >>> Srinath Perera, Ph.D. >>> Director, Research, WSO2 Inc. >>> Visiting Faculty, University of Moratuwa >>> Member, Apache Software Foundation >>> Research Scientist, Lanka Software Foundation >>> Blog: http://srinathsview.blogspot.com/ >>> Photos: http://www.flickr.com/photos/hemapani/ >>> Phone: 0772360902 >>> >> >> >> >> -- >> *Anjana Fernando* >> Technical Lead >> WSO2 Inc. | http://wso2.com >> lean . enterprise . middleware >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > Sanjiva Weerawarana, Ph.D. > Founder, Chairman & CEO; WSO2, Inc.; http://wso2.com/ > email: [email protected]; phone: +94 11 763 9614; cell: +94 77 787 6880 | +1 > 650 265 8311 > blog: http://sanjiva.weerawarana.org/ > > Lean . Enterprise . Middleware > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- *Anjana Fernando* Technical Lead WSO2 Inc. | http://wso2.com lean . enterprise . middleware
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
