Hi Srinath, On Fri, Mar 25, 2016 at 11:26 AM, Srinath Perera <[email protected]> wrote:
> As per meeting ( Paricipants: Sanjiva, Shankar, Sumedha, Anjana, Miyuru, > Seshika, Suho, Nirmal, Nuwan) > > Currently we generate several events per message from our products. For > example, when a message hits APIM, following events will be generated. > > > 1. One from HTTP level > 2. 1-2 from authentication and authorization logic > 3. 1 from Throttling > 4. 1 for ESB level stats > 5. 2 for request and response > > If APIM is handling 10K TPS, that means DAS is receiving events in about > 80K TPS. Although data bridge that transfers events are fast, writing to > Disk ( via RDBMS or Hbase) is a problem. We can scale Hbase. However, that > will run to a scenario where APIM deployment will need a very large > deployment of DAS. > > We decided to figure out a way to collect all the events and send a single > event to DAS. Basically idea is to extend the data publisher library such > that user can keep adding readings to the library, and it will collect the > readings and send them over as a single event to the server. > > However, some flows might terminated in the middle due to failures. There > are two solutions. > > > 1. Get the product to call a flush from a finally block > 2. Get the library to auto flush collected reading every few seconds > > I feel #2 is simpler. > > Do we have any concerns about going to this model? > > Suho, Anjana we need to think how to do this with our stream definition as > we force you to define the streams before hand. > Can't we write something similar to JDBC batch processing where the code would only do a publisher.addBatch() or something similar. The data publisher can be configured to flush the batched requests to DAS when they hit a certain threshold. Ex:- We define the batch size as 10(using code or config xml). Then if we have 5 streams, the publisher would send 5 requests to DAS(for each stream) instead of 50. IMO, this would allow us to keep the existing stream definitions and reduce the number of calls from a server to DAS. WDYT? Thanks, Janaka > > --Srinath > > > > > > -- > ============================ > Blog: http://srinathsview.blogspot.com twitter:@srinath_perera > Site: http://home.apache.org/~hemapani/ > Photos: http://www.flickr.com/photos/hemapani/ > Phone: 0772360902 > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- *Janaka Ranabahu* Associate Technical Lead, WSO2 Inc. http://wso2.com *E-mail: [email protected] <http://wso2.com>**M: **+94 718370861* Lean . Enterprise . Middleware
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
