Hi Sanjeewa,
I think here concern is the data amount we write to DB. IF we accumulated > data for 15 mins then we will have large amount of data. If we did simple > calculation assuming 500 byte per event, 4 events per message, 2000 TPS > then after 15 mins we may have 3GB file. Have we done some test and see > data amount written to database? If not shall we do quick test and see? > We can configure the file rotating time span. If it is a high TPS environment, we can configure the time to rotate the file such that it would not exceed a certain size. Then when it is compressed, size would be less (around 80 kb for 12MB file). > > And why are we uploading file to CXF app via http/s protocol? There are > much more effective protocols to transfer large files. Can we see other > alternatives as well? > > Thanks, > sanjeewa. > >> >> >> >>> /Tishan >>> >>>> >>>>> /Tishan >>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>>>> >>>>>> /Tishan >>>>>>> >>>>>>> On Fri, Jun 29, 2018 at 2:42 PM, Tishan Dahanayakage < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Fazlan, >>>>>>>> >>>>>>>> On Fri, Jun 29, 2018 at 2:17 PM, Fazlan Nazeem <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> At the moment, analytics for microgateway is supported via a >>>>>>>>> JAX-RS web app and a custom component which are deployed in APIM >>>>>>>>> publisher >>>>>>>>> node. The component was responsible for publishing the analytics data >>>>>>>>> persisted in a DB table to the Analytics server via thrift. As an >>>>>>>>> improvement for this, we have planned to move the web app to Analytics >>>>>>>>> server and process the events within itself which will remove the >>>>>>>>> overhead >>>>>>>>> of publishing data via thrift. The micro-gateways will then upload >>>>>>>>> the zip >>>>>>>>> files with analytics data directly to the analytics server so that we >>>>>>>>> can >>>>>>>>> eliminate an unnecessary network hop. >>>>>>>>> >>>>>>>>> For this, we have developed a working prototype which follows the >>>>>>>>> following design. >>>>>>>>> >>>>>>>>> [image: micro-analytics.jpg] >>>>>>>>> >>>>>>>>> With the above design, a user has to follow the following steps to >>>>>>>>> setup analytics in APIM Analytics server for micro-gateway. >>>>>>>>> >>>>>>>>> 1) Deploy the JAX-RS web app. >>>>>>>>> 2) Deploy the custom event receiver jar file to dropins. >>>>>>>>> 3) Deploy the CAPP with the custom event receivers for required >>>>>>>>> streams. >>>>>>>>> 4) Create a table in STATS_DB to persist the zip file >>>>>>>>> 5) Start analytics server with a set of system properties which >>>>>>>>> will configure the Timer task intervals etc. >>>>>>>>> >>>>>>>> Can't we have these as parameters of the custom receiver so that >>>>>>>> we can have them pre-configured offloading tasks from user. Or else set >>>>>>>> reasonable defaults. And I believe timer tasks are started by Custom >>>>>>>> Event >>>>>>>> Receiver. >>>>>>>> >>>>>>>> /Tishan >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> The micro-gateway needs to be configured with the JAX-RS web app's >>>>>>>>> URI so that it can periodically upload files with analytics data to >>>>>>>>> the >>>>>>>>> APIM Analytics server. >>>>>>>>> >>>>>>>>> Any feedback? >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks & Regards, >>>>>>>>> >>>>>>>>> *Fazlan Nazeem* >>>>>>>>> Senior Software Engineer >>>>>>>>> WSO2 Inc >>>>>>>>> Mobile : +94772338839 >>>>>>>>> [email protected] >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Tishan Dahanayakage >>>>>>>> Associate Technical Lead >>>>>>>> WSO2, Inc. >>>>>>>> Mobile:+94 716481328 >>>>>>>> >>>>>>>> Disclaimer: This communication may contain privileged or other >>>>>>>> confidential information and is intended exclusively for the >>>>>>>> addressee/s. >>>>>>>> If you are not the intended recipient/s, or believe that you may have >>>>>>>> received this communication in error, please reply to the sender >>>>>>>> indicating >>>>>>>> that fact and delete the copy you received and in addition, you should >>>>>>>> not >>>>>>>> print, copy, re-transmit, disseminate, or otherwise use the information >>>>>>>> contained in this communication. Internet communications cannot be >>>>>>>> guaranteed to be timely, secure, error or virus-free. The sender does >>>>>>>> not >>>>>>>> accept liability for any errors or omissions. >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Tishan Dahanayakage >>>>>>> Associate Technical Lead >>>>>>> WSO2, Inc. >>>>>>> Mobile:+94 716481328 >>>>>>> >>>>>>> Disclaimer: This communication may contain privileged or other >>>>>>> confidential information and is intended exclusively for the >>>>>>> addressee/s. >>>>>>> If you are not the intended recipient/s, or believe that you may have >>>>>>> received this communication in error, please reply to the sender >>>>>>> indicating >>>>>>> that fact and delete the copy you received and in addition, you should >>>>>>> not >>>>>>> print, copy, re-transmit, disseminate, or otherwise use the information >>>>>>> contained in this communication. Internet communications cannot be >>>>>>> guaranteed to be timely, secure, error or virus-free. The sender does >>>>>>> not >>>>>>> accept liability for any errors or omissions. >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Dinusha Dissanayake >>>>>> Software Engineer >>>>>> WSO2 Inc >>>>>> Mobile: +94712939439 >>>>>> <https://wso2.com/signature> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Tishan Dahanayakage >>>>> Associate Technical Lead >>>>> WSO2, Inc. >>>>> Mobile:+94 716481328 >>>>> >>>>> Disclaimer: This communication may contain privileged or other >>>>> confidential information and is intended exclusively for the addressee/s. >>>>> If you are not the intended recipient/s, or believe that you may have >>>>> received this communication in error, please reply to the sender >>>>> indicating >>>>> that fact and delete the copy you received and in addition, you should not >>>>> print, copy, re-transmit, disseminate, or otherwise use the information >>>>> contained in this communication. Internet communications cannot be >>>>> guaranteed to be timely, secure, error or virus-free. The sender does not >>>>> accept liability for any errors or omissions. >>>>> >>>> >>>> >>>> >>>> -- >>>> Dinusha Dissanayake >>>> Software Engineer >>>> WSO2 Inc >>>> Mobile: +94712939439 >>>> <https://wso2.com/signature> >>>> >>> >> >> -- >> Nuwan Dias >> >> Software Architect - WSO2, Inc. http://wso2.com >> email : [email protected] >> Phone : +94 777 775 729 >> > > > -- > *Sanjeewa Malalgoda* > WSO2 Inc. > Mobile : +94 712933253 > > <http://sanjeewamalalgoda.blogspot.com/>blog > :http://sanjeewamalalgoda.blogspot.com/ > <http://sanjeewamalalgoda.blogspot.com/> > > > -- Dinusha Dissanayake Software Engineer WSO2 Inc Mobile: +94712939439 <https://wso2.com/signature>
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
