On Fri, Jun 29, 2018 at 5:50 PM Nuwan Dias <[email protected]> wrote: > > > On Fri, Jun 29, 2018 at 4:43 PM Tishan Dahanayakage <[email protected]> > wrote: > >> Hi Dinusha, >> >> On Fri, Jun 29, 2018, 4:37 PM Dinusha Dissanayake <[email protected]> >> wrote: >> >>> Hi Tishan, >>> >>>> >>>>>> One more thing. Can't we just save these zip files to file system >>>>>> rather than stressing STATS_DB. We use STATS_DB mainly to store end >>>>>> analytics data which is used by presentation layer(Dashboards). WDYT? >>>>>> >>>>> This would be problematic in HA deployment. If we keep them in the >>>>> file system and if a node goes down, we won't be able to retrieve the >>>>> event data in files in that node. >>>>> >>>> That we can solve by publishing to both DAS nodes from GW. Even >>>> earlier I was discussing with Fazlan to avoid adding file to DB by using >>>> file tail adaptor but later reverted due to zip files. But given that we >>>> are now using custom adaptor we can use files :) >>>> >>> If we publish to both DAS nodes, then the files would be available in >>> both nodes. When event publishing is happening by reading those files, the >>> same file will be processed from both the nodes right? :) >>> Then the same events will be accumulated twice as I see. >>> >> No that is handled by HA implementation. >> > > Didn't get that part. What do you mean by "handled by HA implementation"? > > Another question is, how does the gateways know the number of DAS nodes to > upload to? In a HA scenario, the gateway will only see the LB URL (because > DAS will be proxied via an LB). In that case the gateway only uploads to > the LB url, it has no idea how many DAS nodes are behind that LB and it > doesn't need to know as well. > > To me it sounds like the problems we may have to solve by persisting to > local file systems in each DAS node are much more severe than the overhead > that gets added to the DB. Because in reality, each gateway will only > upload these files like once every 15 minutes. So in a system with 1 > gateway, we're introducing just 1 additional DB read/write per every 15 > minutes. Yes, it increases with the number of gateways in the system, in > which case we may have to reduce the upload frequency. > I think here concern is the data amount we write to DB. IF we accumulated data for 15 mins then we will have large amount of data. If we did simple calculation assuming 500 byte per event, 4 events per message, 2000 TPS then after 15 mins we may have 3GB file. Have we done some test and see data amount written to database? If not shall we do quick test and see?
And why are we uploading file to CXF app via http/s protocol? There are much more effective protocols to transfer large files. Can we see other alternatives as well? Thanks, sanjeewa. > > > >> /Tishan >> >>> >>>> /Tishan >>>> >>>>> >>>>> >>>>> >>>> >>>>>> >>>>> /Tishan >>>>>> >>>>>> On Fri, Jun 29, 2018 at 2:42 PM, Tishan Dahanayakage <[email protected] >>>>>> > wrote: >>>>>> >>>>>>> Fazlan, >>>>>>> >>>>>>> On Fri, Jun 29, 2018 at 2:17 PM, Fazlan Nazeem <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> At the moment, analytics for microgateway is supported via a JAX-RS >>>>>>>> web app and a custom component which are deployed in APIM publisher >>>>>>>> node. >>>>>>>> The component was responsible for publishing the analytics data >>>>>>>> persisted >>>>>>>> in a DB table to the Analytics server via thrift. As an improvement for >>>>>>>> this, we have planned to move the web app to Analytics server and >>>>>>>> process >>>>>>>> the events within itself which will remove the overhead of publishing >>>>>>>> data >>>>>>>> via thrift. The micro-gateways will then upload the zip files >>>>>>>> with analytics data directly to the analytics server so that we can >>>>>>>> eliminate an unnecessary network hop. >>>>>>>> >>>>>>>> For this, we have developed a working prototype which follows the >>>>>>>> following design. >>>>>>>> >>>>>>>> [image: micro-analytics.jpg] >>>>>>>> >>>>>>>> With the above design, a user has to follow the following steps to >>>>>>>> setup analytics in APIM Analytics server for micro-gateway. >>>>>>>> >>>>>>>> 1) Deploy the JAX-RS web app. >>>>>>>> 2) Deploy the custom event receiver jar file to dropins. >>>>>>>> 3) Deploy the CAPP with the custom event receivers for required >>>>>>>> streams. >>>>>>>> 4) Create a table in STATS_DB to persist the zip file >>>>>>>> 5) Start analytics server with a set of system properties which >>>>>>>> will configure the Timer task intervals etc. >>>>>>>> >>>>>>> Can't we have these as parameters of the custom receiver so that we >>>>>>> can have them pre-configured offloading tasks from user. Or else set >>>>>>> reasonable defaults. And I believe timer tasks are started by Custom >>>>>>> Event >>>>>>> Receiver. >>>>>>> >>>>>>> /Tishan >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> The micro-gateway needs to be configured with the JAX-RS web app's >>>>>>>> URI so that it can periodically upload files with analytics data to the >>>>>>>> APIM Analytics server. >>>>>>>> >>>>>>>> Any feedback? >>>>>>>> >>>>>>>> >>>>>>>> Thanks & Regards, >>>>>>>> >>>>>>>> *Fazlan Nazeem* >>>>>>>> Senior Software Engineer >>>>>>>> WSO2 Inc >>>>>>>> Mobile : +94772338839 >>>>>>>> [email protected] >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Tishan Dahanayakage >>>>>>> Associate Technical Lead >>>>>>> WSO2, Inc. >>>>>>> Mobile:+94 716481328 >>>>>>> >>>>>>> Disclaimer: This communication may contain privileged or other >>>>>>> confidential information and is intended exclusively for the >>>>>>> addressee/s. >>>>>>> If you are not the intended recipient/s, or believe that you may have >>>>>>> received this communication in error, please reply to the sender >>>>>>> indicating >>>>>>> that fact and delete the copy you received and in addition, you should >>>>>>> not >>>>>>> print, copy, re-transmit, disseminate, or otherwise use the information >>>>>>> contained in this communication. Internet communications cannot be >>>>>>> guaranteed to be timely, secure, error or virus-free. The sender does >>>>>>> not >>>>>>> accept liability for any errors or omissions. >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Tishan Dahanayakage >>>>>> Associate Technical Lead >>>>>> WSO2, Inc. >>>>>> Mobile:+94 716481328 >>>>>> >>>>>> Disclaimer: This communication may contain privileged or other >>>>>> confidential information and is intended exclusively for the addressee/s. >>>>>> If you are not the intended recipient/s, or believe that you may have >>>>>> received this communication in error, please reply to the sender >>>>>> indicating >>>>>> that fact and delete the copy you received and in addition, you should >>>>>> not >>>>>> print, copy, re-transmit, disseminate, or otherwise use the information >>>>>> contained in this communication. Internet communications cannot be >>>>>> guaranteed to be timely, secure, error or virus-free. The sender does not >>>>>> accept liability for any errors or omissions. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Dinusha Dissanayake >>>>> Software Engineer >>>>> WSO2 Inc >>>>> Mobile: +94712939439 >>>>> <https://wso2.com/signature> >>>>> >>>> >>>> >>>> >>>> -- >>>> Tishan Dahanayakage >>>> Associate Technical Lead >>>> WSO2, Inc. >>>> Mobile:+94 716481328 >>>> >>>> Disclaimer: This communication may contain privileged or other >>>> confidential information and is intended exclusively for the addressee/s. >>>> If you are not the intended recipient/s, or believe that you may have >>>> received this communication in error, please reply to the sender indicating >>>> that fact and delete the copy you received and in addition, you should not >>>> print, copy, re-transmit, disseminate, or otherwise use the information >>>> contained in this communication. Internet communications cannot be >>>> guaranteed to be timely, secure, error or virus-free. The sender does not >>>> accept liability for any errors or omissions. >>>> >>> >>> >>> >>> -- >>> Dinusha Dissanayake >>> Software Engineer >>> WSO2 Inc >>> Mobile: +94712939439 >>> <https://wso2.com/signature> >>> >> > > -- > Nuwan Dias > > Software Architect - WSO2, Inc. http://wso2.com > email : [email protected] > Phone : +94 777 775 729 > -- *Sanjeewa Malalgoda* WSO2 Inc. Mobile : +94 712933253 <http://sanjeewamalalgoda.blogspot.com/>blog :http://sanjeewamalalgoda.blogspot.com/ <http://sanjeewamalalgoda.blogspot.com/>
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
