Hi Sajith, The Gateway will not have access to any of the databases. Therefore it will use the filesystem to temporarily store the zip file until it is uploaded to the Analytics server.
On Tue, Jul 3, 2018 at 8:24 AM Sajith Perera <saji...@wso2.com> wrote: > > Whether we buffering the zip file in micro GW if the analytics server not > available? In such cases what if we give an option to persist zip file > directly to database by micro GW as analytics server anyway reading the > data from database. > > Regards, > SajithD > > On Tue, Jul 3, 2018 at 2:47 AM Tishan Dahanayakage <tis...@wso2.com> > wrote: > >> Sorry for the delayed reply Nuwan. I was traveling. >> >> On Fri, Jun 29, 2018 at 9:19 AM, Nuwan Dias <nuw...@wso2.com> wrote: >> >>> >>> >>> On Fri, Jun 29, 2018 at 4:43 PM Tishan Dahanayakage <tis...@wso2.com> >>> wrote: >>> >>>> Hi Dinusha, >>>> >>>> On Fri, Jun 29, 2018, 4:37 PM Dinusha Dissanayake <dinus...@wso2.com> >>>> wrote: >>>> >>>>> Hi Tishan, >>>>> >>>>>> >>>>>>>> One more thing. Can't we just save these zip files to file system >>>>>>>> rather than stressing STATS_DB. We use STATS_DB mainly to store end >>>>>>>> analytics data which is used by presentation layer(Dashboards). WDYT? >>>>>>>> >>>>>>> This would be problematic in HA deployment. If we keep them in the >>>>>>> file system and if a node goes down, we won't be able to retrieve the >>>>>>> event data in files in that node. >>>>>>> >>>>>> That we can solve by publishing to both DAS nodes from GW. Even >>>>>> earlier I was discussing with Fazlan to avoid adding file to DB by using >>>>>> file tail adaptor but later reverted due to zip files. But given that we >>>>>> are now using custom adaptor we can use files :) >>>>>> >>>>> If we publish to both DAS nodes, then the files would be available in >>>>> both nodes. When event publishing is happening by reading those files, the >>>>> same file will be processed from both the nodes right? :) >>>>> Then the same events will be accumulated twice as I see. >>>>> >>>> No that is handled by HA implementation. >>>> >>> >>> Didn't get that part. What do you mean by "handled by HA >>> implementation"? >>> >> >> When we configure DAS in HA mode, both receivers can receive same event >> and yet active node will do the presentation part. >> >>> >>> Another question is, how does the gateways know the number of DAS nodes >>> to upload to? In a HA scenario, the gateway will only see the LB URL >>> (because DAS will be proxied via an LB). In that case the gateway only >>> uploads to the LB url, it has no idea how many DAS nodes are behind that LB >>> and it doesn't need to know as well. >>> >> Yeah if DAS is proxied via LB then publishing to both nodes is an issue. >> Then file based solution becomes obsolete as we can't share files >> in-between nodes >> >> /Tishan >> >>> >>> To me it sounds like the problems we may have to solve by persisting to >>> local file systems in each DAS node are much more severe than the overhead >>> that gets added to the DB. Because in reality, each gateway will only >>> upload these files like once every 15 minutes. So in a system with 1 >>> gateway, we're introducing just 1 additional DB read/write per every 15 >>> minutes. Yes, it increases with the number of gateways in the system, in >>> which case we may have to reduce the upload frequency. >>> >>> >>> >>>> /Tishan >>>> >>>>> >>>>>> /Tishan >>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>>>> >>>>>>> /Tishan >>>>>>>> >>>>>>>> On Fri, Jun 29, 2018 at 2:42 PM, Tishan Dahanayakage < >>>>>>>> tis...@wso2.com> wrote: >>>>>>>> >>>>>>>>> Fazlan, >>>>>>>>> >>>>>>>>> On Fri, Jun 29, 2018 at 2:17 PM, Fazlan Nazeem <fazl...@wso2.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> At the moment, analytics for microgateway is supported via a >>>>>>>>>> JAX-RS web app and a custom component which are deployed in APIM >>>>>>>>>> publisher >>>>>>>>>> node. The component was responsible for publishing the analytics data >>>>>>>>>> persisted in a DB table to the Analytics server via thrift. As an >>>>>>>>>> improvement for this, we have planned to move the web app to >>>>>>>>>> Analytics >>>>>>>>>> server and process the events within itself which will remove the >>>>>>>>>> overhead >>>>>>>>>> of publishing data via thrift. The micro-gateways will then upload >>>>>>>>>> the zip >>>>>>>>>> files with analytics data directly to the analytics server so that >>>>>>>>>> we can >>>>>>>>>> eliminate an unnecessary network hop. >>>>>>>>>> >>>>>>>>>> For this, we have developed a working prototype which follows the >>>>>>>>>> following design. >>>>>>>>>> >>>>>>>>>> [image: micro-analytics.jpg] >>>>>>>>>> >>>>>>>>>> With the above design, a user has to follow the following steps >>>>>>>>>> to setup analytics in APIM Analytics server for micro-gateway. >>>>>>>>>> >>>>>>>>>> 1) Deploy the JAX-RS web app. >>>>>>>>>> 2) Deploy the custom event receiver jar file to dropins. >>>>>>>>>> 3) Deploy the CAPP with the custom event receivers for required >>>>>>>>>> streams. >>>>>>>>>> 4) Create a table in STATS_DB to persist the zip file >>>>>>>>>> 5) Start analytics server with a set of system properties which >>>>>>>>>> will configure the Timer task intervals etc. >>>>>>>>>> >>>>>>>>> Can't we have these as parameters of the custom receiver so that >>>>>>>>> we can have them pre-configured offloading tasks from user. Or else >>>>>>>>> set >>>>>>>>> reasonable defaults. And I believe timer tasks are started by Custom >>>>>>>>> Event >>>>>>>>> Receiver. >>>>>>>>> >>>>>>>>> /Tishan >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> The micro-gateway needs to be configured with the JAX-RS web >>>>>>>>>> app's URI so that it can periodically upload files with analytics >>>>>>>>>> data to >>>>>>>>>> the APIM Analytics server. >>>>>>>>>> >>>>>>>>>> Any feedback? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks & Regards, >>>>>>>>>> >>>>>>>>>> *Fazlan Nazeem* >>>>>>>>>> Senior Software Engineer >>>>>>>>>> WSO2 Inc >>>>>>>>>> Mobile : +94772338839 >>>>>>>>>> fazl...@wso2.com >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Tishan Dahanayakage >>>>>>>>> Associate Technical Lead >>>>>>>>> WSO2, Inc. >>>>>>>>> Mobile:+94 716481328 >>>>>>>>> >>>>>>>>> Disclaimer: This communication may contain privileged or other >>>>>>>>> confidential information and is intended exclusively for the >>>>>>>>> addressee/s. >>>>>>>>> If you are not the intended recipient/s, or believe that you may have >>>>>>>>> received this communication in error, please reply to the sender >>>>>>>>> indicating >>>>>>>>> that fact and delete the copy you received and in addition, you >>>>>>>>> should not >>>>>>>>> print, copy, re-transmit, disseminate, or otherwise use the >>>>>>>>> information >>>>>>>>> contained in this communication. Internet communications cannot be >>>>>>>>> guaranteed to be timely, secure, error or virus-free. The sender does >>>>>>>>> not >>>>>>>>> accept liability for any errors or omissions. >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Tishan Dahanayakage >>>>>>>> Associate Technical Lead >>>>>>>> WSO2, Inc. >>>>>>>> Mobile:+94 716481328 >>>>>>>> >>>>>>>> Disclaimer: This communication may contain privileged or other >>>>>>>> confidential information and is intended exclusively for the >>>>>>>> addressee/s. >>>>>>>> If you are not the intended recipient/s, or believe that you may have >>>>>>>> received this communication in error, please reply to the sender >>>>>>>> indicating >>>>>>>> that fact and delete the copy you received and in addition, you should >>>>>>>> not >>>>>>>> print, copy, re-transmit, disseminate, or otherwise use the information >>>>>>>> contained in this communication. Internet communications cannot be >>>>>>>> guaranteed to be timely, secure, error or virus-free. The sender does >>>>>>>> not >>>>>>>> accept liability for any errors or omissions. >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Dinusha Dissanayake >>>>>>> Software Engineer >>>>>>> WSO2 Inc >>>>>>> Mobile: +94712939439 >>>>>>> <https://wso2.com/signature> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Tishan Dahanayakage >>>>>> Associate Technical Lead >>>>>> WSO2, Inc. >>>>>> Mobile:+94 716481328 >>>>>> >>>>>> Disclaimer: This communication may contain privileged or other >>>>>> confidential information and is intended exclusively for the addressee/s. >>>>>> If you are not the intended recipient/s, or believe that you may have >>>>>> received this communication in error, please reply to the sender >>>>>> indicating >>>>>> that fact and delete the copy you received and in addition, you should >>>>>> not >>>>>> print, copy, re-transmit, disseminate, or otherwise use the information >>>>>> contained in this communication. Internet communications cannot be >>>>>> guaranteed to be timely, secure, error or virus-free. The sender does not >>>>>> accept liability for any errors or omissions. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Dinusha Dissanayake >>>>> Software Engineer >>>>> WSO2 Inc >>>>> Mobile: +94712939439 >>>>> <https://wso2.com/signature> >>>>> >>>> >>> >>> -- >>> Nuwan Dias >>> >>> Software Architect - WSO2, Inc. http://wso2.com >>> email : nuw...@wso2.com >>> Phone : +94 777 775 729 >>> >> >> >> >> -- >> Tishan Dahanayakage >> Associate Technical Lead >> WSO2, Inc. >> Mobile:+94 716481328 >> >> Disclaimer: This communication may contain privileged or other >> confidential information and is intended exclusively for the addressee/s. >> If you are not the intended recipient/s, or believe that you may have >> received this communication in error, please reply to the sender indicating >> that fact and delete the copy you received and in addition, you should not >> print, copy, re-transmit, disseminate, or otherwise use the information >> contained in this communication. Internet communications cannot be >> guaranteed to be timely, secure, error or virus-free. The sender does not >> accept liability for any errors or omissions. >> _______________________________________________ >> Architecture mailing list >> Architecture@wso2.org >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> > -- > <http://wso2.com/signature> > Sajith Dimal > Software Engineer > Email : saji...@wso2.com > Mobile : +94783101496 > WSO2 Inc. | http://wso2.com > lean.enterprise.middleware > _______________________________________________ > Architecture mailing list > Architecture@wso2.org > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > -- Thanks & Regards, *Fazlan Nazeem* Senior Software Engineer WSO2 Inc Mobile : +94772338839 fazl...@wso2.com
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
