Hi Tishan, > >>> One more thing. Can't we just save these zip files to file system rather >>> than stressing STATS_DB. We use STATS_DB mainly to store end analytics data >>> which is used by presentation layer(Dashboards). WDYT? >>> >> This would be problematic in HA deployment. If we keep them in the file >> system and if a node goes down, we won't be able to retrieve the event >> data in files in that node. >> > That we can solve by publishing to both DAS nodes from GW. Even earlier I > was discussing with Fazlan to avoid adding file to DB by using file tail > adaptor but later reverted due to zip files. But given that we are now > using custom adaptor we can use files :) > If we publish to both DAS nodes, then the files would be available in both nodes. When event publishing is happening by reading those files, the same file will be processed from both the nodes right? :) Then the same events will be accumulated twice as I see.
> > /Tishan > >> >> >> > >>> >> /Tishan >>> >>> On Fri, Jun 29, 2018 at 2:42 PM, Tishan Dahanayakage <[email protected]> >>> wrote: >>> >>>> Fazlan, >>>> >>>> On Fri, Jun 29, 2018 at 2:17 PM, Fazlan Nazeem <[email protected]> >>>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> At the moment, analytics for microgateway is supported via a JAX-RS >>>>> web app and a custom component which are deployed in APIM publisher node. >>>>> The component was responsible for publishing the analytics data persisted >>>>> in a DB table to the Analytics server via thrift. As an improvement for >>>>> this, we have planned to move the web app to Analytics server and process >>>>> the events within itself which will remove the overhead of publishing data >>>>> via thrift. The micro-gateways will then upload the zip files >>>>> with analytics data directly to the analytics server so that we can >>>>> eliminate an unnecessary network hop. >>>>> >>>>> For this, we have developed a working prototype which follows the >>>>> following design. >>>>> >>>>> [image: micro-analytics.jpg] >>>>> >>>>> With the above design, a user has to follow the following steps to >>>>> setup analytics in APIM Analytics server for micro-gateway. >>>>> >>>>> 1) Deploy the JAX-RS web app. >>>>> 2) Deploy the custom event receiver jar file to dropins. >>>>> 3) Deploy the CAPP with the custom event receivers for required >>>>> streams. >>>>> 4) Create a table in STATS_DB to persist the zip file >>>>> 5) Start analytics server with a set of system properties which will >>>>> configure the Timer task intervals etc. >>>>> >>>> Can't we have these as parameters of the custom receiver so that we >>>> can have them pre-configured offloading tasks from user. Or else set >>>> reasonable defaults. And I believe timer tasks are started by Custom Event >>>> Receiver. >>>> >>>> /Tishan >>>> >>>> >>>>> >>>>> The micro-gateway needs to be configured with the JAX-RS web app's URI >>>>> so that it can periodically upload files with analytics data to the APIM >>>>> Analytics server. >>>>> >>>>> Any feedback? >>>>> >>>>> >>>>> Thanks & Regards, >>>>> >>>>> *Fazlan Nazeem* >>>>> Senior Software Engineer >>>>> WSO2 Inc >>>>> Mobile : +94772338839 >>>>> [email protected] >>>>> >>>> >>>> >>>> >>>> -- >>>> Tishan Dahanayakage >>>> Associate Technical Lead >>>> WSO2, Inc. >>>> Mobile:+94 716481328 >>>> >>>> Disclaimer: This communication may contain privileged or other >>>> confidential information and is intended exclusively for the addressee/s. >>>> If you are not the intended recipient/s, or believe that you may have >>>> received this communication in error, please reply to the sender indicating >>>> that fact and delete the copy you received and in addition, you should not >>>> print, copy, re-transmit, disseminate, or otherwise use the information >>>> contained in this communication. Internet communications cannot be >>>> guaranteed to be timely, secure, error or virus-free. The sender does not >>>> accept liability for any errors or omissions. >>>> >>> >>> >>> >>> -- >>> Tishan Dahanayakage >>> Associate Technical Lead >>> WSO2, Inc. >>> Mobile:+94 716481328 >>> >>> Disclaimer: This communication may contain privileged or other >>> confidential information and is intended exclusively for the addressee/s. >>> If you are not the intended recipient/s, or believe that you may have >>> received this communication in error, please reply to the sender indicating >>> that fact and delete the copy you received and in addition, you should not >>> print, copy, re-transmit, disseminate, or otherwise use the information >>> contained in this communication. Internet communications cannot be >>> guaranteed to be timely, secure, error or virus-free. The sender does not >>> accept liability for any errors or omissions. >>> >> >> >> >> -- >> Dinusha Dissanayake >> Software Engineer >> WSO2 Inc >> Mobile: +94712939439 >> <https://wso2.com/signature> >> > > > > -- > Tishan Dahanayakage > Associate Technical Lead > WSO2, Inc. > Mobile:+94 716481328 > > Disclaimer: This communication may contain privileged or other > confidential information and is intended exclusively for the addressee/s. > If you are not the intended recipient/s, or believe that you may have > received this communication in error, please reply to the sender indicating > that fact and delete the copy you received and in addition, you should not > print, copy, re-transmit, disseminate, or otherwise use the information > contained in this communication. Internet communications cannot be > guaranteed to be timely, secure, error or virus-free. The sender does not > accept liability for any errors or omissions. > -- Dinusha Dissanayake Software Engineer WSO2 Inc Mobile: +94712939439 <https://wso2.com/signature>
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
