Hi Dinusha, On Fri, Jun 29, 2018, 4:37 PM Dinusha Dissanayake <[email protected]> wrote:
> Hi Tishan, > >> >>>> One more thing. Can't we just save these zip files to file system >>>> rather than stressing STATS_DB. We use STATS_DB mainly to store end >>>> analytics data which is used by presentation layer(Dashboards). WDYT? >>>> >>> This would be problematic in HA deployment. If we keep them in the file >>> system and if a node goes down, we won't be able to retrieve the event >>> data in files in that node. >>> >> That we can solve by publishing to both DAS nodes from GW. Even earlier >> I was discussing with Fazlan to avoid adding file to DB by using file tail >> adaptor but later reverted due to zip files. But given that we are now >> using custom adaptor we can use files :) >> > If we publish to both DAS nodes, then the files would be available in both > nodes. When event publishing is happening by reading those files, the same > file will be processed from both the nodes right? :) > Then the same events will be accumulated twice as I see. > No that is handled by HA implementation. /Tishan > >> /Tishan >> >>> >>> >>> >> >>>> >>> /Tishan >>>> >>>> On Fri, Jun 29, 2018 at 2:42 PM, Tishan Dahanayakage <[email protected]> >>>> wrote: >>>> >>>>> Fazlan, >>>>> >>>>> On Fri, Jun 29, 2018 at 2:17 PM, Fazlan Nazeem <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> At the moment, analytics for microgateway is supported via a JAX-RS >>>>>> web app and a custom component which are deployed in APIM publisher node. >>>>>> The component was responsible for publishing the analytics data persisted >>>>>> in a DB table to the Analytics server via thrift. As an improvement for >>>>>> this, we have planned to move the web app to Analytics server and process >>>>>> the events within itself which will remove the overhead of publishing >>>>>> data >>>>>> via thrift. The micro-gateways will then upload the zip files >>>>>> with analytics data directly to the analytics server so that we can >>>>>> eliminate an unnecessary network hop. >>>>>> >>>>>> For this, we have developed a working prototype which follows the >>>>>> following design. >>>>>> >>>>>> [image: micro-analytics.jpg] >>>>>> >>>>>> With the above design, a user has to follow the following steps to >>>>>> setup analytics in APIM Analytics server for micro-gateway. >>>>>> >>>>>> 1) Deploy the JAX-RS web app. >>>>>> 2) Deploy the custom event receiver jar file to dropins. >>>>>> 3) Deploy the CAPP with the custom event receivers for required >>>>>> streams. >>>>>> 4) Create a table in STATS_DB to persist the zip file >>>>>> 5) Start analytics server with a set of system properties which will >>>>>> configure the Timer task intervals etc. >>>>>> >>>>> Can't we have these as parameters of the custom receiver so that we >>>>> can have them pre-configured offloading tasks from user. Or else set >>>>> reasonable defaults. And I believe timer tasks are started by Custom Event >>>>> Receiver. >>>>> >>>>> /Tishan >>>>> >>>>> >>>>>> >>>>>> The micro-gateway needs to be configured with the JAX-RS web app's >>>>>> URI so that it can periodically upload files with analytics data to the >>>>>> APIM Analytics server. >>>>>> >>>>>> Any feedback? >>>>>> >>>>>> >>>>>> Thanks & Regards, >>>>>> >>>>>> *Fazlan Nazeem* >>>>>> Senior Software Engineer >>>>>> WSO2 Inc >>>>>> Mobile : +94772338839 >>>>>> [email protected] >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Tishan Dahanayakage >>>>> Associate Technical Lead >>>>> WSO2, Inc. >>>>> Mobile:+94 716481328 >>>>> >>>>> Disclaimer: This communication may contain privileged or other >>>>> confidential information and is intended exclusively for the addressee/s. >>>>> If you are not the intended recipient/s, or believe that you may have >>>>> received this communication in error, please reply to the sender >>>>> indicating >>>>> that fact and delete the copy you received and in addition, you should not >>>>> print, copy, re-transmit, disseminate, or otherwise use the information >>>>> contained in this communication. Internet communications cannot be >>>>> guaranteed to be timely, secure, error or virus-free. The sender does not >>>>> accept liability for any errors or omissions. >>>>> >>>> >>>> >>>> >>>> -- >>>> Tishan Dahanayakage >>>> Associate Technical Lead >>>> WSO2, Inc. >>>> Mobile:+94 716481328 >>>> >>>> Disclaimer: This communication may contain privileged or other >>>> confidential information and is intended exclusively for the addressee/s. >>>> If you are not the intended recipient/s, or believe that you may have >>>> received this communication in error, please reply to the sender indicating >>>> that fact and delete the copy you received and in addition, you should not >>>> print, copy, re-transmit, disseminate, or otherwise use the information >>>> contained in this communication. Internet communications cannot be >>>> guaranteed to be timely, secure, error or virus-free. The sender does not >>>> accept liability for any errors or omissions. >>>> >>> >>> >>> >>> -- >>> Dinusha Dissanayake >>> Software Engineer >>> WSO2 Inc >>> Mobile: +94712939439 >>> <https://wso2.com/signature> >>> >> >> >> >> -- >> Tishan Dahanayakage >> Associate Technical Lead >> WSO2, Inc. >> Mobile:+94 716481328 >> >> Disclaimer: This communication may contain privileged or other >> confidential information and is intended exclusively for the addressee/s. >> If you are not the intended recipient/s, or believe that you may have >> received this communication in error, please reply to the sender indicating >> that fact and delete the copy you received and in addition, you should not >> print, copy, re-transmit, disseminate, or otherwise use the information >> contained in this communication. Internet communications cannot be >> guaranteed to be timely, secure, error or virus-free. The sender does not >> accept liability for any errors or omissions. >> > > > > -- > Dinusha Dissanayake > Software Engineer > WSO2 Inc > Mobile: +94712939439 > <https://wso2.com/signature> >
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
