Re: [Architecture] [APIM][Micro-Gateway][Analytics] Analytics for Micro-gateway

2018-07-03 Thread Sriskandarajah Suhothayan
IMO, we can upload the files to any of the DAS nodes and let it share among its HA seems to be the correct approach. Here we can set the event duplicated in the cluster property as falls all the DAS receivers. Doing so events will be shared and processed on both DAS HA nodes. On Tue, Jul 3, 2018

Re: [Architecture] [APIM][Micro-Gateway][Analytics] Analytics for Micro-gateway

2018-07-02 Thread Fazlan Nazeem
Hi Sajith, The Gateway will not have access to any of the databases. Therefore it will use the filesystem to temporarily store the zip file until it is uploaded to the Analytics server. On Tue, Jul 3, 2018 at 8:24 AM Sajith Perera wrote: > > Whether we buffering the zip file in micro GW if the

Re: [Architecture] [APIM][Micro-Gateway][Analytics] Analytics for Micro-gateway

2018-07-02 Thread Sajith Perera
Whether we buffering the zip file in micro GW if the analytics server not available? In such cases what if we give an option to persist zip file directly to database by micro GW as analytics server anyway reading the data from database. Regards, SajithD On Tue, Jul 3, 2018 at 2:47 AM Tishan

Re: [Architecture] [APIM][Micro-Gateway][Analytics] Analytics for Micro-gateway

2018-07-02 Thread Tishan Dahanayakage
Sorry for the delayed reply Nuwan. I was traveling. On Fri, Jun 29, 2018 at 9:19 AM, Nuwan Dias wrote: > > > On Fri, Jun 29, 2018 at 4:43 PM Tishan Dahanayakage > wrote: > >> Hi Dinusha, >> >> On Fri, Jun 29, 2018, 4:37 PM Dinusha Dissanayake >> wrote: >> >>> Hi Tishan, >>> >> One

Re: [Architecture] [APIM][Micro-Gateway][Analytics] Analytics for Micro-gateway

2018-07-02 Thread Dinusha Dissanayake
Hi Sanjeewa, I think here concern is the data amount we write to DB. IF we accumulated > data for 15 mins then we will have large amount of data. If we did simple > calculation assuming 500 byte per event, 4 events per message, 2000 TPS > then after 15 mins we may have 3GB file. Have we done

Re: [Architecture] [APIM][Micro-Gateway][Analytics] Analytics for Micro-gateway

2018-06-29 Thread Fazlan Nazeem
Hi Tishan, On Fri, Jun 29, 2018 at 4:31 PM Tishan Dahanayakage wrote: > Fazlan, > > On Fri, Jun 29, 2018 at 4:19 PM, Fazlan Nazeem wrote: > >> Hi Tishan, >> On Fri, Jun 29, 2018 at 4:06 PM Tishan Dahanayakage >> wrote: >> >>> Fazlan, >>> >>> On Fri, Jun 29, 2018 at 3:19 PM, Fazlan Nazeem

Re: [Architecture] [APIM][Micro-Gateway][Analytics] Analytics for Micro-gateway

2018-06-29 Thread Tishan Dahanayakage
Hi Dinusha, On Fri, Jun 29, 2018, 4:37 PM Dinusha Dissanayake wrote: > Hi Tishan, > >> One more thing. Can't we just save these zip files to file system rather than stressing STATS_DB. We use STATS_DB mainly to store end analytics data which is used by presentation

Re: [Architecture] [APIM][Micro-Gateway][Analytics] Analytics for Micro-gateway

2018-06-29 Thread Dinusha Dissanayake
Hi Tishan, > >>> One more thing. Can't we just save these zip files to file system rather >>> than stressing STATS_DB. We use STATS_DB mainly to store end analytics data >>> which is used by presentation layer(Dashboards). WDYT? >>> >> This would be problematic in HA deployment. If we keep them

Re: [Architecture] [APIM][Micro-Gateway][Analytics] Analytics for Micro-gateway

2018-06-29 Thread Tishan Dahanayakage
Fazlan, On Fri, Jun 29, 2018 at 4:19 PM, Fazlan Nazeem wrote: > Hi Tishan, > On Fri, Jun 29, 2018 at 4:06 PM Tishan Dahanayakage > wrote: > >> Fazlan, >> >> On Fri, Jun 29, 2018 at 3:19 PM, Fazlan Nazeem wrote: >> >>> Hi Tishan, >>> On Fri, Jun 29, 2018 at 2:43 PM Tishan Dahanayakage >>>

Re: [Architecture] [APIM][Micro-Gateway][Analytics] Analytics for Micro-gateway

2018-06-29 Thread Fazlan Nazeem
Hi Tishan, On Fri, Jun 29, 2018 at 4:06 PM Tishan Dahanayakage wrote: > Fazlan, > > On Fri, Jun 29, 2018 at 3:19 PM, Fazlan Nazeem wrote: > >> Hi Tishan, >> On Fri, Jun 29, 2018 at 2:43 PM Tishan Dahanayakage >> wrote: >> >>> Fazlan, >>> >>> On Fri, Jun 29, 2018 at 2:17 PM, Fazlan Nazeem

Re: [Architecture] [APIM][Micro-Gateway][Analytics] Analytics for Micro-gateway

2018-06-29 Thread Tishan Dahanayakage
Fazlan, On Fri, Jun 29, 2018 at 3:19 PM, Fazlan Nazeem wrote: > Hi Tishan, > On Fri, Jun 29, 2018 at 2:43 PM Tishan Dahanayakage > wrote: > >> Fazlan, >> >> On Fri, Jun 29, 2018 at 2:17 PM, Fazlan Nazeem wrote: >> >>> Hi all, >>> >>> At the moment, analytics for microgateway is supported via

Re: [Architecture] [APIM][Micro-Gateway][Analytics] Analytics for Micro-gateway

2018-06-29 Thread Tishan Dahanayakage
Dinusha, On Fri, Jun 29, 2018 at 3:01 PM, Dinusha Dissanayake wrote: > Hi Tishan, > > >> One more thing. Can't we just save these zip files to file system rather >> than stressing STATS_DB. We use STATS_DB mainly to store end analytics data >> which is used by presentation layer(Dashboards).

Re: [Architecture] [APIM][Micro-Gateway][Analytics] Analytics for Micro-gateway

2018-06-29 Thread Fazlan Nazeem
Hi Tishan, On Fri, Jun 29, 2018 at 2:43 PM Tishan Dahanayakage wrote: > Fazlan, > > On Fri, Jun 29, 2018 at 2:17 PM, Fazlan Nazeem wrote: > >> Hi all, >> >> At the moment, analytics for microgateway is supported via a JAX-RS web >> app and a custom component which are deployed in APIM publisher

Re: [Architecture] [APIM][Micro-Gateway][Analytics] Analytics for Micro-gateway

2018-06-29 Thread Fazlan Nazeem
Hi Rukshan, On Fri, Jun 29, 2018 at 2:39 PM Rukshan Premathunga wrote: > Hi Fazlan, >> >> >> 2) Deploy the custom event receiver jar file to dropins. >> 3) Deploy the CAPP with the custom event receivers for required streams. > > Why we need custom event receiver and capp here? Did we add a new

Re: [Architecture] [APIM][Micro-Gateway][Analytics] Analytics for Micro-gateway

2018-06-29 Thread Tishan Dahanayakage
Hi Fazlan, One more thing. Can't we just save these zip files to file system rather than stressing STATS_DB. We use STATS_DB mainly to store end analytics data which is used by presentation layer(Dashboards). WDYT? /Tishan On Fri, Jun 29, 2018 at 2:42 PM, Tishan Dahanayakage wrote: > Fazlan,

Re: [Architecture] [APIM][Micro-Gateway][Analytics] Analytics for Micro-gateway

2018-06-29 Thread Tishan Dahanayakage
Fazlan, On Fri, Jun 29, 2018 at 2:17 PM, Fazlan Nazeem wrote: > Hi all, > > At the moment, analytics for microgateway is supported via a JAX-RS web > app and a custom component which are deployed in APIM publisher node. The > component was responsible for publishing the analytics data persisted

Re: [Architecture] [APIM][Micro-Gateway][Analytics] Analytics for Micro-gateway

2018-06-29 Thread Rukshan Premathunga
Hi Fazlan, > > > 2) Deploy the custom event receiver jar file to dropins. > 3) Deploy the CAPP with the custom event receivers for required streams. Why we need custom event receiver and capp here? Did we add a new stream? And also when we persisting on statdb, we save it as a zip or events?

Re: [Architecture] [APIM][Micro-Gateway][Analytics] Analytics for Micro-gateway

2018-06-29 Thread Dinusha Dissanayake
Hi Fazlan, Please find my concerns below. > > ​ > With the above design, a user has to follow the following steps to setup > analytics in APIM Analytics server for micro-gateway. > > 1) Deploy the JAX-RS web app. > 2) Deploy the custom event receiver jar file to dropins. > 3) Deploy the CAPP

[Architecture] [APIM][Micro-Gateway][Analytics] Analytics for Micro-gateway

2018-06-29 Thread Fazlan Nazeem
Hi all, At the moment, analytics for microgateway is supported via a JAX-RS web app and a custom component which are deployed in APIM publisher node. The component was responsible for publishing the analytics data persisted in a DB table to the Analytics server via thrift. As an improvement for