Hi Tishan,
On Fri, Jun 29, 2018 at 4:31 PM Tishan Dahanayakage <[email protected]> wrote: > Fazlan, > > On Fri, Jun 29, 2018 at 4:19 PM, Fazlan Nazeem <[email protected]> wrote: > >> Hi Tishan, >> On Fri, Jun 29, 2018 at 4:06 PM Tishan Dahanayakage <[email protected]> >> wrote: >> >>> Fazlan, >>> >>> On Fri, Jun 29, 2018 at 3:19 PM, Fazlan Nazeem <[email protected]> wrote: >>> >>>> Hi Tishan, >>>> On Fri, Jun 29, 2018 at 2:43 PM Tishan Dahanayakage <[email protected]> >>>> wrote: >>>> >>>>> Fazlan, >>>>> >>>>> On Fri, Jun 29, 2018 at 2:17 PM, Fazlan Nazeem <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> At the moment, analytics for microgateway is supported via a JAX-RS >>>>>> web app and a custom component which are deployed in APIM publisher node. >>>>>> The component was responsible for publishing the analytics data persisted >>>>>> in a DB table to the Analytics server via thrift. As an improvement for >>>>>> this, we have planned to move the web app to Analytics server and process >>>>>> the events within itself which will remove the overhead of publishing >>>>>> data >>>>>> via thrift. The micro-gateways will then upload the zip files >>>>>> with analytics data directly to the analytics server so that we can >>>>>> eliminate an unnecessary network hop. >>>>>> >>>>>> For this, we have developed a working prototype which follows the >>>>>> following design. >>>>>> >>>>>> [image: micro-analytics.jpg] >>>>>> >>>>>> With the above design, a user has to follow the following steps to >>>>>> setup analytics in APIM Analytics server for micro-gateway. >>>>>> >>>>>> 1) Deploy the JAX-RS web app. >>>>>> 2) Deploy the custom event receiver jar file to dropins. >>>>>> 3) Deploy the CAPP with the custom event receivers for required >>>>>> streams. >>>>>> 4) Create a table in STATS_DB to persist the zip file >>>>>> 5) Start analytics server with a set of system properties which will >>>>>> configure the Timer task intervals etc. >>>>>> >>>>> Can't we have these as parameters of the custom receiver so that we >>>>> can have them pre-configured offloading tasks from user. Or else set >>>>> reasonable defaults. And I believe timer tasks are started by Custom Event >>>>> Receiver. >>>>> >>>> Yes. We have set reasonable defaults. By default the receiver will not >>>> read from DB unless the a specific property is set to true. if it is set to >>>> true and other parameters are not configured, then the defaults will take >>>> affect. >>>> >>> I don't get this. So receiver will only read from DB if particular >>> property is set. Who sets this property? If this is user configurable it >>> will affect HA >>> >> All configurations have to be set via Java system properties when >> starting the server. >> e.g -Dusage.data.publishing.enabled=true >> If this property is not set true or not specified, even if the DB has >> unprocessed files, the event receiver will not read it from the DB. Only >> when this is set to true, the events will be read and processed. In HA >> mode, if all servers are configured to receive events, then the sytem >> property has to be added to all the servers. Can you elaborate if you see >> an instance where this will not work? >> > > This property will only work if we set "event.duplicated.in.cluster" to > true and set the system property true in both analytics(DAS) nodes. If you > are planning to add this property only to one node then that node should > be the active node in HA deployment. But you can't know active node in > advance. > > What is the rationale to add this property. IMO we shouldn't control event > consumption of a receiver from another path. > Ok. I guess you are suggesting that we should remove this configuration. This config has come into the current design from the earlier implementation, where the publisher node will only publish the file to analytics server when this config is set to true. We will remove this. Thanks for pointing this out. > /Tishan > >> >>> /Tishan >>> >>> >>>> >>>> Yes the timer tasks are started by the custom receiver component. >>>> >>>>> >>>>> /Tishan >>>>> >>>>> >>>>>> >>>>>> The micro-gateway needs to be configured with the JAX-RS web app's >>>>>> URI so that it can periodically upload files with analytics data to the >>>>>> APIM Analytics server. >>>>>> >>>>>> Any feedback? >>>>>> >>>>>> >>>>>> Thanks & Regards, >>>>>> >>>>>> *Fazlan Nazeem* >>>>>> Senior Software Engineer >>>>>> WSO2 Inc >>>>>> Mobile : +94772338839 >>>>>> [email protected] >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Tishan Dahanayakage >>>>> Associate Technical Lead >>>>> WSO2, Inc. >>>>> Mobile:+94 716481328 >>>>> >>>>> Disclaimer: This communication may contain privileged or other >>>>> confidential information and is intended exclusively for the addressee/s. >>>>> If you are not the intended recipient/s, or believe that you may have >>>>> received this communication in error, please reply to the sender >>>>> indicating >>>>> that fact and delete the copy you received and in addition, you should not >>>>> print, copy, re-transmit, disseminate, or otherwise use the information >>>>> contained in this communication. Internet communications cannot be >>>>> guaranteed to be timely, secure, error or virus-free. The sender does not >>>>> accept liability for any errors or omissions. >>>>> >>>> >>>> >>>> -- >>>> Thanks & Regards, >>>> >>>> *Fazlan Nazeem* >>>> Senior Software Engineer >>>> WSO2 Inc >>>> Mobile : +94772338839 >>>> [email protected] >>>> >>> >>> >>> >>> -- >>> Tishan Dahanayakage >>> Associate Technical Lead >>> WSO2, Inc. >>> Mobile:+94 716481328 >>> >>> Disclaimer: This communication may contain privileged or other >>> confidential information and is intended exclusively for the addressee/s. >>> If you are not the intended recipient/s, or believe that you may have >>> received this communication in error, please reply to the sender indicating >>> that fact and delete the copy you received and in addition, you should not >>> print, copy, re-transmit, disseminate, or otherwise use the information >>> contained in this communication. Internet communications cannot be >>> guaranteed to be timely, secure, error or virus-free. The sender does not >>> accept liability for any errors or omissions. >>> >> >> >> -- >> Thanks & Regards, >> >> *Fazlan Nazeem* >> Senior Software Engineer >> WSO2 Inc >> Mobile : +94772338839 >> [email protected] >> > > > > -- > Tishan Dahanayakage > Associate Technical Lead > WSO2, Inc. > Mobile:+94 716481328 > > Disclaimer: This communication may contain privileged or other > confidential information and is intended exclusively for the addressee/s. > If you are not the intended recipient/s, or believe that you may have > received this communication in error, please reply to the sender indicating > that fact and delete the copy you received and in addition, you should not > print, copy, re-transmit, disseminate, or otherwise use the information > contained in this communication. Internet communications cannot be > guaranteed to be timely, secure, error or virus-free. The sender does not > accept liability for any errors or omissions. > -- Thanks & Regards, *Fazlan Nazeem* Senior Software Engineer WSO2 Inc Mobile : +94772338839 [email protected]
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
