Hi Sinthuja, What we are trying to do here is, persist the events coming from micro-gateway so that events would not be lost when publishing them to analytics. Microgateway should be able to run on its own. If analytics server is not present at the moment we are invoking APIs in microgateway, there should be a way to persist those data. Hence we are writing them into a file so that we can publish them into analytics later.
This is similar approach what we have done in APIM 2.2.0 and in cloud with on premises gateway concept. On Fri, Jun 8, 2018 at 5:34 PM, Sinthuja Rajendran <[email protected]> wrote: > Hi Dinusha, > > Can you explain what is the problem we are trying to solve with this > approach? I'm not clear based on our problem statement or use case based on > the explanation given above. > > Thanks, > Sinthuja. > > On Tue, Jun 5, 2018 at 10:35 AM Dinusha Dissanayake <[email protected]> > wrote: > >> Hi all, >> >> We've been working on analytics filter for APIM 2.5 microgateway. >> Following is the design we came up with. >> >> *1. Filtering the event data and writing them to a file.* >> >> >> From the request stream, the relevant fields will be extracted and >> requestDTO will be populated using those attributes. >> EventDTO consist of streamID, timestamp, metadata, correlation data and >> payload data. >> EventDTO can be populated using requestDTO. >> >> Likewise when the response filters are available, we can extract the >> necessary attributes and fill ExecutionTimeDTO and ResponseDTO. Then the >> eventDTO can be populated corresponding to response related attributes. >> >> Once EventDTO is populated, we publish those events in to event stream. >> A method to write those events to a file has been subscribed to the event >> stream at the gateway initialization process. >> Hence whenever a events stream gets an event, it will write that event to >> the file we are defining. >> >> >> *2. Event publishing from files to analytics server.* >> >> Files will be written with .dat extension. >> For example say "api-usage-data.dat". >> >> In APIM v2.2 micro gateway, this file would rotate if the file size >> exceeds 12mb or after a specific time interval. These values could be set >> as configurable values. >> Rolled file will be compressed with the timestamp. >> eg: *api-usage-data.{timstamp}.zip* >> This would reduce 12mb file into (approximately) 4kb file. >> >> Then this zip(s) will be uploaded and persisted in a DB. This will be >> done using ballerina tasks(to make it happen periodically). >> Then the relevant entries (files) will be read from the DB and populate >> the events by reading line by line of that files. Then those events will be >> published to the analytics server using existing data publisher. >> >> >> >> >> Thanks, >> DinushaD >> >> >> >> >> -- >> Dinusha Dissanayake >> Software Engineer >> WSO2 Inc >> Mobile: +94712939439 >> <https://wso2.com/signature> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> > > > -- > *Sinthuja Rajendran* > Senior Technical Lead > WSO2, Inc.:http://wso2.com > > Blog: http://sinthu-rajan.blogspot.com/ > Mobile: +94774273955 > > > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Dinusha Dissanayake Software Engineer WSO2 Inc Mobile: +94712939439 <https://wso2.com/signature>
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
