Hi Dinusha, I have few questions.
1. Where does the file reader reside? Is it in publisher node? 2. What is the database are we using here? AM_DB? 3. Why can't we run a Ballerina task or data publisher in the micro-gateway to publish events to the Analytics server? Thank you! Pubudu. On Mon, Jun 4, 2018 at 10:05 PM Dinusha Dissanayake <[email protected]> wrote: > Hi all, > > We've been working on analytics filter for APIM 2.5 microgateway. > Following is the design we came up with. > > *1. Filtering the event data and writing them to a file.* > > > From the request stream, the relevant fields will be extracted and > requestDTO will be populated using those attributes. > EventDTO consist of streamID, timestamp, metadata, correlation data and > payload data. > EventDTO can be populated using requestDTO. > > Likewise when the response filters are available, we can extract the > necessary attributes and fill ExecutionTimeDTO and ResponseDTO. Then the > eventDTO can be populated corresponding to response related attributes. > > Once EventDTO is populated, we publish those events in to event stream. > A method to write those events to a file has been subscribed to the event > stream at the gateway initialization process. > Hence whenever a events stream gets an event, it will write that event to > the file we are defining. > > > *2. Event publishing from files to analytics server.* > > Files will be written with .dat extension. > For example say "api-usage-data.dat". > > In APIM v2.2 micro gateway, this file would rotate if the file size > exceeds 12mb or after a specific time interval. These values could be set > as configurable values. > Rolled file will be compressed with the timestamp. > eg: *api-usage-data.{timstamp}.zip* > This would reduce 12mb file into (approximately) 4kb file. > > Then this zip(s) will be uploaded and persisted in a DB. This will be done > using ballerina tasks(to make it happen periodically). > Then the relevant entries (files) will be read from the DB and populate > the events by reading line by line of that files. Then those events will be > published to the analytics server using existing data publisher. > > > > > Thanks, > DinushaD > > > > > -- > Dinusha Dissanayake > Software Engineer > WSO2 Inc > Mobile: +94712939439 > <https://wso2.com/signature> > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > -- *Pubudu Gunatilaka* Committer and PMC Member - Apache Stratos Senior Software Engineer WSO2, Inc.: http://wso2.com mobile : +94774078049 <javascript:void(0);>
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
