Hi all,
We've been working on analytics filter for APIM 2.5 microgateway. Following
is the design we came up with.
*1. Filtering the event data and writing them to a file.*
>From the request stream, the relevant fields will be extracted and
requestDTO will be populated using those attributes.
EventDTO consist of streamID, timestamp, metadata, correlation data and
payload data.
EventDTO can be populated using requestDTO.
Likewise when the response filters are available, we can extract the
necessary attributes and fill ExecutionTimeDTO and ResponseDTO. Then the
eventDTO can be populated corresponding to response related attributes.
Once EventDTO is populated, we publish those events in to event stream.
A method to write those events to a file has been subscribed to the event
stream at the gateway initialization process.
Hence whenever a events stream gets an event, it will write that event to
the file we are defining.
*2. Event publishing from files to analytics server.*
Files will be written with .dat extension.
For example say "api-usage-data.dat".
In APIM v2.2 micro gateway, this file would rotate if the file size exceeds
12mb or after a specific time interval. These values could be set as
configurable values.
Rolled file will be compressed with the timestamp.
eg: *api-usage-data.{timstamp}.zip*
This would reduce 12mb file into (approximately) 4kb file.
Then this zip(s) will be uploaded and persisted in a DB. This will be done
using ballerina tasks(to make it happen periodically).
Then the relevant entries (files) will be read from the DB and populate the
events by reading line by line of that files. Then those events will be
published to the analytics server using existing data publisher.
Thanks,
DinushaD
--
Dinusha Dissanayake
Software Engineer
WSO2 Inc
Mobile: +94712939439
<https://wso2.com/signature>
_______________________________________________
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
