Hi Dinusha, How does this approach work when the backend does not respond? We should probably generate a faultDTO and publish it in such cases, as similar to what we are doing for the synapse gateway.
The filter should be intelligent to not generate an EventDTO when backend does not respond. On Thu, Jun 7, 2018 at 6:05 PM Dinusha Dissanayake <[email protected]> wrote: > Hi Pubudu, > > This implementation from reader point onward is similar to what we already > have in APIM v 2.2.0. > 1. So the reader resides in publisher ATM. > 2. Yes. We are using AM_DB since there is a table already for this > AM_USAGE_UPLOADED_FILES so we do not need new work here. > 3. Yes. This can be achieved via ballerina tasks. But we are making a > slight change here to upload file after each rotation. > ATM we are using file rotation based only on timstamps. (say if a file > will be rotated every 10 min ) > After rotating the file, the then rotated file name would be added to a > stream which is containing file names. > We can subscriber the uploading method to this stream. So once this > streams gets an entry, it will call the uploading method and the > corresponding file will be uploaded. > > > Thanks, > DinushaD. > > On Wed, Jun 6, 2018 at 5:52 AM, Pubudu Gunatilaka <[email protected]> > wrote: > >> Hi Dinusha, >> >> I have few questions. >> >> 1. Where does the file reader reside? Is it in publisher node? >> 2. What is the database are we using here? AM_DB? >> 3. Why can't we run a Ballerina task or data publisher in the >> micro-gateway to publish events to the Analytics server? >> >> Thank you! >> Pubudu. >> >> On Mon, Jun 4, 2018 at 10:05 PM Dinusha Dissanayake <[email protected]> >> wrote: >> >>> Hi all, >>> >>> We've been working on analytics filter for APIM 2.5 microgateway. >>> Following is the design we came up with. >>> >>> *1. Filtering the event data and writing them to a file.* >>> >>> >>> From the request stream, the relevant fields will be extracted and >>> requestDTO will be populated using those attributes. >>> EventDTO consist of streamID, timestamp, metadata, correlation data and >>> payload data. >>> EventDTO can be populated using requestDTO. >>> >>> Likewise when the response filters are available, we can extract the >>> necessary attributes and fill ExecutionTimeDTO and ResponseDTO. Then the >>> eventDTO can be populated corresponding to response related attributes. >>> >>> Once EventDTO is populated, we publish those events in to event stream. >>> A method to write those events to a file has been subscribed to the >>> event stream at the gateway initialization process. >>> Hence whenever a events stream gets an event, it will write that event >>> to the file we are defining. >>> >>> >>> *2. Event publishing from files to analytics server.* >>> >>> Files will be written with .dat extension. >>> For example say "api-usage-data.dat". >>> >>> In APIM v2.2 micro gateway, this file would rotate if the file size >>> exceeds 12mb or after a specific time interval. These values could be set >>> as configurable values. >>> Rolled file will be compressed with the timestamp. >>> eg: *api-usage-data.{timstamp}.zip* >>> This would reduce 12mb file into (approximately) 4kb file. >>> >>> Then this zip(s) will be uploaded and persisted in a DB. This will be >>> done using ballerina tasks(to make it happen periodically). >>> Then the relevant entries (files) will be read from the DB and populate >>> the events by reading line by line of that files. Then those events will be >>> published to the analytics server using existing data publisher. >>> >>> >>> >>> >>> Thanks, >>> DinushaD >>> >>> >>> >>> >>> -- >>> Dinusha Dissanayake >>> Software Engineer >>> WSO2 Inc >>> Mobile: +94712939439 >>> <https://wso2.com/signature> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >> >> >> -- >> *Pubudu Gunatilaka* >> Committer and PMC Member - Apache Stratos >> Senior Software Engineer >> WSO2, Inc.: http://wso2.com >> mobile : +94774078049 >> >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > Dinusha Dissanayake > Software Engineer > WSO2 Inc > Mobile: +94712939439 > <https://wso2.com/signature> > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > -- Thanks & Regards, *Fazlan Nazeem* Senior Software Engineer WSO2 Inc Mobile : +94772338839 [email protected]
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
