Hi Chamila
We are handling it through the code.
.dat will be compressed according to a configured time interval (say per
every 10 mins). Then the zip will be uploaded to endpoint as mentioned
above. If it is a success, then the zip will be deleted.
Table clean up task will be running and cleaning the DB periodically.
@Fazlan is working on that I think.
On Thu, Jun 21, 2018 at 2:31 PM, Chamila Adhikarinayake <chami...@wso2.com>
wrote:
> Hi Dinusha,
>
> How do we handle the cleanup tasks related to this feature? (Deleting .dat
> files, removing entries from AM_USAGE_UPLOADED_FILES) . These tables will
> grow with time (same as the auth token table issue currently we have in
> APIM/IS). Are we handling it through code (remove entries/files after
> reading or similar method) or running a separate script to remove them?.
>
>
> Chamila.
>
>
> On Wed, Jun 20, 2018 at 11:29 AM, Dinusha Dissanayake <dinus...@wso2.com>
> wrote:
>
>> Hi Sinthuja,
>>
>> What we are trying to do here is, persist the events coming from
>> micro-gateway so that events would not be lost when publishing them to
>> analytics.
>> Microgateway should be able to run on its own. If analytics server is not
>> present at the moment we are invoking APIs in microgateway, there should be
>> a way to persist those data. Hence we are writing them into a file so that
>> we can publish them into analytics later.
>>
>> This is similar approach what we have done in APIM 2.2.0 and in cloud
>> with on premises gateway concept.
>>
>> On Fri, Jun 8, 2018 at 5:34 PM, Sinthuja Rajendran <sinth...@wso2.com>
>> wrote:
>>
>>> Hi Dinusha,
>>>
>>> Can you explain what is the problem we are trying to solve with this
>>> approach? I'm not clear based on our problem statement or use case based on
>>> the explanation given above.
>>>
>>> Thanks,
>>> Sinthuja.
>>>
>>> On Tue, Jun 5, 2018 at 10:35 AM Dinusha Dissanayake <dinus...@wso2.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> We've been working on analytics filter for APIM 2.5 microgateway.
>>>> Following is the design we came up with.
>>>>
>>>> *1. Filtering the event data and writing them to a file.*
>>>>
>>>>
>>>> From the request stream, the relevant fields will be extracted and
>>>> requestDTO will be populated using those attributes.
>>>> EventDTO consist of streamID, timestamp, metadata, correlation data and
>>>> payload data.
>>>> EventDTO can be populated using requestDTO.
>>>>
>>>> Likewise when the response filters are available, we can extract the
>>>> necessary attributes and fill ExecutionTimeDTO and ResponseDTO. Then the
>>>> eventDTO can be populated corresponding to response related attributes.
>>>>
>>>> Once EventDTO is populated, we publish those events in to event stream.
>>>> A method to write those events to a file has been subscribed to the
>>>> event stream at the gateway initialization process.
>>>> Hence whenever a events stream gets an event, it will write that event
>>>> to the file we are defining.
>>>>
>>>>
>>>> *2. Event publishing from files to analytics server.*
>>>>
>>>> Files will be written with .dat extension.
>>>> For example say "api-usage-data.dat".
>>>>
>>>> In APIM v2.2 micro gateway, this file would rotate if the file size
>>>> exceeds 12mb or after a specific time interval. These values could be set
>>>> as configurable values.
>>>> Rolled file will be compressed with the timestamp.
>>>> eg: *api-usage-data.{timstamp}.zip*
>>>> This would reduce 12mb file into (approximately) 4kb file.
>>>>
>>>> Then this zip(s) will be uploaded and persisted in a DB. This will be
>>>> done using ballerina tasks(to make it happen periodically).
>>>> Then the relevant entries (files) will be read from the DB and populate
>>>> the events by reading line by line of that files. Then those events will be
>>>> published to the analytics server using existing data publisher.
>>>>
>>>>
>>>>
>>>>
>>>> Thanks,
>>>> DinushaD
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Dinusha Dissanayake
>>>> Software Engineer
>>>> WSO2 Inc
>>>> Mobile: +94712939439
>>>> <https://wso2.com/signature>
>>>> _______________________________________________
>>>> Architecture mailing list
>>>> Architecture@wso2.org
>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>>>
>>>
>>>
>>> --
>>> *Sinthuja Rajendran*
>>> Senior Technical Lead
>>> WSO2, Inc.:http://wso2.com
>>>
>>> Blog: http://sinthu-rajan.blogspot.com/
>>> Mobile: +94774273955
>>>
>>>
>>>
>>> _______________________________________________
>>> Architecture mailing list
>>> Architecture@wso2.org
>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>>
>>>
>>
>>
>> --
>> Dinusha Dissanayake
>> Software Engineer
>> WSO2 Inc
>> Mobile: +94712939439
>> <https://wso2.com/signature>
>>
>> _______________________________________________
>> Architecture mailing list
>> Architecture@wso2.org
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>
>
> --
> Regards,
> Chamila Adhikarinayake
> WSO2, Inc.
> Mobile - +94712346437
> Email - chami...@wso2.com
> Blog - http://helpfromadhi.blogspot.com/
>
--
Dinusha Dissanayake
Software Engineer
WSO2 Inc
Mobile: +94712939439
<https://wso2.com/signature>
_______________________________________________
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
