Re: [Architecture] [APIM] Sending Custom headers to Websocket back ends.

[Menaka Jayawardena]

Hi Harsha,

We discussed this in the code review as well. But, the issue is, the only
way to pass any additional information in websocket handshake is as
headers. When the request is forwarded to the websocket transport layer,
all these headers are added to the axis2 message context as properties. So
we cannot distinguish the incomming headers with other properties.

As a solution for this, users can send an additional header which contains
the list of headers that should be preserved or we define a prefix which
users should add to the header where we can filter them from the
properties.

A per our initial requirement, adding jwt header, I have followed the
second option because it can be used for other headers as well. So that the
websocket transport implementation can be done in a generic way.

Thanks and Regards,
Menaka



On Fri, Jun 29, 2018 at 9:32 PM Harsha Kumara  wrote:

> On Fri, Jun 29, 2018 at 8:10 PM Menaka Jayawardena 
> wrote:
>
>> Hi Harsha, Chamin,
>>
>> Please find my answers inline.
>>
>> So this means we are having two ways of handling JWT (normal method and
>>> WS specific method) scenarios? If so, we will need additional methods to
>>> cover this flow. Will there be a code/logic duplication due to this?
>>>
>>
>> No. In this implementation, the same JWT token generation method is used.
>> The default ws token validation method is modified to generate the jwt
>> token.
>>
>>
>> https://github.com/wso2/carbon-apimgt/pull/5519/commits/decc193eddecbaccc8eccc22075d2d9876821480
>>
>> In WS APIs, when user send a Header, isn't it going to back-end by
>>> default? Why we need special prefix as we removed it in the outflow?
>>>
>>
>> In Web Socket apis, the headers that we send in the client - gateway
>> handshake are not being sent in the gateway - backend handshake. Only the
>> default headers were set[1] and the incoming headers are set as the
>> properties in axis2 message context. In order to send the header to the
>> backend, we need to get the specific property and attach it as a header to
>> the gateway - backend handshake.
>>
>> As the transport sender implementation should be generic, we send the
>> headers that should be sent to the backend with a prefix and in the
>> WebSocketTransportSender, we get those properties, extract the actual
>> header and set them as handshake headers.[2] So we do not need to alter the
>> transport implementation if we need to send any headers as required.
>>
> Ok. Is there any reason not to send incoming header to the backend? If not
> we ideally should send the headers to the backend. Can't we give a option
> to client to configure the headers that should forward to the backend?
>
>>
>> [1]
>> https://github.com/wso2/carbon-mediation/blob/master/components/carbon-transports/websocket/org.wso2.carbon.websocket.transport/src/main/java/org/wso2/carbon/websocket/transport/WebsocketConnectionFactory.java#L170
>> [2]
>> https://github.com/wso2/carbon-mediation/pull/1068/commits/a3d204dfc53138aab7097d6e168d1c0df7382c01
>>
>> On Fri, Jun 29, 2018 at 7:55 PM, Harsha Kumara  wrote:
>>
>>>
>>>
>>> On Fri, Jun 29, 2018 at 11:25 AM Menaka Jayawardena 
>>> wrote:
>>>
 Hi,


 *With reference to [RRT][APIM] Code Review - Sending Enduser
 information to WS Backends and based on the offline discussion with Kevin.*

 *Initial Requirement:* When the JWT token generation is enabled in API
 Manager, the jwt token should be sent to the Web socket backend.

 *Current Approach:* As the websocket communication happens as frames,
 we could not add the jwt token into the frames. And also it is not a best
 practice as it is a overhead for the message that is being sent.
 So, the token will be attached as a header to the initial web socket
 handshake.

 In the current implementation, we generate the jwt token and  set as an
 intermediate header from the api gateway. This header is then picked up
 from the axis2 message context in the WebSocketTransportSender and attach
 to the Gateway - WS-BackEnd handshake requst.

 But, as per this implementation, if the user needs to send another
 header, the WebSocketTransportSender implementation should be changed to
 support the new header. To avoid this, the implementation will be done in a
 generic manner.

 *Solution:*
 The headers that should be sent to the websocket backends, have to be
 sent with a prefix. The format of would be .

 Ex: If we need to send the header X-JWT-Assertion to the backend, it
 should be sent as *websocket.header.**X-JWT-Assertion*.

 In WebSocketTransportSender, it will get only the properties with the
 *websocket.header.* prefix, extract the header string and attach them
 as new headers to the Handshake request.

>>> In WS APIs, when user send a Header, isn't it going to back-end by
>>> default? Why we need special prefix as we removed it in the outflow?
>>>

 

Re: [Architecture] [APIM] Sending Custom headers to Websocket back ends.

[Harsha Kumara]

On Fri, Jun 29, 2018 at 8:10 PM Menaka Jayawardena  wrote:

> Hi Harsha, Chamin,
>
> Please find my answers inline.
>
> So this means we are having two ways of handling JWT (normal method and
>> WS specific method) scenarios? If so, we will need additional methods to
>> cover this flow. Will there be a code/logic duplication due to this?
>>
>
> No. In this implementation, the same JWT token generation method is used.
> The default ws token validation method is modified to generate the jwt
> token.
>
>
> https://github.com/wso2/carbon-apimgt/pull/5519/commits/decc193eddecbaccc8eccc22075d2d9876821480
>
> In WS APIs, when user send a Header, isn't it going to back-end by
>> default? Why we need special prefix as we removed it in the outflow?
>>
>
> In Web Socket apis, the headers that we send in the client - gateway
> handshake are not being sent in the gateway - backend handshake. Only the
> default headers were set[1] and the incoming headers are set as the
> properties in axis2 message context. In order to send the header to the
> backend, we need to get the specific property and attach it as a header to
> the gateway - backend handshake.
>
> As the transport sender implementation should be generic, we send the
> headers that should be sent to the backend with a prefix and in the
> WebSocketTransportSender, we get those properties, extract the actual
> header and set them as handshake headers.[2] So we do not need to alter the
> transport implementation if we need to send any headers as required.
>
Ok. Is there any reason not to send incoming header to the backend? If not
we ideally should send the headers to the backend. Can't we give a option
to client to configure the headers that should forward to the backend?

>
> [1]
> https://github.com/wso2/carbon-mediation/blob/master/components/carbon-transports/websocket/org.wso2.carbon.websocket.transport/src/main/java/org/wso2/carbon/websocket/transport/WebsocketConnectionFactory.java#L170
> [2]
> https://github.com/wso2/carbon-mediation/pull/1068/commits/a3d204dfc53138aab7097d6e168d1c0df7382c01
>
> On Fri, Jun 29, 2018 at 7:55 PM, Harsha Kumara  wrote:
>
>>
>>
>> On Fri, Jun 29, 2018 at 11:25 AM Menaka Jayawardena 
>> wrote:
>>
>>> Hi,
>>>
>>>
>>> *With reference to [RRT][APIM] Code Review - Sending Enduser information
>>> to WS Backends and based on the offline discussion with Kevin.*
>>>
>>> *Initial Requirement:* When the JWT token generation is enabled in API
>>> Manager, the jwt token should be sent to the Web socket backend.
>>>
>>> *Current Approach:* As the websocket communication happens as frames,
>>> we could not add the jwt token into the frames. And also it is not a best
>>> practice as it is a overhead for the message that is being sent.
>>> So, the token will be attached as a header to the initial web socket
>>> handshake.
>>>
>>> In the current implementation, we generate the jwt token and  set as an
>>> intermediate header from the api gateway. This header is then picked up
>>> from the axis2 message context in the WebSocketTransportSender and attach
>>> to the Gateway - WS-BackEnd handshake requst.
>>>
>>> But, as per this implementation, if the user needs to send another
>>> header, the WebSocketTransportSender implementation should be changed to
>>> support the new header. To avoid this, the implementation will be done in a
>>> generic manner.
>>>
>>> *Solution:*
>>> The headers that should be sent to the websocket backends, have to be
>>> sent with a prefix. The format of would be .
>>>
>>> Ex: If we need to send the header X-JWT-Assertion to the backend, it
>>> should be sent as *websocket.header.**X-JWT-Assertion*.
>>>
>>> In WebSocketTransportSender, it will get only the properties with the
>>> *websocket.header.* prefix, extract the header string and attach them
>>> as new headers to the Handshake request.
>>>
>> In WS APIs, when user send a Header, isn't it going to back-end by
>> default? Why we need special prefix as we removed it in the outflow?
>>
>>>
>>> Any comments, suggestions are highly appreciated.
>>>
>>> Thanks and Regards,
>>> Menaka
>>>
>>> --
>>>
>>> *Menaka Jayawardena*
>>> Senior Software Engineer
>>> WSO2 Inc.
>>>
>>> Phone: +94 71 350 5470
>>> LinkedIn : https://lk.linkedin.com/in/menakajayawardena
>>> Blog   : https://menakamadushanka.wordpress.com/
>>>
>>>
>>
>> --
>> Harsha Kumara
>> Associate Technical Lead, WSO2 Inc.
>> Mobile: +94775505618
>> Blog:harshcreationz.blogspot.com
>>
>
>
>
> --
>
> *Menaka Jayawardena*
> Senior Software Engineer
> WSO2 Inc.
>
> Phone: +94 71 350 5470
> LinkedIn : https://lk.linkedin.com/in/menakajayawardena
> Blog   : https://menakamadushanka.wordpress.com/
>
>

-- 
Harsha Kumara
Associate Technical Lead, WSO2 Inc.
Mobile: +94775505618
Blog:harshcreationz.blogspot.com
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

[Architecture] WSO2 API Manager Micro Gateway Toolkit 2.5.0-Beta2 Released!

[Rukshan Premathunga]

The WSO2 API Manager team is pleased to announce the release of API Manager
Micro Gateway Toolkit 2.5.0-Beta2. It's now available to download.
Distribution

https://github.com/wso2/product-microgateway/releases/download/v2.5.0-beta2/wso2am-micro-gw-toolkit-2.5.0-beta2.zip
Documentation

https://docs.wso2.com/display/AM2xx/Configuring+the+API+Microgateway
Introduction

The Microgateway Toolkit provides the capability to create specialized
gateway distribution (Microgateway distributions) where only a single API or
a group of APIs are included. Once a Microgateway distribution is started,
it will start serving those specific API(s) right away.

In summary, a Microgateway is a specialized form of the WSO2 API Gateway by
having below main characteristics:

   - Its ability to execute in isolation without mandatory connections to
   other components (Key Manager, Traffic Manager, etc).
   - Ability to host a subset of APIs of choice (defined on the API
   Publisher) instead of all.
   - Immutability - if you update an API you need to re-create the
   container/instance, no hot deployment.
   Microgateway offers you a proxy that is capable of performing security
   validations (OAuth, Basic Auth, Signed JWT), in-memory (local) rate
   limiting and operational analytics.

Design Goals

Following are some of its main expectations of Microgateway


   - Ability to host just one or a selected set (subset) of APIs only.
   - Ability to execute in complete isolation once setup, without having
   the need to contact the Management or Security components.
   - Easy integration with CI/CD processes.
   - Seamless integration with deployment automation tools and techniques.

Architecture

The following diagram illustrates the process of getting an API (or a
selected set of APIs) to be hosted on a Microgateway.

[image: Architecture]

Setting up Microgateway Toolkit

This product will include a CLI, the B7a platform distribution and a few
B7a extensions (Endpoints and Filters). The Toolkit will have two main
responsibilities.

   - Setting up a Microgateway project.
   - Running the Microgateway project.

These two steps will be treated as two phases. One will first complete the
"setup" phase and move on to the "build" phase. The reason for treating
them as phases is to make it possible for developers to take control of the
runtime if and when required. For example, what gets run as default on a
Microgateway is a simple API proxy. If a developer needs to perform some
sort of an integration or change the Ballerina source files for some other
reason, he could engage with the project after the setup phase and do the
required modifications before the distribution is built.
Bug Fixes And Improvements in 2.5.0-Beta2

   - GitHub (Product-Microgateway
   

   )

Known Issues

All the open issues pertaining to WSO2 API Manager Microgateway Toolkit are
reported at the following location:

   - GitHub (Product-microgateway
   
   )

How You Can ContributeMailing Lists

Join our mailing list and correspond with the developers directly.

   -

   Developer List: d...@wso2.org | Subscribe | Mail Archive
   -

   User List: u...@wso2.org | Subscribe | Mail Archive

Reporting Issues

We encourage you to report issues, documentation faults, and feature
requests regarding WSO2 API Manager Micro Gateway through the public API
Manager Micro Gateway Git Repo
.
-- The WSO2 API Manager Team --

-- 
Rukshan Chathuranga.
WSO2, Inc.
+94711822074
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [APIM] Sending Custom headers to Websocket back ends.

[Menaka Jayawardena]

Hi Harsha, Chamin,

Please find my answers inline.

So this means we are having two ways of handling JWT (normal method and WS
> specific method) scenarios? If so, we will need additional methods to cover
> this flow. Will there be a code/logic duplication due to this?
>

No. In this implementation, the same JWT token generation method is used.
The default ws token validation method is modified to generate the jwt
token.

https://github.com/wso2/carbon-apimgt/pull/5519/commits/decc193eddecbaccc8eccc22075d2d9876821480

In WS APIs, when user send a Header, isn't it going to back-end by default?
> Why we need special prefix as we removed it in the outflow?
>

In Web Socket apis, the headers that we send in the client - gateway
handshake are not being sent in the gateway - backend handshake. Only the
default headers were set[1] and the incoming headers are set as the
properties in axis2 message context. In order to send the header to the
backend, we need to get the specific property and attach it as a header to
the gateway - backend handshake.

As the transport sender implementation should be generic, we send the
headers that should be sent to the backend with a prefix and in the
WebSocketTransportSender, we get those properties, extract the actual
header and set them as handshake headers.[2] So we do not need to alter the
transport implementation if we need to send any headers as required.

[1]
https://github.com/wso2/carbon-mediation/blob/master/components/carbon-transports/websocket/org.wso2.carbon.websocket.transport/src/main/java/org/wso2/carbon/websocket/transport/WebsocketConnectionFactory.java#L170
[2]
https://github.com/wso2/carbon-mediation/pull/1068/commits/a3d204dfc53138aab7097d6e168d1c0df7382c01

On Fri, Jun 29, 2018 at 7:55 PM, Harsha Kumara  wrote:

>
>
> On Fri, Jun 29, 2018 at 11:25 AM Menaka Jayawardena 
> wrote:
>
>> Hi,
>>
>>
>> *With reference to [RRT][APIM] Code Review - Sending Enduser information
>> to WS Backends and based on the offline discussion with Kevin.*
>>
>> *Initial Requirement:* When the JWT token generation is enabled in API
>> Manager, the jwt token should be sent to the Web socket backend.
>>
>> *Current Approach:* As the websocket communication happens as frames, we
>> could not add the jwt token into the frames. And also it is not a best
>> practice as it is a overhead for the message that is being sent.
>> So, the token will be attached as a header to the initial web socket
>> handshake.
>>
>> In the current implementation, we generate the jwt token and  set as an
>> intermediate header from the api gateway. This header is then picked up
>> from the axis2 message context in the WebSocketTransportSender and attach
>> to the Gateway - WS-BackEnd handshake requst.
>>
>> But, as per this implementation, if the user needs to send another
>> header, the WebSocketTransportSender implementation should be changed to
>> support the new header. To avoid this, the implementation will be done in a
>> generic manner.
>>
>> *Solution:*
>> The headers that should be sent to the websocket backends, have to be
>> sent with a prefix. The format of would be .
>>
>> Ex: If we need to send the header X-JWT-Assertion to the backend, it
>> should be sent as *websocket.header.**X-JWT-Assertion*.
>>
>> In WebSocketTransportSender, it will get only the properties with the
>> *websocket.header.* prefix, extract the header string and attach them as
>> new headers to the Handshake request.
>>
> In WS APIs, when user send a Header, isn't it going to back-end by
> default? Why we need special prefix as we removed it in the outflow?
>
>>
>> Any comments, suggestions are highly appreciated.
>>
>> Thanks and Regards,
>> Menaka
>>
>> --
>>
>> *Menaka Jayawardena*
>> Senior Software Engineer
>> WSO2 Inc.
>>
>> Phone: +94 71 350 5470
>> LinkedIn : https://lk.linkedin.com/in/menakajayawardena
>> Blog   : https://menakamadushanka.wordpress.com/
>>
>>
>
> --
> Harsha Kumara
> Associate Technical Lead, WSO2 Inc.
> Mobile: +94775505618
> Blog:harshcreationz.blogspot.com
>



-- 

*Menaka Jayawardena*
Senior Software Engineer
WSO2 Inc.

Phone: +94 71 350 5470
LinkedIn : https://lk.linkedin.com/in/menakajayawardena
Blog   : https://menakamadushanka.wordpress.com/
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [APIM] Sending Custom headers to Websocket back ends.

[Harsha Kumara]

On Fri, Jun 29, 2018 at 11:25 AM Menaka Jayawardena  wrote:

> Hi,
>
>
> *With reference to [RRT][APIM] Code Review - Sending Enduser information
> to WS Backends and based on the offline discussion with Kevin.*
>
> *Initial Requirement:* When the JWT token generation is enabled in API
> Manager, the jwt token should be sent to the Web socket backend.
>
> *Current Approach:* As the websocket communication happens as frames, we
> could not add the jwt token into the frames. And also it is not a best
> practice as it is a overhead for the message that is being sent.
> So, the token will be attached as a header to the initial web socket
> handshake.
>
> In the current implementation, we generate the jwt token and  set as an
> intermediate header from the api gateway. This header is then picked up
> from the axis2 message context in the WebSocketTransportSender and attach
> to the Gateway - WS-BackEnd handshake requst.
>
> But, as per this implementation, if the user needs to send another header,
> the WebSocketTransportSender implementation should be changed to support
> the new header. To avoid this, the implementation will be done in a generic
> manner.
>
> *Solution:*
> The headers that should be sent to the websocket backends, have to be sent
> with a prefix. The format of would be .
>
> Ex: If we need to send the header X-JWT-Assertion to the backend, it
> should be sent as *websocket.header.**X-JWT-Assertion*.
>
> In WebSocketTransportSender, it will get only the properties with the
> *websocket.header.* prefix, extract the header string and attach them as
> new headers to the Handshake request.
>
In WS APIs, when user send a Header, isn't it going to back-end by default?
Why we need special prefix as we removed it in the outflow?

>
> Any comments, suggestions are highly appreciated.
>
> Thanks and Regards,
> Menaka
>
> --
>
> *Menaka Jayawardena*
> Senior Software Engineer
> WSO2 Inc.
>
> Phone: +94 71 350 5470
> LinkedIn : https://lk.linkedin.com/in/menakajayawardena
> Blog   : https://menakamadushanka.wordpress.com/
>
>

-- 
Harsha Kumara
Associate Technical Lead, WSO2 Inc.
Mobile: +94775505618
Blog:harshcreationz.blogspot.com
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [APIM] Sending Custom headers to Websocket back ends.

[Chamin Dias]

Hi,

On Fri, Jun 29, 2018 at 11:25 AM, Menaka Jayawardena 
wrote:

> Hi,
>
>
> *With reference to [RRT][APIM] Code Review - Sending Enduser information
> to WS Backends and based on the offline discussion with Kevin.*
>
> *Initial Requirement:* When the JWT token generation is enabled in API
> Manager, the jwt token should be sent to the Web socket backend.
>
> *Current Approach:* As the websocket communication happens as frames, we
> could not add the jwt token into the frames. And also it is not a best
> practice as it is a overhead for the message that is being sent.
> So, the token will be attached as a header to the initial web socket
> handshake.
>
> In the current implementation, we generate the jwt token and  set as an
> intermediate header from the api gateway. This header is then picked up
> from the axis2 message context in the WebSocketTransportSender and attach
> to the Gateway - WS-BackEnd handshake requst.
>
> But, as per this implementation, if the user needs to send another header,
> the WebSocketTransportSender implementation should be changed to support
> the new header. To avoid this, the implementation will be done in a generic
> manner.
>
> *Solution:*
> The headers that should be sent to the websocket backends, have to be sent
> with a prefix. The format of would be .
>

So this means we are having two ways of handling JWT (normal method and WS
specific method) scenarios? If so, we will need additional methods to cover
this flow. Will there be a code/logic duplication due to this?


> Ex: If we need to send the header X-JWT-Assertion to the backend, it
> should be sent as *websocket.header.**X-JWT-Assertion*.
>
> In WebSocketTransportSender, it will get only the properties with the
> *websocket.header.* prefix, extract the header string and attach them as
> new headers to the Handshake request.
>
> Any comments, suggestions are highly appreciated.
>
> Thanks and Regards,
> Menaka
>
> --
>
> *Menaka Jayawardena*
> Senior Software Engineer
> WSO2 Inc.
>
> Phone: +94 71 350 5470
> LinkedIn : https://lk.linkedin.com/in/menakajayawardena
> Blog   : https://menakamadushanka.wordpress.com/
>
>


-- 
Chamin Dias
Mobile : 0716097455
Email : cham...@wso2.com
LinkedIn : https://www.linkedin.com/in/chamindias
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [APIM][Micro-Gateway][Analytics] Analytics for Micro-gateway

[Fazlan Nazeem]

Hi Tishan,


On Fri, Jun 29, 2018 at 4:31 PM Tishan Dahanayakage  wrote:

> Fazlan,
>
> On Fri, Jun 29, 2018 at 4:19 PM, Fazlan Nazeem  wrote:
>
>> Hi Tishan,
>> On Fri, Jun 29, 2018 at 4:06 PM Tishan Dahanayakage 
>> wrote:
>>
>>> Fazlan,
>>>
>>> On Fri, Jun 29, 2018 at 3:19 PM, Fazlan Nazeem  wrote:
>>>
 Hi Tishan,
 On Fri, Jun 29, 2018 at 2:43 PM Tishan Dahanayakage 
 wrote:

> Fazlan,
>
> On Fri, Jun 29, 2018 at 2:17 PM, Fazlan Nazeem 
> wrote:
>
>> Hi all,
>>
>> At the moment, analytics for microgateway is supported via a JAX-RS
>> web app and a custom component which are deployed in APIM publisher node.
>> The component was responsible for publishing the analytics data persisted
>> in a DB table to the Analytics server via thrift. As an improvement for
>> this, we have planned to move the web app to Analytics server and process
>> the events within itself which will remove the overhead of publishing 
>> data
>> via thrift. The micro-gateways will then upload the zip files
>> with analytics data directly to the analytics server so that we can
>> eliminate an unnecessary network hop.
>>
>> For this, we have developed a working prototype which follows the
>> following design.
>>
>> [image: micro-analytics.jpg]
>> ​
>> With the above design, a user has to follow the following steps to
>> setup analytics in APIM Analytics server for micro-gateway.
>>
>> 1) Deploy the JAX-RS web app.
>> 2) Deploy the custom event receiver jar file to dropins.
>> 3) Deploy the CAPP with the custom event receivers for required
>> streams.
>> 4) Create a table in STATS_DB to persist the zip file
>> 5) Start analytics server with a set of system properties which will
>> configure the Timer task intervals etc.
>>
> ​Can't we have these as parameters of the custom receiver so that we
> can have them pre-configured offloading tasks from user. Or else set
> reasonable defaults. And I believe timer tasks are started by Custom Event
> Receiver.
>
 Yes. We have set reasonable defaults. By default the receiver will not
 read from DB unless the a specific property is set to true. if it is set to
 true and other parameters are not configured, then the defaults will take
 affect.

>>> ​I don't get this. So receiver will only read from DB if particular
>>> property is set. Who sets this property? If this is user configurable it
>>> will affect HA
>>>
>> All configurations have to be set via Java system properties when
>> starting the server.
>> e.g   -Dusage.data.publishing.enabled=true
>> If this property is not set true or not specified, even if the DB has
>> unprocessed files, the event receiver will not read it from the DB. Only
>> when this is set to true, the events will be read and processed. In HA
>> mode, if all servers are configured to receive events, then the sytem
>> property has to be added to all the servers. Can you elaborate if you see
>> an instance where this will not work?
>>
>
> ​This property will only work if we set "event.duplicated.in.cluster" to
> true and set the system property true in both analytics(DAS) nodes. If you
> are planning to add this property only to one node then ​that node should
> be the active node in HA deployment. But you can't know active node in
> advance.
>
> What is the rationale to add this property. IMO we shouldn't control event
> consumption of a receiver from another path.
>
Ok. I guess you are suggesting that we should remove this configuration.

This config has come into the current design from the earlier
implementation, where the publisher node will only publish the file to
analytics server when this config is set to true.

We will remove this. Thanks for pointing this out.


> /Tishan
>
>>
>>> /Tishan​
>>>
>>>

 Yes the timer tasks are started by the custom receiver component.

>
> /Tishan​
>
>
>>
>> The micro-gateway needs to be configured with the JAX-RS web app's
>> URI so that it can periodically upload files with analytics data to the
>> APIM Analytics server.
>>
>> Any feedback?
>>
>>
>> Thanks & Regards,
>>
>> *Fazlan Nazeem*
>> Senior Software Engineer
>> WSO2 Inc
>> Mobile : +94772338839
>> fazl...@wso2.com
>>
>
>
>
> --
> Tishan Dahanayakage
> Associate Technical Lead
> WSO2, Inc.
> Mobile:+94 716481328
>
> Disclaimer: This communication may contain privileged or other
> confidential information and is intended exclusively for the addressee/s.
> If you are not the intended recipient/s, or believe that you may have
> received this communication in error, please reply to the sender 
> indicating
> that fact and delete the copy you received and in addition, you should not
> print, copy, re-transmit, disseminate, or otherwise use 

Re: [Architecture] [APIM][Micro-Gateway][Analytics] Analytics for Micro-gateway

[Tishan Dahanayakage]

Hi Dinusha,

On Fri, Jun 29, 2018, 4:37 PM Dinusha Dissanayake  wrote:

> Hi Tishan,
>
>>
 One more thing. Can't we just save these zip files to file system
 rather than stressing STATS_DB. We use STATS_DB mainly to store end
 analytics data which is used by presentation layer(Dashboards). WDYT?

>>> This would be problematic in HA deployment. If we keep them in the file
>>> system and if a node goes down, we won't be able to retrieve  the event
>>> data in files in that node.
>>>
>> ​That we can solve by publishing to both DAS nodes from GW. Even earlier
>> I was discussing with Fazlan to avoid adding file to DB by using file tail
>> adaptor but later reverted due to zip files. But given that we are now
>> using custom adaptor we can use files :)
>>
> If we publish to both DAS nodes, then the files would be available in both
> nodes. When event publishing is happening by reading those files, the same
> file will be processed from both the nodes right? :)
> Then the same events will be accumulated twice as I see.
>
No that is handled by HA implementation.
/Tishan

>
>> /Tishan
>>
>>> ​
>>> ​
>>>
>>

>>> /Tishan

 On Fri, Jun 29, 2018 at 2:42 PM, Tishan Dahanayakage 
 wrote:

> Fazlan,
>
> On Fri, Jun 29, 2018 at 2:17 PM, Fazlan Nazeem 
> wrote:
>
>> Hi all,
>>
>> At the moment, analytics for microgateway is supported via a JAX-RS
>> web app and a custom component which are deployed in APIM publisher node.
>> The component was responsible for publishing the analytics data persisted
>> in a DB table to the Analytics server via thrift. As an improvement for
>> this, we have planned to move the web app to Analytics server and process
>> the events within itself which will remove the overhead of publishing 
>> data
>> via thrift. The micro-gateways will then upload the zip files
>> with analytics data directly to the analytics server so that we can
>> eliminate an unnecessary network hop.
>>
>> For this, we have developed a working prototype which follows the
>> following design.
>>
>> [image: micro-analytics.jpg]
>> ​
>> With the above design, a user has to follow the following steps to
>> setup analytics in APIM Analytics server for micro-gateway.
>>
>> 1) Deploy the JAX-RS web app.
>> 2) Deploy the custom event receiver jar file to dropins.
>> 3) Deploy the CAPP with the custom event receivers for required
>> streams.
>> 4) Create a table in STATS_DB to persist the zip file
>> 5) Start analytics server with a set of system properties which will
>> configure the Timer task intervals etc.
>>
> ​Can't we have these as parameters of the custom receiver so that we
> can have them pre-configured offloading tasks from user. Or else set
> reasonable defaults. And I believe timer tasks are started by Custom Event
> Receiver.
>
> /Tishan​
>
>
>>
>> The micro-gateway needs to be configured with the JAX-RS web app's
>> URI so that it can periodically upload files with analytics data to the
>> APIM Analytics server.
>>
>> Any feedback?
>>
>>
>> Thanks & Regards,
>>
>> *Fazlan Nazeem*
>> Senior Software Engineer
>> WSO2 Inc
>> Mobile : +94772338839
>> fazl...@wso2.com
>>
>
>
>
> --
> Tishan Dahanayakage
> Associate Technical Lead
> WSO2, Inc.
> Mobile:+94 716481328
>
> Disclaimer: This communication may contain privileged or other
> confidential information and is intended exclusively for the addressee/s.
> If you are not the intended recipient/s, or believe that you may have
> received this communication in error, please reply to the sender 
> indicating
> that fact and delete the copy you received and in addition, you should not
> print, copy, re-transmit, disseminate, or otherwise use the information
> contained in this communication. Internet communications cannot be
> guaranteed to be timely, secure, error or virus-free. The sender does not
> accept liability for any errors or omissions.
>



 --
 Tishan Dahanayakage
 Associate Technical Lead
 WSO2, Inc.
 Mobile:+94 716481328

 Disclaimer: This communication may contain privileged or other
 confidential information and is intended exclusively for the addressee/s.
 If you are not the intended recipient/s, or believe that you may have
 received this communication in error, please reply to the sender indicating
 that fact and delete the copy you received and in addition, you should not
 print, copy, re-transmit, disseminate, or otherwise use the information
 contained in this communication. Internet communications cannot be
 guaranteed to be timely, secure, error or virus-free. The sender does not
 accept liability for any errors or omissions.

>>>
>>>

Re: [Architecture] [APIM][Micro-Gateway][Analytics] Analytics for Micro-gateway

[Dinusha Dissanayake]

Hi Tishan,

>
>>> One more thing. Can't we just save these zip files to file system rather
>>> than stressing STATS_DB. We use STATS_DB mainly to store end analytics data
>>> which is used by presentation layer(Dashboards). WDYT?
>>>
>> This would be problematic in HA deployment. If we keep them in the file
>> system and if a node goes down, we won't be able to retrieve  the event
>> data in files in that node.
>>
> ​That we can solve by publishing to both DAS nodes from GW. Even earlier I
> was discussing with Fazlan to avoid adding file to DB by using file tail
> adaptor but later reverted due to zip files. But given that we are now
> using custom adaptor we can use files :)
>
If we publish to both DAS nodes, then the files would be available in both
nodes. When event publishing is happening by reading those files, the same
file will be processed from both the nodes right? :)
Then the same events will be accumulated twice as I see.

>
> /Tishan
>
>> ​
>> ​
>>
>
>>>
>> /Tishan
>>>
>>> On Fri, Jun 29, 2018 at 2:42 PM, Tishan Dahanayakage 
>>> wrote:
>>>
 Fazlan,

 On Fri, Jun 29, 2018 at 2:17 PM, Fazlan Nazeem 
 wrote:

> Hi all,
>
> At the moment, analytics for microgateway is supported via a JAX-RS
> web app and a custom component which are deployed in APIM publisher node.
> The component was responsible for publishing the analytics data persisted
> in a DB table to the Analytics server via thrift. As an improvement for
> this, we have planned to move the web app to Analytics server and process
> the events within itself which will remove the overhead of publishing data
> via thrift. The micro-gateways will then upload the zip files
> with analytics data directly to the analytics server so that we can
> eliminate an unnecessary network hop.
>
> For this, we have developed a working prototype which follows the
> following design.
>
> [image: micro-analytics.jpg]
> ​
> With the above design, a user has to follow the following steps to
> setup analytics in APIM Analytics server for micro-gateway.
>
> 1) Deploy the JAX-RS web app.
> 2) Deploy the custom event receiver jar file to dropins.
> 3) Deploy the CAPP with the custom event receivers for required
> streams.
> 4) Create a table in STATS_DB to persist the zip file
> 5) Start analytics server with a set of system properties which will
> configure the Timer task intervals etc.
>
 ​Can't we have these as parameters of the custom receiver so that we
 can have them pre-configured offloading tasks from user. Or else set
 reasonable defaults. And I believe timer tasks are started by Custom Event
 Receiver.

 /Tishan​


>
> The micro-gateway needs to be configured with the JAX-RS web app's URI
> so that it can periodically upload files with analytics data to the APIM
> Analytics server.
>
> Any feedback?
>
>
> Thanks & Regards,
>
> *Fazlan Nazeem*
> Senior Software Engineer
> WSO2 Inc
> Mobile : +94772338839
> fazl...@wso2.com
>



 --
 Tishan Dahanayakage
 Associate Technical Lead
 WSO2, Inc.
 Mobile:+94 716481328

 Disclaimer: This communication may contain privileged or other
 confidential information and is intended exclusively for the addressee/s.
 If you are not the intended recipient/s, or believe that you may have
 received this communication in error, please reply to the sender indicating
 that fact and delete the copy you received and in addition, you should not
 print, copy, re-transmit, disseminate, or otherwise use the information
 contained in this communication. Internet communications cannot be
 guaranteed to be timely, secure, error or virus-free. The sender does not
 accept liability for any errors or omissions.

>>>
>>>
>>>
>>> --
>>> Tishan Dahanayakage
>>> Associate Technical Lead
>>> WSO2, Inc.
>>> Mobile:+94 716481328
>>>
>>> Disclaimer: This communication may contain privileged or other
>>> confidential information and is intended exclusively for the addressee/s.
>>> If you are not the intended recipient/s, or believe that you may have
>>> received this communication in error, please reply to the sender indicating
>>> that fact and delete the copy you received and in addition, you should not
>>> print, copy, re-transmit, disseminate, or otherwise use the information
>>> contained in this communication. Internet communications cannot be
>>> guaranteed to be timely, secure, error or virus-free. The sender does not
>>> accept liability for any errors or omissions.
>>>
>>
>>
>>
>> --
>> Dinusha Dissanayake
>> Software Engineer
>> WSO2 Inc
>> Mobile: +94712939439
>> 
>>
>
>
>
> --
> Tishan Dahanayakage
> Associate Technical Lead
> WSO2, Inc.
> Mobile:+94 716481328
>
> Disclaimer: This communication may contain 

Re: [Architecture] [APIM][Micro-Gateway][Analytics] Analytics for Micro-gateway

[Tishan Dahanayakage]

Fazlan,

On Fri, Jun 29, 2018 at 4:19 PM, Fazlan Nazeem  wrote:

> Hi Tishan,
> On Fri, Jun 29, 2018 at 4:06 PM Tishan Dahanayakage 
> wrote:
>
>> Fazlan,
>>
>> On Fri, Jun 29, 2018 at 3:19 PM, Fazlan Nazeem  wrote:
>>
>>> Hi Tishan,
>>> On Fri, Jun 29, 2018 at 2:43 PM Tishan Dahanayakage 
>>> wrote:
>>>
 Fazlan,

 On Fri, Jun 29, 2018 at 2:17 PM, Fazlan Nazeem 
 wrote:

> Hi all,
>
> At the moment, analytics for microgateway is supported via a JAX-RS
> web app and a custom component which are deployed in APIM publisher node.
> The component was responsible for publishing the analytics data persisted
> in a DB table to the Analytics server via thrift. As an improvement for
> this, we have planned to move the web app to Analytics server and process
> the events within itself which will remove the overhead of publishing data
> via thrift. The micro-gateways will then upload the zip files
> with analytics data directly to the analytics server so that we can
> eliminate an unnecessary network hop.
>
> For this, we have developed a working prototype which follows the
> following design.
>
> [image: micro-analytics.jpg]
> ​
> With the above design, a user has to follow the following steps to
> setup analytics in APIM Analytics server for micro-gateway.
>
> 1) Deploy the JAX-RS web app.
> 2) Deploy the custom event receiver jar file to dropins.
> 3) Deploy the CAPP with the custom event receivers for required
> streams.
> 4) Create a table in STATS_DB to persist the zip file
> 5) Start analytics server with a set of system properties which will
> configure the Timer task intervals etc.
>
 ​Can't we have these as parameters of the custom receiver so that we
 can have them pre-configured offloading tasks from user. Or else set
 reasonable defaults. And I believe timer tasks are started by Custom Event
 Receiver.

>>> Yes. We have set reasonable defaults. By default the receiver will not
>>> read from DB unless the a specific property is set to true. if it is set to
>>> true and other parameters are not configured, then the defaults will take
>>> affect.
>>>
>> ​I don't get this. So receiver will only read from DB if particular
>> property is set. Who sets this property? If this is user configurable it
>> will affect HA
>>
> All configurations have to be set via Java system properties when starting
> the server.
> e.g   -Dusage.data.publishing.enabled=true
> If this property is not set true or not specified, even if the DB has
> unprocessed files, the event receiver will not read it from the DB. Only
> when this is set to true, the events will be read and processed. In HA
> mode, if all servers are configured to receive events, then the sytem
> property has to be added to all the servers. Can you elaborate if you see
> an instance where this will not work?
>

​This property will only work if we set "event.duplicated.in.cluster" to
true and set the system property true in both analytics(DAS) nodes. If you
are planning to add this property only to one node then ​that node should
be the active node in HA deployment. But you can't know active node in
advance.

What is the rationale to add this property. IMO we shouldn't control event
consumption of a receiver from another path.

/Tishan

>
>> /Tishan​
>>
>>
>>>
>>> Yes the timer tasks are started by the custom receiver component.
>>>

 /Tishan​


>
> The micro-gateway needs to be configured with the JAX-RS web app's URI
> so that it can periodically upload files with analytics data to the APIM
> Analytics server.
>
> Any feedback?
>
>
> Thanks & Regards,
>
> *Fazlan Nazeem*
> Senior Software Engineer
> WSO2 Inc
> Mobile : +94772338839
> fazl...@wso2.com
>



 --
 Tishan Dahanayakage
 Associate Technical Lead
 WSO2, Inc.
 Mobile:+94 716481328

 Disclaimer: This communication may contain privileged or other
 confidential information and is intended exclusively for the addressee/s.
 If you are not the intended recipient/s, or believe that you may have
 received this communication in error, please reply to the sender indicating
 that fact and delete the copy you received and in addition, you should not
 print, copy, re-transmit, disseminate, or otherwise use the information
 contained in this communication. Internet communications cannot be
 guaranteed to be timely, secure, error or virus-free. The sender does not
 accept liability for any errors or omissions.

>>>
>>>
>>> --
>>> Thanks & Regards,
>>>
>>> *Fazlan Nazeem*
>>> Senior Software Engineer
>>> WSO2 Inc
>>> Mobile : +94772338839
>>> fazl...@wso2.com
>>>
>>
>>
>>
>> --
>> Tishan Dahanayakage
>> Associate Technical Lead
>> WSO2, Inc.
>> Mobile:+94 716481328
>>
>> Disclaimer: This communication may contain 

Re: [Architecture] [APIM][Micro-Gateway][Analytics] Analytics for Micro-gateway

[Fazlan Nazeem]

Hi Tishan,
On Fri, Jun 29, 2018 at 4:06 PM Tishan Dahanayakage  wrote:

> Fazlan,
>
> On Fri, Jun 29, 2018 at 3:19 PM, Fazlan Nazeem  wrote:
>
>> Hi Tishan,
>> On Fri, Jun 29, 2018 at 2:43 PM Tishan Dahanayakage 
>> wrote:
>>
>>> Fazlan,
>>>
>>> On Fri, Jun 29, 2018 at 2:17 PM, Fazlan Nazeem  wrote:
>>>
 Hi all,

 At the moment, analytics for microgateway is supported via a JAX-RS web
 app and a custom component which are deployed in APIM publisher node. The
 component was responsible for publishing the analytics data persisted in a
 DB table to the Analytics server via thrift. As an improvement for this, we
 have planned to move the web app to Analytics server and process the events
 within itself which will remove the overhead of publishing data via thrift.
 The micro-gateways will then upload the zip files with analytics data
 directly to the analytics server so that we can eliminate an unnecessary
 network hop.

 For this, we have developed a working prototype which follows the
 following design.

 [image: micro-analytics.jpg]
 ​
 With the above design, a user has to follow the following steps to
 setup analytics in APIM Analytics server for micro-gateway.

 1) Deploy the JAX-RS web app.
 2) Deploy the custom event receiver jar file to dropins.
 3) Deploy the CAPP with the custom event receivers for required streams.
 4) Create a table in STATS_DB to persist the zip file
 5) Start analytics server with a set of system properties which will
 configure the Timer task intervals etc.

>>> ​Can't we have these as parameters of the custom receiver so that we can
>>> have them pre-configured offloading tasks from user. Or else set reasonable
>>> defaults. And I believe timer tasks are started by Custom Event Receiver.
>>>
>> Yes. We have set reasonable defaults. By default the receiver will not
>> read from DB unless the a specific property is set to true. if it is set to
>> true and other parameters are not configured, then the defaults will take
>> affect.
>>
> ​I don't get this. So receiver will only read from DB if particular
> property is set. Who sets this property? If this is user configurable it
> will affect HA
>
All configurations have to be set via Java system properties when starting
the server.
e.g   -Dusage.data.publishing.enabled=true
If this property is not set true or not specified, even if the DB has
unprocessed files, the event receiver will not read it from the DB. Only
when this is set to true, the events will be read and processed. In HA
mode, if all servers are configured to receive events, then the sytem
property has to be added to all the servers. Can you elaborate if you see
an instance where this will not work?

>
> /Tishan​
>
>
>>
>> Yes the timer tasks are started by the custom receiver component.
>>
>>>
>>> /Tishan​
>>>
>>>

 The micro-gateway needs to be configured with the JAX-RS web app's URI
 so that it can periodically upload files with analytics data to the APIM
 Analytics server.

 Any feedback?


 Thanks & Regards,

 *Fazlan Nazeem*
 Senior Software Engineer
 WSO2 Inc
 Mobile : +94772338839
 fazl...@wso2.com

>>>
>>>
>>>
>>> --
>>> Tishan Dahanayakage
>>> Associate Technical Lead
>>> WSO2, Inc.
>>> Mobile:+94 716481328
>>>
>>> Disclaimer: This communication may contain privileged or other
>>> confidential information and is intended exclusively for the addressee/s.
>>> If you are not the intended recipient/s, or believe that you may have
>>> received this communication in error, please reply to the sender indicating
>>> that fact and delete the copy you received and in addition, you should not
>>> print, copy, re-transmit, disseminate, or otherwise use the information
>>> contained in this communication. Internet communications cannot be
>>> guaranteed to be timely, secure, error or virus-free. The sender does not
>>> accept liability for any errors or omissions.
>>>
>>
>>
>> --
>> Thanks & Regards,
>>
>> *Fazlan Nazeem*
>> Senior Software Engineer
>> WSO2 Inc
>> Mobile : +94772338839
>> fazl...@wso2.com
>>
>
>
>
> --
> Tishan Dahanayakage
> Associate Technical Lead
> WSO2, Inc.
> Mobile:+94 716481328
>
> Disclaimer: This communication may contain privileged or other
> confidential information and is intended exclusively for the addressee/s.
> If you are not the intended recipient/s, or believe that you may have
> received this communication in error, please reply to the sender indicating
> that fact and delete the copy you received and in addition, you should not
> print, copy, re-transmit, disseminate, or otherwise use the information
> contained in this communication. Internet communications cannot be
> guaranteed to be timely, secure, error or virus-free. The sender does not
> accept liability for any errors or omissions.
>


-- 
Thanks & Regards,

*Fazlan Nazeem*
Senior Software Engineer
WSO2 Inc

Re: [Architecture] [APIM][Micro-Gateway][Analytics] Analytics for Micro-gateway

[Tishan Dahanayakage]

Fazlan,

On Fri, Jun 29, 2018 at 3:19 PM, Fazlan Nazeem  wrote:

> Hi Tishan,
> On Fri, Jun 29, 2018 at 2:43 PM Tishan Dahanayakage 
> wrote:
>
>> Fazlan,
>>
>> On Fri, Jun 29, 2018 at 2:17 PM, Fazlan Nazeem  wrote:
>>
>>> Hi all,
>>>
>>> At the moment, analytics for microgateway is supported via a JAX-RS web
>>> app and a custom component which are deployed in APIM publisher node. The
>>> component was responsible for publishing the analytics data persisted in a
>>> DB table to the Analytics server via thrift. As an improvement for this, we
>>> have planned to move the web app to Analytics server and process the events
>>> within itself which will remove the overhead of publishing data via thrift.
>>> The micro-gateways will then upload the zip files with analytics data
>>> directly to the analytics server so that we can eliminate an unnecessary
>>> network hop.
>>>
>>> For this, we have developed a working prototype which follows the
>>> following design.
>>>
>>> [image: micro-analytics.jpg]
>>> ​
>>> With the above design, a user has to follow the following steps to setup
>>> analytics in APIM Analytics server for micro-gateway.
>>>
>>> 1) Deploy the JAX-RS web app.
>>> 2) Deploy the custom event receiver jar file to dropins.
>>> 3) Deploy the CAPP with the custom event receivers for required streams.
>>> 4) Create a table in STATS_DB to persist the zip file
>>> 5) Start analytics server with a set of system properties which will
>>> configure the Timer task intervals etc.
>>>
>> ​Can't we have these as parameters of the custom receiver so that we can
>> have them pre-configured offloading tasks from user. Or else set reasonable
>> defaults. And I believe timer tasks are started by Custom Event Receiver.
>>
> Yes. We have set reasonable defaults. By default the receiver will not
> read from DB unless the a specific property is set to true. if it is set to
> true and other parameters are not configured, then the defaults will take
> affect.
>
​I don't get this. So receiver will only read from DB if particular
property is set. Who sets this property? If this is user configurable it
will affect HA

/Tishan​


>
> Yes the timer tasks are started by the custom receiver component.
>
>>
>> /Tishan​
>>
>>
>>>
>>> The micro-gateway needs to be configured with the JAX-RS web app's URI
>>> so that it can periodically upload files with analytics data to the APIM
>>> Analytics server.
>>>
>>> Any feedback?
>>>
>>>
>>> Thanks & Regards,
>>>
>>> *Fazlan Nazeem*
>>> Senior Software Engineer
>>> WSO2 Inc
>>> Mobile : +94772338839
>>> fazl...@wso2.com
>>>
>>
>>
>>
>> --
>> Tishan Dahanayakage
>> Associate Technical Lead
>> WSO2, Inc.
>> Mobile:+94 716481328
>>
>> Disclaimer: This communication may contain privileged or other
>> confidential information and is intended exclusively for the addressee/s.
>> If you are not the intended recipient/s, or believe that you may have
>> received this communication in error, please reply to the sender indicating
>> that fact and delete the copy you received and in addition, you should not
>> print, copy, re-transmit, disseminate, or otherwise use the information
>> contained in this communication. Internet communications cannot be
>> guaranteed to be timely, secure, error or virus-free. The sender does not
>> accept liability for any errors or omissions.
>>
>
>
> --
> Thanks & Regards,
>
> *Fazlan Nazeem*
> Senior Software Engineer
> WSO2 Inc
> Mobile : +94772338839
> fazl...@wso2.com
>



-- 
Tishan Dahanayakage
Associate Technical Lead
WSO2, Inc.
Mobile:+94 716481328

Disclaimer: This communication may contain privileged or other confidential
information and is intended exclusively for the addressee/s. If you are not
the intended recipient/s, or believe that you may have received this
communication in error, please reply to the sender indicating that fact and
delete the copy you received and in addition, you should not print, copy,
re-transmit, disseminate, or otherwise use the information contained in
this communication. Internet communications cannot be guaranteed to be
timely, secure, error or virus-free. The sender does not accept liability
for any errors or omissions.
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [APIM][Micro-Gateway][Analytics] Analytics for Micro-gateway

[Tishan Dahanayakage]

Dinusha,

On Fri, Jun 29, 2018 at 3:01 PM, Dinusha Dissanayake 
wrote:

> Hi Tishan,
>
>
>> One more thing. Can't we just save these zip files to file system rather
>> than stressing STATS_DB. We use STATS_DB mainly to store end analytics data
>> which is used by presentation layer(Dashboards). WDYT?
>>
> This would be problematic in HA deployment. If we keep them in the file
> system and if a node goes down, we won't be able to retrieve  the event
> data in files in that node.
>
​That we can solve by publishing to both DAS nodes from GW. Even earlier I
was discussing with Fazlan to avoid adding file to DB by using file tail
adaptor but later reverted due to zip files. But given that we are now
using custom adaptor we can use files :)

/Tishan

> ​
> ​
>

>>
> /Tishan
>>
>> On Fri, Jun 29, 2018 at 2:42 PM, Tishan Dahanayakage 
>> wrote:
>>
>>> Fazlan,
>>>
>>> On Fri, Jun 29, 2018 at 2:17 PM, Fazlan Nazeem  wrote:
>>>
 Hi all,

 At the moment, analytics for microgateway is supported via a JAX-RS web
 app and a custom component which are deployed in APIM publisher node. The
 component was responsible for publishing the analytics data persisted in a
 DB table to the Analytics server via thrift. As an improvement for this, we
 have planned to move the web app to Analytics server and process the events
 within itself which will remove the overhead of publishing data via thrift.
 The micro-gateways will then upload the zip files with analytics data
 directly to the analytics server so that we can eliminate an unnecessary
 network hop.

 For this, we have developed a working prototype which follows the
 following design.

 [image: micro-analytics.jpg]
 ​
 With the above design, a user has to follow the following steps to
 setup analytics in APIM Analytics server for micro-gateway.

 1) Deploy the JAX-RS web app.
 2) Deploy the custom event receiver jar file to dropins.
 3) Deploy the CAPP with the custom event receivers for required streams.
 4) Create a table in STATS_DB to persist the zip file
 5) Start analytics server with a set of system properties which will
 configure the Timer task intervals etc.

>>> ​Can't we have these as parameters of the custom receiver so that we can
>>> have them pre-configured offloading tasks from user. Or else set reasonable
>>> defaults. And I believe timer tasks are started by Custom Event Receiver.
>>>
>>> /Tishan​
>>>
>>>

 The micro-gateway needs to be configured with the JAX-RS web app's URI
 so that it can periodically upload files with analytics data to the APIM
 Analytics server.

 Any feedback?


 Thanks & Regards,

 *Fazlan Nazeem*
 Senior Software Engineer
 WSO2 Inc
 Mobile : +94772338839
 fazl...@wso2.com

>>>
>>>
>>>
>>> --
>>> Tishan Dahanayakage
>>> Associate Technical Lead
>>> WSO2, Inc.
>>> Mobile:+94 716481328
>>>
>>> Disclaimer: This communication may contain privileged or other
>>> confidential information and is intended exclusively for the addressee/s.
>>> If you are not the intended recipient/s, or believe that you may have
>>> received this communication in error, please reply to the sender indicating
>>> that fact and delete the copy you received and in addition, you should not
>>> print, copy, re-transmit, disseminate, or otherwise use the information
>>> contained in this communication. Internet communications cannot be
>>> guaranteed to be timely, secure, error or virus-free. The sender does not
>>> accept liability for any errors or omissions.
>>>
>>
>>
>>
>> --
>> Tishan Dahanayakage
>> Associate Technical Lead
>> WSO2, Inc.
>> Mobile:+94 716481328
>>
>> Disclaimer: This communication may contain privileged or other
>> confidential information and is intended exclusively for the addressee/s.
>> If you are not the intended recipient/s, or believe that you may have
>> received this communication in error, please reply to the sender indicating
>> that fact and delete the copy you received and in addition, you should not
>> print, copy, re-transmit, disseminate, or otherwise use the information
>> contained in this communication. Internet communications cannot be
>> guaranteed to be timely, secure, error or virus-free. The sender does not
>> accept liability for any errors or omissions.
>>
>
>
>
> --
> Dinusha Dissanayake
> Software Engineer
> WSO2 Inc
> Mobile: +94712939439
> 
>



-- 
Tishan Dahanayakage
Associate Technical Lead
WSO2, Inc.
Mobile:+94 716481328

Disclaimer: This communication may contain privileged or other confidential
information and is intended exclusively for the addressee/s. If you are not
the intended recipient/s, or believe that you may have received this
communication in error, please reply to the sender indicating that fact and
delete the copy you received and in addition, you should not print, copy,
re-transmit, disseminate, or 

[Architecture] WSO2 Identity Server 5.7.0-M1 Released!

[Senthalan Kanagalingam]

*WSO2 Identity and Access Management team is pleased to announce the
release of Identity Server 5.7.0 M1!DownloadYou can download WSO2 Identity
Server 5.7.0 M1 from here
.How
to run 1. Extract the downloaded zip file. 2. Go to the bin directory in
the extracted folder. 3. Run the wso2server.sh file if you are on a
Linux/Mac OS or run the wso2server.bat file if you are on a Windows OS.4.
Optionally, if you need to start the OSGi console with the server, use the
-DosgiConsole property when starting the server.What's new in WSO2 Identity
Server 5.7.0 M1A list of all the new features and bug fixes shipped with
this release can be found here
Known IssuesAll
the open issues pertaining to WSO2 Identity Server are reported at the
following location: - IS Runtime
How You Can ContributeMailing
ListsJoin our mailing list and correspond with the developers directly.
Developer list: d...@wso2.org  | Subscribe | Mail Archive
 User forum: StackOverflow
Reporting IssuesWe
encourage you to report issues, improvements, documentation faults, and
feature requests regarding WSO2 Identity Server through WSO2 Identity
Server GIT Issues .For more
information about WSO2 Identity Server, please see
https://wso2.com/identity-and-access-management
 or visit the WSO2 Oxygen
Tank  developer portal for additional resources.~
The WSO2 Identity and Access Management Team ~*

-- 

*Senthalan Kanagalingam*
*Software Engineer - WSO2 Inc.*
*Mobile : +94 (0) 77 18 77 466*

___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [APIM][Micro-Gateway][Analytics] Analytics for Micro-gateway

[Fazlan Nazeem]

Hi Tishan,
On Fri, Jun 29, 2018 at 2:43 PM Tishan Dahanayakage  wrote:

> Fazlan,
>
> On Fri, Jun 29, 2018 at 2:17 PM, Fazlan Nazeem  wrote:
>
>> Hi all,
>>
>> At the moment, analytics for microgateway is supported via a JAX-RS web
>> app and a custom component which are deployed in APIM publisher node. The
>> component was responsible for publishing the analytics data persisted in a
>> DB table to the Analytics server via thrift. As an improvement for this, we
>> have planned to move the web app to Analytics server and process the events
>> within itself which will remove the overhead of publishing data via thrift.
>> The micro-gateways will then upload the zip files with analytics data
>> directly to the analytics server so that we can eliminate an unnecessary
>> network hop.
>>
>> For this, we have developed a working prototype which follows the
>> following design.
>>
>> [image: micro-analytics.jpg]
>> ​
>> With the above design, a user has to follow the following steps to setup
>> analytics in APIM Analytics server for micro-gateway.
>>
>> 1) Deploy the JAX-RS web app.
>> 2) Deploy the custom event receiver jar file to dropins.
>> 3) Deploy the CAPP with the custom event receivers for required streams.
>> 4) Create a table in STATS_DB to persist the zip file
>> 5) Start analytics server with a set of system properties which will
>> configure the Timer task intervals etc.
>>
> ​Can't we have these as parameters of the custom receiver so that we can
> have them pre-configured offloading tasks from user. Or else set reasonable
> defaults. And I believe timer tasks are started by Custom Event Receiver.
>
Yes. We have set reasonable defaults. By default the receiver will not read
from DB unless the a specific property is set to true. if it is set to true
and other parameters are not configured, then the defaults will take
affect.

Yes the timer tasks are started by the custom receiver component.

>
> /Tishan​
>
>
>>
>> The micro-gateway needs to be configured with the JAX-RS web app's URI so
>> that it can periodically upload files with analytics data to the APIM
>> Analytics server.
>>
>> Any feedback?
>>
>>
>> Thanks & Regards,
>>
>> *Fazlan Nazeem*
>> Senior Software Engineer
>> WSO2 Inc
>> Mobile : +94772338839
>> fazl...@wso2.com
>>
>
>
>
> --
> Tishan Dahanayakage
> Associate Technical Lead
> WSO2, Inc.
> Mobile:+94 716481328
>
> Disclaimer: This communication may contain privileged or other
> confidential information and is intended exclusively for the addressee/s.
> If you are not the intended recipient/s, or believe that you may have
> received this communication in error, please reply to the sender indicating
> that fact and delete the copy you received and in addition, you should not
> print, copy, re-transmit, disseminate, or otherwise use the information
> contained in this communication. Internet communications cannot be
> guaranteed to be timely, secure, error or virus-free. The sender does not
> accept liability for any errors or omissions.
>


-- 
Thanks & Regards,

*Fazlan Nazeem*
Senior Software Engineer
WSO2 Inc
Mobile : +94772338839
fazl...@wso2.com
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [APIM][Micro-Gateway][Analytics] Analytics for Micro-gateway

[Fazlan Nazeem]

Hi Rukshan,

On Fri, Jun 29, 2018 at 2:39 PM Rukshan Premathunga 
wrote:

> Hi Fazlan,
>>
>>
>> 2) Deploy the custom event receiver jar file to dropins.
>> 3) Deploy the CAPP with the custom event receivers for required streams.
>
> Why we need custom event receiver and capp here? Did we add a new stream?
>
No we did not add any new stream. The custom event receiver jar is
responsible to read the zip files persisted in the DB and then map it to
the relevant streams. For this to work, each stream (request, response,
execution time) then needs an event receiver file of the custom receiver
type. Therefore we have to deploy a CAPP with those receiver xml files.

> And also when we persisting on statdb, we save it as a zip or events?
>
We persist as a zip file.

>
> Thanks and Regards
>
>
> On Fri, Jun 29, 2018 at 2:33 PM, Fazlan Nazeem  wrote:
>
>> Hi Dinusha,
>>
>> The system properties will configure the following.
>>
>> 1) Whether the receivers should read data from the DB
>> 2) The time interval to read the unprocessed zip files from the DB
>> 3) File retention days for already processed files
>> 4) The time interval to run the cleanup task has been set to 1 hour for
>> now. I am planning to move this also to a system property.
>>
>> On Fri, Jun 29, 2018 at 2:27 PM Dinusha Dissanayake 
>> wrote:
>>
>>> Hi Fazlan,
>>>
>>> Please find my concerns below.
>>>

 ​
 With the above design, a user has to follow the following steps to
 setup analytics in APIM Analytics server for micro-gateway.

 1) Deploy the JAX-RS web app.
 2) Deploy the custom event receiver jar file to dropins.
 3) Deploy the CAPP with the custom event receivers for required streams.
 4) Create a table in STATS_DB to persist the zip file
 5) Start analytics server with a set of system properties which will
 configure the Timer task intervals etc.

>>> I believe we have to provide data publishing time interval and clean up
>>> tasks interval here right? Will there be any additional fields?
>>>

 The micro-gateway needs to be configured with the JAX-RS web app's URI
 so that it can periodically upload files with analytics data to the APIM
 Analytics server.

 Any feedback?


 Thanks & Regards,

 *Fazlan Nazeem*
 Senior Software Engineer
 WSO2 Inc
 Mobile : +94772338839
 fazl...@wso2.com

>>>
>>>
>>>
>>> --
>>> Dinusha Dissanayake
>>> Software Engineer
>>> WSO2 Inc
>>> Mobile: +94712939439
>>> 
>>>
>>
>>
>> --
>> Thanks & Regards,
>>
>> *Fazlan Nazeem*
>> Senior Software Engineer
>> WSO2 Inc
>> Mobile : +94772338839
>> fazl...@wso2.com
>>
>
>
>
> --
> Rukshan Chathuranga.
> Software Engineer.
> WSO2, Inc.
> +94711822074
>


-- 
Thanks & Regards,

*Fazlan Nazeem*
Senior Software Engineer
WSO2 Inc
Mobile : +94772338839
fazl...@wso2.com
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [APIM][Micro-Gateway][Analytics] Analytics for Micro-gateway

[Tishan Dahanayakage]

Hi Fazlan,

One more thing. Can't we just save these zip files to file system rather
than stressing STATS_DB. We use STATS_DB mainly to store end analytics data
which is used by presentation layer(Dashboards). WDYT?

/Tishan

On Fri, Jun 29, 2018 at 2:42 PM, Tishan Dahanayakage 
wrote:

> Fazlan,
>
> On Fri, Jun 29, 2018 at 2:17 PM, Fazlan Nazeem  wrote:
>
>> Hi all,
>>
>> At the moment, analytics for microgateway is supported via a JAX-RS web
>> app and a custom component which are deployed in APIM publisher node. The
>> component was responsible for publishing the analytics data persisted in a
>> DB table to the Analytics server via thrift. As an improvement for this, we
>> have planned to move the web app to Analytics server and process the events
>> within itself which will remove the overhead of publishing data via thrift.
>> The micro-gateways will then upload the zip files with analytics data
>> directly to the analytics server so that we can eliminate an unnecessary
>> network hop.
>>
>> For this, we have developed a working prototype which follows the
>> following design.
>>
>> [image: micro-analytics.jpg]
>> ​
>> With the above design, a user has to follow the following steps to setup
>> analytics in APIM Analytics server for micro-gateway.
>>
>> 1) Deploy the JAX-RS web app.
>> 2) Deploy the custom event receiver jar file to dropins.
>> 3) Deploy the CAPP with the custom event receivers for required streams.
>> 4) Create a table in STATS_DB to persist the zip file
>> 5) Start analytics server with a set of system properties which will
>> configure the Timer task intervals etc.
>>
> ​Can't we have these as parameters of the custom receiver so that we can
> have them pre-configured offloading tasks from user. Or else set reasonable
> defaults. And I believe timer tasks are started by Custom Event Receiver.
>
> /Tishan​
>
>
>>
>> The micro-gateway needs to be configured with the JAX-RS web app's URI so
>> that it can periodically upload files with analytics data to the APIM
>> Analytics server.
>>
>> Any feedback?
>>
>>
>> Thanks & Regards,
>>
>> *Fazlan Nazeem*
>> Senior Software Engineer
>> WSO2 Inc
>> Mobile : +94772338839
>> fazl...@wso2.com
>>
>
>
>
> --
> Tishan Dahanayakage
> Associate Technical Lead
> WSO2, Inc.
> Mobile:+94 716481328
>
> Disclaimer: This communication may contain privileged or other
> confidential information and is intended exclusively for the addressee/s.
> If you are not the intended recipient/s, or believe that you may have
> received this communication in error, please reply to the sender indicating
> that fact and delete the copy you received and in addition, you should not
> print, copy, re-transmit, disseminate, or otherwise use the information
> contained in this communication. Internet communications cannot be
> guaranteed to be timely, secure, error or virus-free. The sender does not
> accept liability for any errors or omissions.
>



-- 
Tishan Dahanayakage
Associate Technical Lead
WSO2, Inc.
Mobile:+94 716481328

Disclaimer: This communication may contain privileged or other confidential
information and is intended exclusively for the addressee/s. If you are not
the intended recipient/s, or believe that you may have received this
communication in error, please reply to the sender indicating that fact and
delete the copy you received and in addition, you should not print, copy,
re-transmit, disseminate, or otherwise use the information contained in
this communication. Internet communications cannot be guaranteed to be
timely, secure, error or virus-free. The sender does not accept liability
for any errors or omissions.
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [APIM][Micro-Gateway][Analytics] Analytics for Micro-gateway

[Tishan Dahanayakage]

Fazlan,

On Fri, Jun 29, 2018 at 2:17 PM, Fazlan Nazeem  wrote:

> Hi all,
>
> At the moment, analytics for microgateway is supported via a JAX-RS web
> app and a custom component which are deployed in APIM publisher node. The
> component was responsible for publishing the analytics data persisted in a
> DB table to the Analytics server via thrift. As an improvement for this, we
> have planned to move the web app to Analytics server and process the events
> within itself which will remove the overhead of publishing data via thrift.
> The micro-gateways will then upload the zip files with analytics data
> directly to the analytics server so that we can eliminate an unnecessary
> network hop.
>
> For this, we have developed a working prototype which follows the
> following design.
>
> [image: micro-analytics.jpg]
> ​
> With the above design, a user has to follow the following steps to setup
> analytics in APIM Analytics server for micro-gateway.
>
> 1) Deploy the JAX-RS web app.
> 2) Deploy the custom event receiver jar file to dropins.
> 3) Deploy the CAPP with the custom event receivers for required streams.
> 4) Create a table in STATS_DB to persist the zip file
> 5) Start analytics server with a set of system properties which will
> configure the Timer task intervals etc.
>
​Can't we have these as parameters of the custom receiver so that we can
have them pre-configured offloading tasks from user. Or else set reasonable
defaults. And I believe timer tasks are started by Custom Event Receiver.

/Tishan​


>
> The micro-gateway needs to be configured with the JAX-RS web app's URI so
> that it can periodically upload files with analytics data to the APIM
> Analytics server.
>
> Any feedback?
>
>
> Thanks & Regards,
>
> *Fazlan Nazeem*
> Senior Software Engineer
> WSO2 Inc
> Mobile : +94772338839
> fazl...@wso2.com
>



-- 
Tishan Dahanayakage
Associate Technical Lead
WSO2, Inc.
Mobile:+94 716481328

Disclaimer: This communication may contain privileged or other confidential
information and is intended exclusively for the addressee/s. If you are not
the intended recipient/s, or believe that you may have received this
communication in error, please reply to the sender indicating that fact and
delete the copy you received and in addition, you should not print, copy,
re-transmit, disseminate, or otherwise use the information contained in
this communication. Internet communications cannot be guaranteed to be
timely, secure, error or virus-free. The sender does not accept liability
for any errors or omissions.
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [APIM][Micro-Gateway][Analytics] Analytics for Micro-gateway

[Rukshan Premathunga]

Hi Fazlan,
>
>
> 2) Deploy the custom event receiver jar file to dropins.
> 3) Deploy the CAPP with the custom event receivers for required streams.

Why we need custom event receiver and capp here? Did we add a new stream?

And also when we persisting on statdb, we save it as a zip or events?

Thanks and Regards


On Fri, Jun 29, 2018 at 2:33 PM, Fazlan Nazeem  wrote:

> Hi Dinusha,
>
> The system properties will configure the following.
>
> 1) Whether the receivers should read data from the DB
> 2) The time interval to read the unprocessed zip files from the DB
> 3) File retention days for already processed files
> 4) The time interval to run the cleanup task has been set to 1 hour for
> now. I am planning to move this also to a system property.
>
> On Fri, Jun 29, 2018 at 2:27 PM Dinusha Dissanayake 
> wrote:
>
>> Hi Fazlan,
>>
>> Please find my concerns below.
>>
>>>
>>> ​
>>> With the above design, a user has to follow the following steps to setup
>>> analytics in APIM Analytics server for micro-gateway.
>>>
>>> 1) Deploy the JAX-RS web app.
>>> 2) Deploy the custom event receiver jar file to dropins.
>>> 3) Deploy the CAPP with the custom event receivers for required streams.
>>> 4) Create a table in STATS_DB to persist the zip file
>>> 5) Start analytics server with a set of system properties which will
>>> configure the Timer task intervals etc.
>>>
>> I believe we have to provide data publishing time interval and clean up
>> tasks interval here right? Will there be any additional fields?
>>
>>>
>>> The micro-gateway needs to be configured with the JAX-RS web app's URI
>>> so that it can periodically upload files with analytics data to the APIM
>>> Analytics server.
>>>
>>> Any feedback?
>>>
>>>
>>> Thanks & Regards,
>>>
>>> *Fazlan Nazeem*
>>> Senior Software Engineer
>>> WSO2 Inc
>>> Mobile : +94772338839
>>> fazl...@wso2.com
>>>
>>
>>
>>
>> --
>> Dinusha Dissanayake
>> Software Engineer
>> WSO2 Inc
>> Mobile: +94712939439
>> 
>>
>
>
> --
> Thanks & Regards,
>
> *Fazlan Nazeem*
> Senior Software Engineer
> WSO2 Inc
> Mobile : +94772338839
> fazl...@wso2.com
>



-- 
Rukshan Chathuranga.
Software Engineer.
WSO2, Inc.
+94711822074
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [APIM][Micro-Gateway][Analytics] Analytics for Micro-gateway

[Dinusha Dissanayake]

Hi Fazlan,

Please find my concerns below.

>
> ​
> With the above design, a user has to follow the following steps to setup
> analytics in APIM Analytics server for micro-gateway.
>
> 1) Deploy the JAX-RS web app.
> 2) Deploy the custom event receiver jar file to dropins.
> 3) Deploy the CAPP with the custom event receivers for required streams.
> 4) Create a table in STATS_DB to persist the zip file
> 5) Start analytics server with a set of system properties which will
> configure the Timer task intervals etc.
>
I believe we have to provide data publishing time interval and clean up
tasks interval here right? Will there be any additional fields?

>
> The micro-gateway needs to be configured with the JAX-RS web app's URI so
> that it can periodically upload files with analytics data to the APIM
> Analytics server.
>
> Any feedback?
>
>
> Thanks & Regards,
>
> *Fazlan Nazeem*
> Senior Software Engineer
> WSO2 Inc
> Mobile : +94772338839
> fazl...@wso2.com
>



-- 
Dinusha Dissanayake
Software Engineer
WSO2 Inc
Mobile: +94712939439

___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

[Architecture] [APIM][Micro-Gateway][Analytics] Analytics for Micro-gateway

[Fazlan Nazeem]

Hi all,

At the moment, analytics for microgateway is supported via a JAX-RS web app
and a custom component which are deployed in APIM publisher node. The
component was responsible for publishing the analytics data persisted in a
DB table to the Analytics server via thrift. As an improvement for this, we
have planned to move the web app to Analytics server and process the events
within itself which will remove the overhead of publishing data via thrift.
The micro-gateways will then upload the zip files with analytics data
directly to the analytics server so that we can eliminate an unnecessary
network hop.

For this, we have developed a working prototype which follows the following
design.

[image: micro-analytics.jpg]
​
With the above design, a user has to follow the following steps to setup
analytics in APIM Analytics server for micro-gateway.

1) Deploy the JAX-RS web app.
2) Deploy the custom event receiver jar file to dropins.
3) Deploy the CAPP with the custom event receivers for required streams.
4) Create a table in STATS_DB to persist the zip file
5) Start analytics server with a set of system properties which will
configure the Timer task intervals etc.

The micro-gateway needs to be configured with the JAX-RS web app's URI so
that it can periodically upload files with analytics data to the APIM
Analytics server.

Any feedback?


Thanks & Regards,

*Fazlan Nazeem*
Senior Software Engineer
WSO2 Inc
Mobile : +94772338839
fazl...@wso2.com
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture