Re: [Architecture] [IS] Printing Correlation ID on every Request on the logs.

[Nuwan Dias]

How does this particular feature differ from the Message Tracing feature
that is already available in Carbon? Please see blog [1]. Also doesn't IS
call out to any services in any case?

How do we make sure that we can trace the message from APIM for example?

[1] -
https://dinushasblog.blogspot.com/2013/12/message-tracing-with-wso2-api-manager.html

On Wed, Sep 5, 2018 at 10:12 AM Dulanja Liyanage  wrote:

>
>
> On Wed, Sep 5, 2018 at 9:45 AM, Ruwan Linton  wrote:
>
>> +1 for the idea in general, what we need to do is to insert this ID as a
>> MDC value and configure the logging appender pattern to include that ID
>> into the logs printed into the file/console.
>>
>> We are already working on something similar to this for tracing
>> transactions across the products, for that we are using a correlation id
>> (transaction id) between products.
>>
>> This effort has to work seamlessly with that too, in the case of a
>> request that will be initiated by APIM, there will be a correlation id
>> coming in, in the request (as a HTTP header) so the ID generation logic
>> should look at this header, and generate a new ID only if that is not
>> available.
>>
>
> +1. Yes, a new ID should be generated only if an existing one is not
> available.
>
>>
>> Thanks,
>> Ruwan
>>
>> On Tue, Sep 4, 2018 at 6:38 PM Dulanja Liyanage  wrote:
>>
>>> How can we map a particular request between the wso2carbon.log and the
>>> HTTP access log?
>>>
>>> IMO, we should generate this Correlation ID from a custom valve and add
>>> that as a request attribute, and then extend the Access Log valve to print
>>> that ID in the HTTP Access Log as well. (I haven't tried this, but I
>>> believe this approach is feasible)
>>>
>>> Also, by doing this way you don't have to individually add the filter to
>>> each webapp in the product, because valves get engaged for all the contexts.
>>>
>>> WDYT?
>>>
>>> Thanks,
>>> Dulanja
>>>
>>> On Mon, Aug 20, 2018 at 4:53 PM, Thumilan Mikunthan 
>>> wrote:
>>>
 Hi All,
 *Problem*

 Whenever an error occurred, analyzing error log line could not give
 enough information to diagnose the error. A mechanism to track each request
 flow  is helpful to diagnose.

 *Solution*
 Assign unique id to each request and track it down through log files.

-

Use a Request filter to filter all requests which come to WSO2 IS.
-

Assign a unique id for each request.
-

Print the unique id along with log lines in order to diagnose
errors.

 *Solution Diagram*
 [image: UUID Archi.jpg]


 *Sample Scenario*
 [1] - Sample Scenario (Correlation ID-sample.jpg)
 We can easily understand that above error occurred because of illegal
 access by tracking the unique id.

 Please give feedback regarding this architecture.

 Best Regards,
 M.Thumilan

>>>
>>>
>>>
>>> --
>>> Thanks & Regards,
>>> Dulanja Liyanage
>>> Lead, Platform Security Team
>>> WSO2 Inc.
>>>
>>
>>
>> --
>> Ruwan Linton
>> Director - Delivery, WSO2; https://wso2.com
>> Member, Apache Software Foundation; http://www.apache.org
>> 
>>
>> email: ru...@wso2.com; cell: +94 77 341 3097; phone: +94 11 2833 436
>> linkedin: http://www.linkedin.com/in/ruwanlinton
>>
>
>
>
> --
> Thanks & Regards,
> Dulanja Liyanage
> Lead, Platform Security Team
> WSO2 Inc.
>


-- 
Nuwan Dias

Director - WSO2, Inc. http://wso2.com
email : nuw...@wso2.com
Phone : +94 777 775 729
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [IS] Printing Correlation ID on every Request on the logs.

[Dulanja Liyanage]

On Wed, Sep 5, 2018 at 9:49 AM, Thumilan Mikunthan 
wrote:

> -1 for "Also, by doing this way you don't have to individually add the
> filter to each webapp in the product, because valves get engaged for all
> the contexts."
>   Instead of adding individual flters, common custom filter which
> can filter all requests able to do the job.
>
> +1 for print Correlation ID in HTTP access log.
>
> Full workflow
>
>-
>
>A custom servlet filter checks all request headers whether they have
>‘coorelation-ID’ or not.(need to change tomcat’s web.xml file)
>-
>
>   If not create a unique ID and add it to the request header.
>
> I have a vague memory that filters in tomcat's global web.xml didn't
effect the Jaggery apps. Better to check that.

>
>-
>   -
>-
>
>Use MDC put to log the corrrelation ID.(need to modify
>log4j.properties file)
>
> Dushan already finished the project. [1] - project repo.
>
> [1]- https://github.com/dushansilva/timing-log
>
>
> On Tue, Sep 4, 2018 at 6:38 PM Dulanja Liyanage  wrote:
>
>> How can we map a particular request between the wso2carbon.log and the
>> HTTP access log?
>>
>> IMO, we should generate this Correlation ID from a custom valve and add
>> that as a request attribute, and then extend the Access Log valve to print
>> that ID in the HTTP Access Log as well. (I haven't tried this, but I
>> believe this approach is feasible)
>>
>> Also, by doing this way you don't have to individually add the filter to
>> each webapp in the product, because valves get engaged for all the contexts.
>>
>> WDYT?
>>
>> Thanks,
>> Dulanja
>>
>> On Mon, Aug 20, 2018 at 4:53 PM, Thumilan Mikunthan 
>> wrote:
>>
>>> Hi All,
>>> *Problem*
>>>
>>> Whenever an error occurred, analyzing error log line could not give
>>> enough information to diagnose the error. A mechanism to track each request
>>> flow  is helpful to diagnose.
>>>
>>> *Solution*
>>> Assign unique id to each request and track it down through log files.
>>>
>>>-
>>>
>>>Use a Request filter to filter all requests which come to WSO2 IS.
>>>-
>>>
>>>Assign a unique id for each request.
>>>-
>>>
>>>Print the unique id along with log lines in order to diagnose errors.
>>>
>>> *Solution Diagram*
>>> [image: UUID Archi.jpg]
>>>
>>>
>>> *Sample Scenario*
>>> [1] - Sample Scenario (Correlation ID-sample.jpg)
>>> We can easily understand that above error occurred because of illegal
>>> access by tracking the unique id.
>>>
>>> Please give feedback regarding this architecture.
>>>
>>> Best Regards,
>>> M.Thumilan
>>>
>>
>>
>>
>> --
>> Thanks & Regards,
>> Dulanja Liyanage
>> Lead, Platform Security Team
>> WSO2 Inc.
>>
>
>
> --
> Best Regards,
> M.Thumilan
>



-- 
Thanks & Regards,
Dulanja Liyanage
Lead, Platform Security Team
WSO2 Inc.
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [IS] Printing Correlation ID on every Request on the logs.

[Dulanja Liyanage]

On Wed, Sep 5, 2018 at 9:45 AM, Ruwan Linton  wrote:

> +1 for the idea in general, what we need to do is to insert this ID as a
> MDC value and configure the logging appender pattern to include that ID
> into the logs printed into the file/console.
>
> We are already working on something similar to this for tracing
> transactions across the products, for that we are using a correlation id
> (transaction id) between products.
>
> This effort has to work seamlessly with that too, in the case of a request
> that will be initiated by APIM, there will be a correlation id coming in,
> in the request (as a HTTP header) so the ID generation logic should look at
> this header, and generate a new ID only if that is not available.
>

+1. Yes, a new ID should be generated only if an existing one is not
available.

>
> Thanks,
> Ruwan
>
> On Tue, Sep 4, 2018 at 6:38 PM Dulanja Liyanage  wrote:
>
>> How can we map a particular request between the wso2carbon.log and the
>> HTTP access log?
>>
>> IMO, we should generate this Correlation ID from a custom valve and add
>> that as a request attribute, and then extend the Access Log valve to print
>> that ID in the HTTP Access Log as well. (I haven't tried this, but I
>> believe this approach is feasible)
>>
>> Also, by doing this way you don't have to individually add the filter to
>> each webapp in the product, because valves get engaged for all the contexts.
>>
>> WDYT?
>>
>> Thanks,
>> Dulanja
>>
>> On Mon, Aug 20, 2018 at 4:53 PM, Thumilan Mikunthan 
>> wrote:
>>
>>> Hi All,
>>> *Problem*
>>>
>>> Whenever an error occurred, analyzing error log line could not give
>>> enough information to diagnose the error. A mechanism to track each request
>>> flow  is helpful to diagnose.
>>>
>>> *Solution*
>>> Assign unique id to each request and track it down through log files.
>>>
>>>-
>>>
>>>Use a Request filter to filter all requests which come to WSO2 IS.
>>>-
>>>
>>>Assign a unique id for each request.
>>>-
>>>
>>>Print the unique id along with log lines in order to diagnose errors.
>>>
>>> *Solution Diagram*
>>> [image: UUID Archi.jpg]
>>>
>>>
>>> *Sample Scenario*
>>> [1] - Sample Scenario (Correlation ID-sample.jpg)
>>> We can easily understand that above error occurred because of illegal
>>> access by tracking the unique id.
>>>
>>> Please give feedback regarding this architecture.
>>>
>>> Best Regards,
>>> M.Thumilan
>>>
>>
>>
>>
>> --
>> Thanks & Regards,
>> Dulanja Liyanage
>> Lead, Platform Security Team
>> WSO2 Inc.
>>
>
>
> --
> Ruwan Linton
> Director - Delivery, WSO2; https://wso2.com
> Member, Apache Software Foundation; http://www.apache.org
> 
>
> email: ru...@wso2.com; cell: +94 77 341 3097; phone: +94 11 2833 436
> linkedin: http://www.linkedin.com/in/ruwanlinton
>



-- 
Thanks & Regards,
Dulanja Liyanage
Lead, Platform Security Team
WSO2 Inc.
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [IS] Printing Correlation ID on every Request on the logs.

[Thumilan Mikunthan]

-1 for "Also, by doing this way you don't have to individually add the
filter to each webapp in the product, because valves get engaged for all
the contexts."
  Instead of adding individual flters, common custom filter which
can filter all requests able to do the job.

+1 for print Correlation ID in HTTP access log.

Full workflow

   -

   A custom servlet filter checks all request headers whether they have
   ‘coorelation-ID’ or not.(need to change tomcat’s web.xml file)
   -

  If not create a unique ID and add it to the request header.
  -

   Use MDC put to log the corrrelation ID.(need to modify log4j.properties
   file)

Dushan already finished the project. [1] - project repo.

[1]- https://github.com/dushansilva/timing-log


On Tue, Sep 4, 2018 at 6:38 PM Dulanja Liyanage  wrote:

> How can we map a particular request between the wso2carbon.log and the
> HTTP access log?
>
> IMO, we should generate this Correlation ID from a custom valve and add
> that as a request attribute, and then extend the Access Log valve to print
> that ID in the HTTP Access Log as well. (I haven't tried this, but I
> believe this approach is feasible)
>
> Also, by doing this way you don't have to individually add the filter to
> each webapp in the product, because valves get engaged for all the contexts.
>
> WDYT?
>
> Thanks,
> Dulanja
>
> On Mon, Aug 20, 2018 at 4:53 PM, Thumilan Mikunthan 
> wrote:
>
>> Hi All,
>> *Problem*
>>
>> Whenever an error occurred, analyzing error log line could not give
>> enough information to diagnose the error. A mechanism to track each request
>> flow  is helpful to diagnose.
>>
>> *Solution*
>> Assign unique id to each request and track it down through log files.
>>
>>-
>>
>>Use a Request filter to filter all requests which come to WSO2 IS.
>>-
>>
>>Assign a unique id for each request.
>>-
>>
>>Print the unique id along with log lines in order to diagnose errors.
>>
>> *Solution Diagram*
>> [image: UUID Archi.jpg]
>>
>>
>> *Sample Scenario*
>> [1] - Sample Scenario (Correlation ID-sample.jpg)
>> We can easily understand that above error occurred because of illegal
>> access by tracking the unique id.
>>
>> Please give feedback regarding this architecture.
>>
>> Best Regards,
>> M.Thumilan
>>
>
>
>
> --
> Thanks & Regards,
> Dulanja Liyanage
> Lead, Platform Security Team
> WSO2 Inc.
>


-- 
Best Regards,
M.Thumilan
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [IS] Printing Correlation ID on every Request on the logs.

[Ruwan Linton]

+1 for the idea in general, what we need to do is to insert this ID as a
MDC value and configure the logging appender pattern to include that ID
into the logs printed into the file/console.

We are already working on something similar to this for tracing
transactions across the products, for that we are using a correlation id
(transaction id) between products.

This effort has to work seamlessly with that too, in the case of a request
that will be initiated by APIM, there will be a correlation id coming in,
in the request (as a HTTP header) so the ID generation logic should look at
this header, and generate a new ID only if that is not available.

Thanks,
Ruwan

On Tue, Sep 4, 2018 at 6:38 PM Dulanja Liyanage  wrote:

> How can we map a particular request between the wso2carbon.log and the
> HTTP access log?
>
> IMO, we should generate this Correlation ID from a custom valve and add
> that as a request attribute, and then extend the Access Log valve to print
> that ID in the HTTP Access Log as well. (I haven't tried this, but I
> believe this approach is feasible)
>
> Also, by doing this way you don't have to individually add the filter to
> each webapp in the product, because valves get engaged for all the contexts.
>
> WDYT?
>
> Thanks,
> Dulanja
>
> On Mon, Aug 20, 2018 at 4:53 PM, Thumilan Mikunthan 
> wrote:
>
>> Hi All,
>> *Problem*
>>
>> Whenever an error occurred, analyzing error log line could not give
>> enough information to diagnose the error. A mechanism to track each request
>> flow  is helpful to diagnose.
>>
>> *Solution*
>> Assign unique id to each request and track it down through log files.
>>
>>-
>>
>>Use a Request filter to filter all requests which come to WSO2 IS.
>>-
>>
>>Assign a unique id for each request.
>>-
>>
>>Print the unique id along with log lines in order to diagnose errors.
>>
>> *Solution Diagram*
>> [image: UUID Archi.jpg]
>>
>>
>> *Sample Scenario*
>> [1] - Sample Scenario (Correlation ID-sample.jpg)
>> We can easily understand that above error occurred because of illegal
>> access by tracking the unique id.
>>
>> Please give feedback regarding this architecture.
>>
>> Best Regards,
>> M.Thumilan
>>
>
>
>
> --
> Thanks & Regards,
> Dulanja Liyanage
> Lead, Platform Security Team
> WSO2 Inc.
>


-- 
Ruwan Linton
Director - Delivery, WSO2; https://wso2.com
Member, Apache Software Foundation; http://www.apache.org


email: ru...@wso2.com; cell: +94 77 341 3097; phone: +94 11 2833 436
linkedin: http://www.linkedin.com/in/ruwanlinton
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [IS] Printing Correlation ID on every Request on the logs.

[Dulanja Liyanage]

How can we map a particular request between the wso2carbon.log and the HTTP
access log?

IMO, we should generate this Correlation ID from a custom valve and add
that as a request attribute, and then extend the Access Log valve to print
that ID in the HTTP Access Log as well. (I haven't tried this, but I
believe this approach is feasible)

Also, by doing this way you don't have to individually add the filter to
each webapp in the product, because valves get engaged for all the contexts.

WDYT?

Thanks,
Dulanja

On Mon, Aug 20, 2018 at 4:53 PM, Thumilan Mikunthan 
wrote:

> Hi All,
> *Problem*
>
> Whenever an error occurred, analyzing error log line could not give enough
> information to diagnose the error. A mechanism to track each request flow
> is helpful to diagnose.
>
> *Solution*
> Assign unique id to each request and track it down through log files.
>
>-
>
>Use a Request filter to filter all requests which come to WSO2 IS.
>-
>
>Assign a unique id for each request.
>-
>
>Print the unique id along with log lines in order to diagnose errors.
>
> *Solution Diagram*
> [image: UUID Archi.jpg]
>
>
> *Sample Scenario*
> [1] - Sample Scenario (Correlation ID-sample.jpg)
> We can easily understand that above error occurred because of illegal
> access by tracking the unique id.
>
> Please give feedback regarding this architecture.
>
> Best Regards,
> M.Thumilan
>



-- 
Thanks & Regards,
Dulanja Liyanage
Lead, Platform Security Team
WSO2 Inc.
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture