Re: [Architecture] BAM Log monitoring for Cloud
Hi all, Thank you for your suggestions on improvement. I ll update you on the progress Thanks, On Thu, Sep 25, 2014 at 4:38 PM, Thomas Wieger developer.wie...@gmail.com wrote: Am Donnerstag, 25. September 2014 schrieb Manjula Rathnayake : Hi Gimantha, If we can correlate log event based on timestamp range from all services(AS,BPS,AF,etc) that is really useful when identifying issues. I would recommend that you take a look on the Google Dapper paper http://research.google.com/pubs/pub36356.html regarding distributed tracing. I think this would perfectly solve all correlation issues. For implementation you should have a look on Twitters Zipkin http://twitter.github.io/zipkin/ or Brave https://github.com/kristofa/brave. regards, Thomas ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- Gimantha Bandara Software Engineer WSO2. Inc : http://wso2.com Mobile : +94714961919 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] BAM Log monitoring for Cloud
Am Donnerstag, 25. September 2014 schrieb Manjula Rathnayake : Hi Gimantha, If we can correlate log event based on timestamp range from all services(AS,BPS,AF,etc) that is really useful when identifying issues. I would recommend that you take a look on the Google Dapper paper http://research.google.com/pubs/pub36356.html regarding distributed tracing. I think this would perfectly solve all correlation issues. For implementation you should have a look on Twitters Zipkin http://twitter.github.io/zipkin/ or Brave https://github.com/kristofa/brave . regards, Thomas ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
[Architecture] BAM Log monitoring for Cloud
Hi All, Currently I am working on the Front end. These were developed keeping Kibana as the reference. Currently the UI supports the following tasks. 1. Setting the refresh rate ( Refresh the logging graph and the log table / Log view )* 2. Setting the Time range ( custom time range or in the format of ''Last 5 min, Last 10 mins...etc) * 3. Searchbox for searching ( Queries will be regex or Lucene ) 4. Filters (For searching) 5. Log graph ( hits per time) 6. Filters for log table/view 7. Log table view (in progress) 8. Micro panel which is similar to Kibana micro panel (in progress) The graph is created using jqplot[1] library. So the graph support all the features jqplot offers. Other UIs are based on JQuery/JQueryUI[2] The Micro Panel will be developed using Bootstrap[3]. Note that the theme used for the UI can be changed and the log table is still in progress. Currently the UI is integrated with ElasticSearch to view real log data. Here are some screenshots of the current UIs. Update-24.09.2014 https://docs.google.com/a/wso2.com/folderview?id=0B7luxEF9AEBxSHc3aEI3YUxYRVkusp=drive_web [1] http://www.jqplot.com/ [2] https://jquery.org/projects/ [3] http://getbootstrap.com/javascript/ -- Gimantha Bandara Software Engineer WSO2. Inc : http://wso2.com Mobile : +94714961919 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] BAM Log monitoring for Cloud
Is there any specific reason for using jqplot? AFAIK platform wide, we are using flotchart lib[1] due to advance customization capabilities. 1. http://www.flotcharts.org/ *Dakshika Jayathilaka* Software Engineer WSO2, Inc. lean.enterprise.middleware 0771100911 On Wed, Sep 24, 2014 at 2:48 PM, Gimantha Bandara giman...@wso2.com wrote: Hi All, Currently I am working on the Front end. These were developed keeping Kibana as the reference. Currently the UI supports the following tasks. 1. Setting the refresh rate ( Refresh the logging graph and the log table / Log view )* 2. Setting the Time range ( custom time range or in the format of ''Last 5 min, Last 10 mins...etc) * 3. Searchbox for searching ( Queries will be regex or Lucene ) 4. Filters (For searching) 5. Log graph ( hits per time) 6. Filters for log table/view 7. Log table view (in progress) 8. Micro panel which is similar to Kibana micro panel (in progress) The graph is created using jqplot[1] library. So the graph support all the features jqplot offers. Other UIs are based on JQuery/JQueryUI[2] The Micro Panel will be developed using Bootstrap[3]. Note that the theme used for the UI can be changed and the log table is still in progress. Currently the UI is integrated with ElasticSearch to view real log data. Here are some screenshots of the current UIs. Update-24.09.2014 https://docs.google.com/a/wso2.com/folderview?id=0B7luxEF9AEBxSHc3aEI3YUxYRVkusp=drive_web [1] http://www.jqplot.com/ [2] https://jquery.org/projects/ [3] http://getbootstrap.com/javascript/ -- Gimantha Bandara Software Engineer WSO2. Inc : http://wso2.com Mobile : +94714961919 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] BAM Log monitoring for Cloud
Hi Dakshika, There is no specific reason. It seems like both jqplot and Flot have the same customization capabilities. It will not be hard to change the graph libraries, if it is needed. Thanks, On Wed, Sep 24, 2014 at 3:28 PM, Dakshika Jayathilaka daksh...@wso2.com wrote: Is there any specific reason for using jqplot? AFAIK platform wide, we are using flotchart lib[1] due to advance customization capabilities. 1. http://www.flotcharts.org/ *Dakshika Jayathilaka* Software Engineer WSO2, Inc. lean.enterprise.middleware 0771100911 On Wed, Sep 24, 2014 at 2:48 PM, Gimantha Bandara giman...@wso2.com wrote: Hi All, Currently I am working on the Front end. These were developed keeping Kibana as the reference. Currently the UI supports the following tasks. 1. Setting the refresh rate ( Refresh the logging graph and the log table / Log view )* 2. Setting the Time range ( custom time range or in the format of ''Last 5 min, Last 10 mins...etc) * 3. Searchbox for searching ( Queries will be regex or Lucene ) 4. Filters (For searching) 5. Log graph ( hits per time) 6. Filters for log table/view 7. Log table view (in progress) 8. Micro panel which is similar to Kibana micro panel (in progress) The graph is created using jqplot[1] library. So the graph support all the features jqplot offers. Other UIs are based on JQuery/JQueryUI[2] The Micro Panel will be developed using Bootstrap[3]. Note that the theme used for the UI can be changed and the log table is still in progress. Currently the UI is integrated with ElasticSearch to view real log data. Here are some screenshots of the current UIs. Update-24.09.2014 https://docs.google.com/a/wso2.com/folderview?id=0B7luxEF9AEBxSHc3aEI3YUxYRVkusp=drive_web [1] http://www.jqplot.com/ [2] https://jquery.org/projects/ [3] http://getbootstrap.com/javascript/ -- Gimantha Bandara Software Engineer WSO2. Inc : http://wso2.com Mobile : +94714961919 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- Gimantha Bandara Software Engineer WSO2. Inc : http://wso2.com Mobile : +94714961919 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] BAM Log monitoring for Cloud
Yes we should use the same library everywhere .. no need to be different just to be different :-). On Wed, Sep 24, 2014 at 5:36 PM, Gimantha Bandara giman...@wso2.com wrote: Hi Dakshika, There is no specific reason. It seems like both jqplot and Flot have the same customization capabilities. It will not be hard to change the graph libraries, if it is needed. Thanks, On Wed, Sep 24, 2014 at 3:28 PM, Dakshika Jayathilaka daksh...@wso2.com wrote: Is there any specific reason for using jqplot? AFAIK platform wide, we are using flotchart lib[1] due to advance customization capabilities. 1. http://www.flotcharts.org/ *Dakshika Jayathilaka* Software Engineer WSO2, Inc. lean.enterprise.middleware 0771100911 On Wed, Sep 24, 2014 at 2:48 PM, Gimantha Bandara giman...@wso2.com wrote: Hi All, Currently I am working on the Front end. These were developed keeping Kibana as the reference. Currently the UI supports the following tasks. 1. Setting the refresh rate ( Refresh the logging graph and the log table / Log view )* 2. Setting the Time range ( custom time range or in the format of ''Last 5 min, Last 10 mins...etc) * 3. Searchbox for searching ( Queries will be regex or Lucene ) 4. Filters (For searching) 5. Log graph ( hits per time) 6. Filters for log table/view 7. Log table view (in progress) 8. Micro panel which is similar to Kibana micro panel (in progress) The graph is created using jqplot[1] library. So the graph support all the features jqplot offers. Other UIs are based on JQuery/JQueryUI[2] The Micro Panel will be developed using Bootstrap[3]. Note that the theme used for the UI can be changed and the log table is still in progress. Currently the UI is integrated with ElasticSearch to view real log data. Here are some screenshots of the current UIs. Update-24.09.2014 https://docs.google.com/a/wso2.com/folderview?id=0B7luxEF9AEBxSHc3aEI3YUxYRVkusp=drive_web [1] http://www.jqplot.com/ [2] https://jquery.org/projects/ [3] http://getbootstrap.com/javascript/ -- Gimantha Bandara Software Engineer WSO2. Inc : http://wso2.com Mobile : +94714961919 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- Gimantha Bandara Software Engineer WSO2. Inc : http://wso2.com Mobile : +94714961919 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- Sanjiva Weerawarana, Ph.D. Founder, Chairman CEO; WSO2, Inc.; http://wso2.com/ email: sanj...@wso2.com; office: (+1 650 745 4499 | +94 11 214 5345) x5700; cell: +94 77 787 6880 | +1 408 466 5099; voip: +1 650 265 8311 blog: http://sanjiva.weerawarana.org/; twitter: @sanjiva Lean . Enterprise . Middleware ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] BAM Log monitoring for Cloud
Gimantha, The important pieces of functionality from my perspective (also, including Amila, Dimuthu, and Majula - they are using current non-BAM workaround so might have a lot of feedback as well) are: 1. Like you mentioned, ability to set time/date period with custom values and presets like last hour/day/week/month/year. 2. Ability to set grouping interval (minute, hour, day, week, month, year), 3. Be able to tag sources of data for grouping (WSO2 Cloud has many logs from many servers, these servers jointly form solutions like App Cloud, API Cloud) - we need to be able to specify that so we can see not just joint picture but split by solutions, 4. Filter by any field for drill-down: severity (error/warning/informational), solution (see #3), server, component, message, tenant, user, 5. For the currently selected time period and filters, see grouping/count by any of the fields: e.g.: App Cloud: 55% (110), API Cloud: 45% (90). OR Error: 7% (14), Warning: 3% (6), Informational: 90% (180). Etc. 6. Preserve last view - so I can quickly get to the view of my favorite filters (ideally have ability to also have multiple shortcuts), 7. Views encoded as URL parameters, so I can just forward the URL to a team mate, 8. Be able to see all messages that fall under current filters, and drill into any of these to see full error trace, get the actual log file, etc. I also like your suggestion on supporting search and regex - would help troubleshooting! Hope this helps! Dmitry On Wed, Sep 24, 2014 at 10:48 AM, Sanjiva Weerawarana sanj...@wso2.com wrote: Yes we should use the same library everywhere .. no need to be different just to be different :-). On Wed, Sep 24, 2014 at 5:36 PM, Gimantha Bandara giman...@wso2.com wrote: Hi Dakshika, There is no specific reason. It seems like both jqplot and Flot have the same customization capabilities. It will not be hard to change the graph libraries, if it is needed. Thanks, On Wed, Sep 24, 2014 at 3:28 PM, Dakshika Jayathilaka daksh...@wso2.com wrote: Is there any specific reason for using jqplot? AFAIK platform wide, we are using flotchart lib[1] due to advance customization capabilities. 1. http://www.flotcharts.org/ *Dakshika Jayathilaka* Software Engineer WSO2, Inc. lean.enterprise.middleware 0771100911 On Wed, Sep 24, 2014 at 2:48 PM, Gimantha Bandara giman...@wso2.com wrote: Hi All, Currently I am working on the Front end. These were developed keeping Kibana as the reference. Currently the UI supports the following tasks. 1. Setting the refresh rate ( Refresh the logging graph and the log table / Log view )* 2. Setting the Time range ( custom time range or in the format of ''Last 5 min, Last 10 mins...etc) * 3. Searchbox for searching ( Queries will be regex or Lucene ) 4. Filters (For searching) 5. Log graph ( hits per time) 6. Filters for log table/view 7. Log table view (in progress) 8. Micro panel which is similar to Kibana micro panel (in progress) The graph is created using jqplot[1] library. So the graph support all the features jqplot offers. Other UIs are based on JQuery/JQueryUI[2] The Micro Panel will be developed using Bootstrap[3]. Note that the theme used for the UI can be changed and the log table is still in progress. Currently the UI is integrated with ElasticSearch to view real log data. Here are some screenshots of the current UIs. Update-24.09.2014 https://docs.google.com/a/wso2.com/folderview?id=0B7luxEF9AEBxSHc3aEI3YUxYRVkusp=drive_web [1] http://www.jqplot.com/ [2] https://jquery.org/projects/ [3] http://getbootstrap.com/javascript/ -- Gimantha Bandara Software Engineer WSO2. Inc : http://wso2.com Mobile : +94714961919 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- Gimantha Bandara Software Engineer WSO2. Inc : http://wso2.com Mobile : +94714961919 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- Sanjiva Weerawarana, Ph.D. Founder, Chairman CEO; WSO2, Inc.; http://wso2.com/ email: sanj...@wso2.com; office: (+1 650 745 4499 | +94 11 214 5345) x5700; cell: +94 77 787 6880 | +1 408 466 5099; voip: +1 650 265 8311 blog: http://sanjiva.weerawarana.org/; twitter: @sanjiva Lean . Enterprise . Middleware ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- Dmitry Sotnikov VP of Cloud; WSO2, Inc.; http://wso2.com/ email: dmi...@wso2.com; cell: +1.949.303.9653; Skype: DSotnikov Lean . Enterprise . Middleware http://wso2.com/events/ ___ Architecture mailing list Architecture@wso2.org
Re: [Architecture] BAM Log monitoring for Cloud
Hi Gimantha, If we can correlate log event based on timestamp range from all services(AS,BPS,AF,etc) that is really useful when identifying issues. However there are workarounds as well, like opening multiple tabs for each service and search for same timestamp range. Apart from that, I think Dmitry mentioned all other features that is really useful. I find it is very useful to see the part of the log when an error log is identified. This is because, sometimes we need to see what are other INFO, DEBUG logs around particular error log. Filtering support based on regex is a must feature. Because we need to prioritize the issues based on their frequency in addition to severity and fix those issues first. thank you. On Thu, Sep 25, 2014 at 2:08 AM, Dmitry Sotnikov dmi...@wso2.com wrote: Gimantha, The important pieces of functionality from my perspective (also, including Amila, Dimuthu, and Majula - they are using current non-BAM workaround so might have a lot of feedback as well) are: 1. Like you mentioned, ability to set time/date period with custom values and presets like last hour/day/week/month/year. 2. Ability to set grouping interval (minute, hour, day, week, month, year), 3. Be able to tag sources of data for grouping (WSO2 Cloud has many logs from many servers, these servers jointly form solutions like App Cloud, API Cloud) - we need to be able to specify that so we can see not just joint picture but split by solutions, 4. Filter by any field for drill-down: severity (error/warning/informational), solution (see #3), server, component, message, tenant, user, 5. For the currently selected time period and filters, see grouping/count by any of the fields: e.g.: App Cloud: 55% (110), API Cloud: 45% (90). OR Error: 7% (14), Warning: 3% (6), Informational: 90% (180). Etc. 6. Preserve last view - so I can quickly get to the view of my favorite filters (ideally have ability to also have multiple shortcuts), 7. Views encoded as URL parameters, so I can just forward the URL to a team mate, 8. Be able to see all messages that fall under current filters, and drill into any of these to see full error trace, get the actual log file, etc. I also like your suggestion on supporting search and regex - would help troubleshooting! Hope this helps! Dmitry On Wed, Sep 24, 2014 at 10:48 AM, Sanjiva Weerawarana sanj...@wso2.com wrote: Yes we should use the same library everywhere .. no need to be different just to be different :-). On Wed, Sep 24, 2014 at 5:36 PM, Gimantha Bandara giman...@wso2.com wrote: Hi Dakshika, There is no specific reason. It seems like both jqplot and Flot have the same customization capabilities. It will not be hard to change the graph libraries, if it is needed. Thanks, On Wed, Sep 24, 2014 at 3:28 PM, Dakshika Jayathilaka daksh...@wso2.com wrote: Is there any specific reason for using jqplot? AFAIK platform wide, we are using flotchart lib[1] due to advance customization capabilities. 1. http://www.flotcharts.org/ *Dakshika Jayathilaka* Software Engineer WSO2, Inc. lean.enterprise.middleware 0771100911 On Wed, Sep 24, 2014 at 2:48 PM, Gimantha Bandara giman...@wso2.com wrote: Hi All, Currently I am working on the Front end. These were developed keeping Kibana as the reference. Currently the UI supports the following tasks. 1. Setting the refresh rate ( Refresh the logging graph and the log table / Log view )* 2. Setting the Time range ( custom time range or in the format of ''Last 5 min, Last 10 mins...etc) * 3. Searchbox for searching ( Queries will be regex or Lucene ) 4. Filters (For searching) 5. Log graph ( hits per time) 6. Filters for log table/view 7. Log table view (in progress) 8. Micro panel which is similar to Kibana micro panel (in progress) The graph is created using jqplot[1] library. So the graph support all the features jqplot offers. Other UIs are based on JQuery/JQueryUI[2] The Micro Panel will be developed using Bootstrap[3]. Note that the theme used for the UI can be changed and the log table is still in progress. Currently the UI is integrated with ElasticSearch to view real log data. Here are some screenshots of the current UIs. Update-24.09.2014 https://docs.google.com/a/wso2.com/folderview?id=0B7luxEF9AEBxSHc3aEI3YUxYRVkusp=drive_web [1] http://www.jqplot.com/ [2] https://jquery.org/projects/ [3] http://getbootstrap.com/javascript/ -- Gimantha Bandara Software Engineer WSO2. Inc : http://wso2.com Mobile : +94714961919 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- Gimantha Bandara Software Engineer WSO2. Inc : http://wso2.com Mobile : +94714961919 ___ Architecture mailing list Architecture@wso2.org
