This is an automated email from the ASF dual-hosted git repository. heneveld pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/brooklyn-docs.git
commit c30a584351b7f0989eda08bb1e7f7d087f0c6950 Author: Alex Heneveld <[email protected]> AuthorDate: Fri Aug 20 17:30:42 2021 +0100 add tips for setting up ES - with thx to @jcabrerizo --- guide/ops/logging.md | 141 ++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 128 insertions(+), 13 deletions(-) diff --git a/guide/ops/logging.md b/guide/ops/logging.md index ef6d3c5..2fe535c 100644 --- a/guide/ops/logging.md +++ b/guide/ops/logging.md @@ -73,24 +73,43 @@ For example (on mac): mv $LOG_FILE /path/to/archive/brooklyn.debug-$TIMESTAMP.log.zip -## Logging aggregators +## Logging Aggregators Integration with systems like Logstash and Splunk is possible using standard log4j configuration. Log4j can be configured to write to syslog using the SyslogAppender which can then [feed its logs to Logstash](http://www.logstash.net/docs/1.4.2/inputs/syslog). + ## Logbook The logbook offers the possibility to query and view logs in the UI. By default, logs are stored in files as per configuration -in `etc/org.ops4j.pax.logging.cfg`. The logbook can be configured against different log aggregation sources by adding the -following parameters in `brooklyn.cfg`: +in `etc/org.ops4j.pax.logging.cfg`. The logbook can be configured against different log aggregation sources by setting the +`brooklyn.logbook.logStore` parameter in `brooklyn.cfg`, and depending which backend is used, other parameters. -* plain log files +For example to use the local log files written by Apache Brooklyn, you could configure: brooklyn.logbook.logStore=org.apache.brooklyn.util.core.logbook.file.FileLogStore brooklyn.logbook.fileLogStore.path=/var/logs/brooklyn/brooklyn.debug.log -* or Elasticsearch released under the Apache License, version 2.0 fork created by AWS +In production environments where log data is desired to be retained, Apache Brooklyn supports Elasticsearch backends. +This can be a dedicated ES environment for use by Apache Brooklyn or a shared/managed ES facility that handles many logs, +or -- for lightweight usage -- a simple local ES server running on the same instance as Apache Brooklyn. +As with any log storage requirement, the sizing, scaling, backup and maintenance of the logging environment +requires careful attention. Elasticsearch includes numerous options to configure these, with one suggested configuration +outlined in more detail below. + +By default, only users with the `root` entitlement or an explicit `logViewer` entitlement are able to see log info through Apache Brooklyn. + +The Logbook UI widget can be found throughout the product: +in the About section, where all logs can be viewed; +on the App Inspector Entity Summary view, and +on the App Inspector Activity view, where logs filtered by entity ID and activity ID respectively are shown. + + +### Suggested Elasticsearch Setup + +[OpenSearch (OpenDistro for Elasticsearch)](https://opendistro.github.io/for-elasticsearch) is an Apache-licensed open-source +backend that works well with Apache Brooklyn, with this configuration in `brooklyn.cfg`: brooklyn.logbook.logStore=org.apache.brooklyn.util.core.logbook.opensearch.OpenSearchLogStore brooklyn.logbook.openSearchLogStore.host=https://localhost:9200 @@ -99,16 +118,12 @@ following parameters in `brooklyn.cfg`: brooklyn.logbook.openSearchLogStore.password=admin brooklyn.logbook.openSearchLogStore.verifySsl=false -Users with `root` entitlement only can query and view logs in the logbook. - -Logbook UI widget can be found in About section where all logs can be viewed, as well as in App Inspector Entity view and -Activity view where logs filtered by entity ID and activity ID respectively. -### Elasticsearch setup +#### Routing Logs to Elastic Search -Refer to the [official documentation](https://opendistro.github.io/for-elasticsearch/downloads.html#try) for - installation guide. [Fluentd](https://www.fluentd.org/download) daemon can be configured to read the log files -for Elasticsearch. See example of Fluentd `td-agent.conf` below: +There are many solutions to routing log messages from Apache Brooklyn to Elasticsearch, either plugging in to the log4j subsystem +or routing the log files from disk. [Fluentd](https://www.fluentd.org/download), with the following configuration in `td-agent.conf`, +is a good simple way to forward content added to the log files: ``` <source> @@ -135,6 +150,102 @@ for Elasticsearch. See example of Fluentd `td-agent.conf` below: </match> ``` +#### Sizing and Rotating Logs + +Keeping log data obviously consumes disk storage, and serving the data requires memory. +The log levels in Apache Brooklyn can be configured on a fine-grained log-category basis, +and different levels and categories can be routed to different indexes with different +retention policies. + +When designing your strategy for maintaining logs, some good rules of thumb are: + +* Allocate 4 GB RAM for a production ES instance plus 2 GB RAM for every TB of log data that is searchable +* Consider a small number of tiers with different retention policies, + e.g. INFO and selected DEBUG messages (logins) to one index, + and all other DEBUG and lower messages sent to another index +* Consider using rolling indices on a nightly or weekly basis, and an auto-close job and/or an auto-delete job to keep memory and disk usage at a steady state; + for example the INFO and selected DEBUG messages might go to an index which rotates weekly and is deleted or moved to cold storage after two years, + whereas DEBUG and lower messages might rotate daily and be deleted after a week +* The amount of log data can vary depending how Apache Brooklyn is used, so monitor usage to get a feel for what is + maintainable, and put in place notifications if disk and/or memory usage become high +* Review the logs and turn off unnecessary categories + +Instructions and links to assist with this are below. + + +#### Index partitioning + +It’s possible to configure fluentd for sending the information to an index using an index name generated using datetime markers. +This example will create and send the data to a new index every day: + +``` +<match td.apachebrokyn.*> + @type elasticsearch + hosts https://localhost:9200 + user admin + password admin + ssl_verify false + + include_timestamp true + index_name brooklyn-rotating-%Y.%m.%d + flush_interval 5s + <buffer tag, time> + timekey 60 # chunks per hours ("3600" also available) + flush_interval 5s + </buffer> +</match> +``` + +Apache Brooklyn can be configured to use an index _pattern_ for querying, eg: + + brooklyn.logbook.openSearchLogStore.index = brooklyn-rotating-* + + +#### Index lifecycle management + +Policies also allow handling the lifecycle of the indexes. +For example, to delete indexes after a period of time: + +``` +{ + "policy": { + "description": "Delete workflow", + "default_state": "new", + "schema_version": 1, + "states": [ + { + "name": "new", + "transitions": [ + { + "state_name": "delete", + "conditions": { + "min_index_age": "60d" + } + } + ] + }, + { + "name": "delete", + "actions": [ + { + "delete": {} + } + ] + } + ], + "ism_template": { + "index_patterns": ["brooklyn-rotating*"], + "priority": 100 + } + } +} +``` + +With these building blocks, and others linked below, you can configure the retention policy that suits your environment, +balancing the trade-off between data availability and resource usage. + + + ## For More Information The following resources may be useful when configuring logging: @@ -144,3 +255,7 @@ The following resources may be useful when configuring logging: * The [Logback Project](http://logback.qos.ch/) home page * [Brooklyn Developer Guide]({{book.path.docs}}/dev/tips/logging.md) logging tips * [OPS4J Pax Logging](https://ops4j1.jira.com/wiki/display/paxlogging/Configuration) +* [Elasticsearch Best Practices](https://www.elastic.co/guide/en/elasticsearch/reference/7.x/best_practices.html) +* [Elasticsearch Memory Usage](https://www.elastic.co/blog/significantly-decrease-your-elasticsearch-heap-memory-usage) +* [OpenSearch Index Management](https://opensearch.org/docs/im-plugin/ism/index/) and [policies](https://opensearch.org/docs/im-plugin/ism/policies/) +
