Repository: trafficserver Updated Branches: refs/heads/master a17db36fb -> 0aabfb0bf
TS-2695: docs: howto on basics with logstash and circonus Project: http://git-wip-us.apache.org/repos/asf/trafficserver/repo Commit: http://git-wip-us.apache.org/repos/asf/trafficserver/commit/0aabfb0b Tree: http://git-wip-us.apache.org/repos/asf/trafficserver/tree/0aabfb0b Diff: http://git-wip-us.apache.org/repos/asf/trafficserver/diff/0aabfb0b Branch: refs/heads/master Commit: 0aabfb0bf7c63f8a2dd062b703cecb3c7f6eced8 Parents: a17db36 Author: Jon Sime <[email protected]> Authored: Tue Feb 16 22:59:36 2016 +0000 Committer: Jon Sime <[email protected]> Committed: Tue Feb 16 23:00:59 2016 +0000 ---------------------------------------------------------------------- .../monitoring/third-party/circonus.en.rst | 125 ++++++++++++ .../monitoring/third-party/index.en.rst | 15 +- .../monitoring/third-party/logstash.en.rst | 198 +++++++++++++++++++ .../admin/monitor/circonus/check-config-1.png | Bin 0 -> 160451 bytes .../admin/monitor/circonus/check-config-2.png | Bin 0 -> 49277 bytes .../admin/monitor/circonus/check-config-3.png | Bin 0 -> 54633 bytes .../admin/monitor/circonus/metric-grid.png | Bin 0 -> 83094 bytes .../admin/monitor/circonus/new-check-button.png | Bin 0 -> 10076 bytes 8 files changed, 335 insertions(+), 3 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/trafficserver/blob/0aabfb0b/doc/admin-guide/monitoring/monitoring/third-party/circonus.en.rst ---------------------------------------------------------------------- diff --git a/doc/admin-guide/monitoring/monitoring/third-party/circonus.en.rst b/doc/admin-guide/monitoring/monitoring/third-party/circonus.en.rst new file mode 100644 index 0000000..27ae6d2 --- /dev/null +++ b/doc/admin-guide/monitoring/monitoring/third-party/circonus.en.rst @@ -0,0 +1,125 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. include:: ../../../../common.defs + +.. _admin-monitoring-circonus: + +Circonus +******** + +`Circonus <http://www.circonus.com/>`_ is a commercial systems monitoring and +analytics product, available in hosted (SaaS) and on-site configurations. + +While it does not currently offer a module specific to |ATS| statistics, it is +very easy to add monitoring by combining the |TS| plugin :ref:`Stats Over HTTP +<admin-plugins-stats-over-http>` with Circonus's JSON metrics collector. + +Setup +===== + +The following steps assume that you already have an active Circonus account on +their hosted service, or that you already have an on-site installation. If you +do not, that must be set up first. Trial accounts are available and will +suffice for this guide. + +Enable Stats Over HTTP +---------------------- + +Follow the directions in the :ref:`admin-plugins-stats-over-http` chapter to +configure the plugin. At a bare minimum, you will need to add the plugin to +:file:`plugin.config`. For this guide, we will assume the following entry:: + + stats_over_http.so _statistics + +This enables the plugin and makes |TS| statistics available at the path +``/_statistics``. For production environments, you are strongly encouraged to +use a different (and obfuscated) path for your statistics since they reveal +many internal details of your caching proxy. The plugin documentation provides +more details. + +Create Circonus Check +--------------------- + +For these steps, we will assume that |TS| is listening on port ``80`` at the +domain ``trafficserver.example.com``. You should adjust these to match your +environment. + +#. Begin the new check creation process from within your Circonus account by + clicking the *New Check* button near the top-right of the checks screen. + + .. image:: ../../../../static/images/admin/monitor/circonus/new-check-button.png + :alt: Circonus New Check button + :align: center + +#. For the check type, you should select *JSON* under the *Custom* list, and + then choose the *Pull* type. The broker you choose to use is entirely up to + you and will depend largely on whether you are using on-site Circonus or the + hosted service, as well as the geographic location of your |TS| instance(s). + + .. image:: ../../../../static/images/admin/monitor/circonus/check-config-1.png + :alt: Choosing a check type + :align: center + +#. Click on *Configure Check* to proceed. + +#. Click on the *Show Advanced Configuration* link to expand the check options. + Enter your |TS| instance's externally-accessible domain for *Host* and make + sure to set *URI* to the path at which the statistics are available. Adjust + any of the other options to match your environment if necessary (for this + guide, only *Host* and *URI* will need to be entered). + + .. image:: ../../../../static/images/admin/monitor/circonus/check-config-2.png + :alt: Advanced check configuration + :align: center + +#. Click on *Test Check* to proceed. + +#. If all is well, you should see a long list (several hundred entries) of |TS| + statistics, with their associated values. If you do not, verify that |TS| is + running and use ``curl`` to manually fetch the statistics data from your + server. + + .. image:: ../../../../static/images/admin/monitor/circonus/check-config-3.png + :alt: Check test + :align: center + +#. You may want to limit the metrics you actually track from |TS|, since so + many are made available. If so, simply uncheck those you aren't interested + in from the list. If you only want to track a few metrics, click + *Select None* and then check the ones you want. You can filter the results + to easily find metrics in the list. + +#. Save your new check. + +#. Verify the collected metrics in Circonus by opening your new check and + clicking on *Metrics Grid* for an overview visualization of all the data + being collected. + + .. image:: ../../../../static/images/admin/monitor/circonus/metric-grid.png + :alt: Circonus metric grid + :align: center + +Congratulations! You're now ready to begin setting up alerts, visualizations, +and dashboards for your |TS| statistics. You can repeat the above steps for any +additional |TS| instances you have running. + +Further Reading +=============== + +* `Official Circonus Website <http://www.circonus.com/>`_ + http://git-wip-us.apache.org/repos/asf/trafficserver/blob/0aabfb0b/doc/admin-guide/monitoring/monitoring/third-party/index.en.rst ---------------------------------------------------------------------- diff --git a/doc/admin-guide/monitoring/monitoring/third-party/index.en.rst b/doc/admin-guide/monitoring/monitoring/third-party/index.en.rst index fb52da9..2e947d6 100644 --- a/doc/admin-guide/monitoring/monitoring/third-party/index.en.rst +++ b/doc/admin-guide/monitoring/monitoring/third-party/index.en.rst @@ -22,11 +22,20 @@ Integrating Third-Party Tools ***************************** +There are many monitoring and alerting systems available, too many for us to +hope to cover every possible option here. We can, however, attempt to document +using some of the more common options. + +Some of these third party monitoring applications and services are able to tap +into the extensive list of |TS| statistics, others make use of |TS| log files, +and yet others are aimed at simple health check reporting. An extensive service +monitoring configuration may take advantage of more than one service to play to +their complementary strengths. Which tool, or combination of tools, is right +for your infrastructure will likely vary. + .. toctree:: - :maxdepth: 2 + :maxdepth: 1 circonus.en logstash.en - nagios.en - zabbix.en http://git-wip-us.apache.org/repos/asf/trafficserver/blob/0aabfb0b/doc/admin-guide/monitoring/monitoring/third-party/logstash.en.rst ---------------------------------------------------------------------- diff --git a/doc/admin-guide/monitoring/monitoring/third-party/logstash.en.rst b/doc/admin-guide/monitoring/monitoring/third-party/logstash.en.rst new file mode 100644 index 0000000..9ae39f2 --- /dev/null +++ b/doc/admin-guide/monitoring/monitoring/third-party/logstash.en.rst @@ -0,0 +1,198 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. include:: ../../../../common.defs + +.. _admin-monitoring-logstash: + +Logstash +******** + +`Logstash <https://www.elastic.co/products/logstash>`_ is a powerful, open +source, unstructured data processing program that can accept text data from +many different sources (directly over TCP/UDP, via Unix sockets, or by reading +in files from disk for example), in many different formats and transform those +inputs into structured, searchable documents. + +One of the most common use cases is to take text based system and application +logs, extract individual fields (e.g. host names, error codes, timing data, and +so on), and make the data available in Elasticsearch for searching and +reporting. + +In this guide, we will cover the basics of getting your |TS| log data into +Logstash. Going the next step and building fancy Kibana dashboards on top of +that is currently left as an exercise for the reader. + +|TS| Log Formats +================ + +|TS| provides a very flexible set of logging outputs. Almost any format can be +constructed. The full range of options is covered in the +:ref:`admin-monitoring-logging` chapter. + +This guide will walk you through using the appropriate filters in Logstash for +the common logging formats in |TS|. If you have constructed your own custom log +formats, you will need to build upon these examples and refer to the Logstash +documentation to produce custom filters capable of parsing your own formats. + +Logstash Input +============== + +For the on-disk logs produced by |TS|, you will want to use Logstash's ``file`` +input plugin. Note that your logs must be in ``ASCII`` format, not ``binary``, +for the plugin to work. + +Assuming that your |TS| event logs are named ``access-<rotationtimestamp>.log`` +and stored at ``/var/log/trafficserver/``, the following Logstash input +configuration should work:: + + input { + file { + path => /var/log/trafficserver/access-*.log + } + } + +Logstash provides some additional tweaking options, which are explained in the +`file plugin documentation <https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html>`_ +but the above provides the bare minimum required to have Logstash read log data +from local disks. + +Logstash Filters +================ + +The `grok filter <https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html>`_ +in Logstash allows you to completely tailor the parsing of your source data and +extract as many or as few fields as you like. + +Some patterns are already built and can be used very easily. If you have built +custom log formats for |TS|, you may need to write your own patterns, however. + +Squid Compatible +---------------- + +The :ref:`admin-logging-format-squid` log format includes, unsurprisingly, a +few useful fields for proxy servers. Using the following grok pattern will +extract this information from your |TS| logs if you employ the Squid compatible +log format:: + + filter { + grok { + match => { "message" => "%{NUMBER:timestamp} %{NUMBER:timetoserve} %{IPORHOST:clientip} %{WORD:cachecode}/%{NUMBER:response} %{NUMBER:bytes} %{WORD:verb} %{NOTSPACE:request} %{USER:auth} %{NOTSPACE:route} %{DATA:contenttype}" } + } + date { + match => [ "timestamp", "UNIX" ] + } + } + +The resulting structured document will contain the following fields: + +=========== =================================================================== +Field Description +=========== =================================================================== +timestamp Date and time of the client request. +timetoserve Time, in seconds, from initial client connection to |TS| until the + last byte has been sent back to client from |TS|. +clientip Client IP address or hostname. +cachecode :ref:`admin-monitoring-logging-cache-result-codes`. +response HTTP response status code sent by |TS| to the client. +bytes Length, in bytes, of the |TS| response to the client, including + headers. +verb HTTP method (e.g. ``GET``, ``POST``, etc.) of the client request. +request URL specified by the client request. +auth Authentication username supplied by the client, if present. +route Proxy hierarchy route; the route used by |TS| to retrieve the cache + object. +contenttype Content type of the response. +=========== =================================================================== + +Netscape Common +--------------- + +If your |TS| instance is already outputting :ref:`admin-logging-format-common` +format logs, then Logstash's ``COMMONAPACHELOG`` pattern will handle your logs +out of the box. Add the following filter block to your Logstash configuration:: + + filter { + grok { + match => { "message" => "%{COMMONAPACHELOG}" } + } + } + +This will produce a structured document for each log entry with the following +fields: + +=========== =================================================================== +Field Description +=========== =================================================================== +clientip Client IP address or hostname. +ident Always a literal ``-`` character for |TS| logs. +auth The authentication username for the client request. A ``-`` means + no authentication was required (or supplied). +timestamp The date and time of the client request. +verb HTTP method used for the request (e.g. ``GET``, ``POST``, etc.). +request URL specified by the client request. +httpversion HTTP version (e.g. ``1.1``) used by the client. +rawrequest *See note below.* +response HTTP status code used for |TS| response (not the origin's response + code). +bytes Length of |TS| response to client, in bytes. +=========== =================================================================== + +.. note:: + + ``rawrequest`` is populated when the usual ``"<verb> <request> http/<httpversion>"`` + pattern was not matched. In that event, those three fields will be missing + from the document, and instead ``rawrequest`` will have the original string. + +Netscape Extended +----------------- + +The following pattern adds to Common Apache to support the additional fields +found in :ref:`admin-logging-format-extended`:: + + filter { + grok { + match => { "message" => "%{COMMONAPACHELOG} %{NUMBER:originstatus} %{NUMBER:originrespbytes} %{NUMBER:clientreqbytes} %{NUMBER:proxyreqbytes} %{NUMBER:clienthdrbytes} %{NUMBER:proxyresphdrbytes} %{NUMBER:proxyreqhdrbytes} %{NUMBER:originhdrbytes} %{NUMBER:timetoserve}" } + } + } + +Because this starts out with the ``COMMONAPACHELOG`` pattern, you will get all +of the fields mentioned in `Netscape Common`_ above, as well as the following: + +================= ============================================================= +Field Description +================= ============================================================= +originstatus HTTP status code returned by origin server. +originrespbytes Body length, in bytes, of origin's response to |TS|. +clientreqbytes Body length, in bytes, of client request to |TS|. +proxyreqbytes Body length, in bytes, of |TS| request to origin. +clienthdrbytes Header length, in bytes, of client request to |TS|. +proxyresphdrbytes Header length, in bytes, of |TS| response to client. +proxyreqhdrbytes Header length, in bytes, of |TS| request to origin. +originhdrbytes Header length, in bytes, of origin's response to |TS|. +timetoserve Time, in seconds, from initial client connection to |TS| + until the last byte has been sent back to client from |TS|. +================= ============================================================= + +Further Reading +=============== + +* `Logstash Documentation <https://www.elastic.co/guide/en/logstash/current/index.html>`_ + +* `Grok Patterns <https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns>`_ + +* `Elasticsearch Documentation <https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html>`_ http://git-wip-us.apache.org/repos/asf/trafficserver/blob/0aabfb0b/doc/static/images/admin/monitor/circonus/check-config-1.png ---------------------------------------------------------------------- diff --git a/doc/static/images/admin/monitor/circonus/check-config-1.png b/doc/static/images/admin/monitor/circonus/check-config-1.png new file mode 100644 index 0000000..be98f99 Binary files /dev/null and b/doc/static/images/admin/monitor/circonus/check-config-1.png differ http://git-wip-us.apache.org/repos/asf/trafficserver/blob/0aabfb0b/doc/static/images/admin/monitor/circonus/check-config-2.png ---------------------------------------------------------------------- diff --git a/doc/static/images/admin/monitor/circonus/check-config-2.png b/doc/static/images/admin/monitor/circonus/check-config-2.png new file mode 100644 index 0000000..afd7fe9 Binary files /dev/null and b/doc/static/images/admin/monitor/circonus/check-config-2.png differ http://git-wip-us.apache.org/repos/asf/trafficserver/blob/0aabfb0b/doc/static/images/admin/monitor/circonus/check-config-3.png ---------------------------------------------------------------------- diff --git a/doc/static/images/admin/monitor/circonus/check-config-3.png b/doc/static/images/admin/monitor/circonus/check-config-3.png new file mode 100644 index 0000000..dfd025a Binary files /dev/null and b/doc/static/images/admin/monitor/circonus/check-config-3.png differ http://git-wip-us.apache.org/repos/asf/trafficserver/blob/0aabfb0b/doc/static/images/admin/monitor/circonus/metric-grid.png ---------------------------------------------------------------------- diff --git a/doc/static/images/admin/monitor/circonus/metric-grid.png b/doc/static/images/admin/monitor/circonus/metric-grid.png new file mode 100644 index 0000000..8cbb222 Binary files /dev/null and b/doc/static/images/admin/monitor/circonus/metric-grid.png differ http://git-wip-us.apache.org/repos/asf/trafficserver/blob/0aabfb0b/doc/static/images/admin/monitor/circonus/new-check-button.png ---------------------------------------------------------------------- diff --git a/doc/static/images/admin/monitor/circonus/new-check-button.png b/doc/static/images/admin/monitor/circonus/new-check-button.png new file mode 100644 index 0000000..a7107d2 Binary files /dev/null and b/doc/static/images/admin/monitor/circonus/new-check-button.png differ
