This is an automated email from the ASF dual-hosted git repository. jsime pushed a commit to branch master in repository https://git-dual.apache.org/repos/asf/trafficserver.git
commit 8b5ff0afdd9d855f63f775771f6e8c04b0ce273a Author: Jon Sime <[email protected]> AuthorDate: Tue Aug 23 07:41:17 2016 +0000 TS-4738: docs: add new logging.config documentation (including updates to scattered references), and remove all mentions of logs_xml.config --- doc/admin-guide/files/logging.config.en.rst | 396 +++++++++++++++++++++ doc/admin-guide/files/records.config.en.rst | 2 +- .../monitoring/logging/log-collation.en.rst | 30 +- .../monitoring/logging/log-formats.en.rst | 55 +-- .../monitoring/logging/managing-logs.en.rst | 8 +- .../monitoring/logging/summary-logs.en.rst | 71 ++-- 6 files changed, 490 insertions(+), 72 deletions(-) diff --git a/doc/admin-guide/files/logging.config.en.rst b/doc/admin-guide/files/logging.config.en.rst new file mode 100644 index 0000000..328c829 --- /dev/null +++ b/doc/admin-guide/files/logging.config.en.rst @@ -0,0 +1,396 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. include:: ../../common.defs + +.. configfile:: logging.config + +logging.config +************** + +The :file:`logging.config` file defines all custom log file formats, filters, +and processing options. The file itself is a Lua script. + +.. important:: + + This configuration file replaces the XML based logs_xml.config from past + |TS| releases. If you are upgrading from a |TS| release which used that + configuration file, and you have created custom log formats, filters, and + destinations, you will need to update those settings to this format. + +.. _admin-custom-logs: + +Log Definitions +=============== + +Custom logs are configured by the combination of three key elements: a format, +an optional filter, and a log destination. + +A :ref:`format <admin-custom-logs-formats>` defines how log lines will appear +(as well as whether the logs using the format will be event logs or summary +logs). + +A :ref:`filter <admin-custom-logs-filters>` defines what events do, and what +events don't, make it into the logs employing the filter. + +A :ref:`log <admin-custom-logs-logs>` defines where the record of events or +summaries ends up. + +.. _admin-custom-logs-formats: + +Formats +------- + +Custom logging formats may be provided directly to a log definition, or they +may be defined as a reusable variable in your :file:`logging.config` for ease +of reference, particularly when you may have more than one log using the same +format. Which approach you use is entirely up to you, though it's strongly +recommended to create an explicit format object if you intend to reuse the same +format for multiple log files. + +To create a format object, store the result of the ``format`` function in a +variable. The function takes a table with two attributes: a mandatory string +``Format`` which defines the output format string for every event; and an +optional number ``Interval`` defining the aggregation interval for summary +logs. + +.. code:: lua + + -- A one-line-per-event format that just prints event timestamps. + myformat = format { + Format = '%<cqtq>' + } + + -- An aggregation/summary format that prints the last event timestamp from + -- the interval along with the total count of events in the same interval. + -- (Doing so every 30 seconds.) + mysummaryformat = format { + Format = '%<LAST(cqtq)> %<COUNT(*)>', + Interval = 30 + } + +You may define as many and as varied a collection of format objects as you +desire. + +Format Specification +~~~~~~~~~~~~~~~~~~~~ + +The format specification provided as the required ``Format`` entry of the table +passed to the format function is a simple string, containing whatever mixture +of logging field variables and literal characters meet your needs. Logging +fields are discussed in great detail in the :ref:`custom-logging-fields` +section. + +Flexible enough to not only emulate the logging formats of most other proxy and +HTTP servers, but also to provide even finer detail than many of them, the +logging fields are very easy to use. Within the format string, logging fields +are indicated by enclosing their name within angle brackets (``<`` and ``>``), +preceded by a percent symbol (``%``). For example, returning to the altogether +too simple format shown earlier, the following format string:: + + '%<cqtq>' + +Defines a format in which nothing but the value of the logging field +:ref:`cqtq` is interpolated for each event's entry in the log. We could include +some literal characters in the log output by updating the format specification +as so:: + + 'Event received at %<cqtq>' + +Because the string "Event received at " (including the trailing space) is just +a bunch of characters, not enclosed in ``%<...>``, it is repeated verbatim in +the logging output. + +Multiple logging fields may of course be used:: + + '%<cqtq> %<chi> %<cqhm> %<cqtx>' + +Each logging field is separately enclosed in its own percent-brace set. + +There are a small number of logging fields which extend this simple format, +primarily those dealing with request and response headers. Instead of defining +a separate logging field name for every single possible HTTP header (an +impossible task, given that arbitrary vendor/application headers may be present +in both requests and responses), there are instead single logging fields for +each of the major stages of an event lifecycle that permit access to named +headers, such as:: + + '%<{User-Agent}cqh>' + +Which emits to the log the value of the client request's ``User-Agent`` HTTP +header. Other stages of the event lifecycle have similar logging fields: +``pqh`` (proxy requests), ``ssh`` (origin server responses), and ``psh`` +(proxy responses). + +You will find a complete listing of the available fields in +:ref:`custom-logging-fields`. + +Aggregation Interval +~~~~~~~~~~~~~~~~~~~~ + +Every format may be given an optional ``Interval`` value, specified as the +number of seconds over which events destined for a log using the format are +aggregated and summarized. Logs which use formats containing an aggregation +interval do not behave like regular logs, with a single line for every event. +Instead, they emit a single line only every *interval*\ -seconds. + +These types of logs are described in more detail in +:ref:`admin-monitoring-logging-summary-logs`. + +Formats have no interval by default, and will generate event-based logs unless +given one. + +.. _admin-custom-logs-filters: + +Filters +------- + +Filters may be used, optionally, to accept or reject logging for matching +events, or to scrub the values of individual fields from logging output (while +retaining other information; useful for ensuring that sensitive information +cannot inadvertently make it into log files). + +Filter objects are created by calling one of the following functions: + +filter.accept(string) + Creates a filter object which accepts events for logging which match the + rule specified in ``string``. + +filter.reject(string) + Creates a filter object which rejects events for logging which match the + rule specified in ``string``. + +filter.wipe(string) + Creates a filter object which clears the values of query parameters listed + in ``string``. + +For both ``accept`` and ``wipe`` filters, the string passed defines a rule in +the following format:: + + <field> <operator> <value> + +Filter Fields +~~~~~~~~~~~~~ + +The log fields have already been discussed in the `Formats`_ section above. For +a reference to the available log field names, see :ref:`custom-logging-fields`. +Unlike with the log format specification, you do not wrap the log field names +in any additional markup. + +Filter Operators +~~~~~~~~~~~~~~~~ + +The operators describe how to perform the matching in the filter rule, and may +be any one of the following: + +``MATCH`` + True if the values of ``field`` and ``value`` are identical. + Case-sensitive. + +``CASE_INSENSITIVE_MATCH`` + True if the values of ``field`` and ``value`` are identical. + Case-insensitive. + +``CONTAIN`` + True if the value of ``field`` contains ``value`` (i.e. ``value`` is a + substring of the contents of ``field``). Case-sensitive. + +``CASE_INSENSITIVE_CONTAIN`` + True if the value of ``field`` contains ``value`` (i.e. ``value`` is a + substring of the contents of ``field``). Case-insensitive. + +Filter Values +~~~~~~~~~~~~~ + +The final component of a filter string specifies the value against which the +name field will be compared. For integer matches, all of the operators are +effectively equivalent and require the field to be equal to the given integer. +For IP addresses, ranges may be specified by separating the first address and +the last of the range with a single ``-`` dash, as ``10.0.0.0-10.255.255.255`` +which gives the ranges for the 10/8 network. Other network notations are not +supported at this time. + +Wiping Filters +~~~~~~~~~~~~~~ + +Filters created with ``filter.wipe`` function differently than the accept and +reject filters. Instead of a rule, as described above for those filter types, +the wiping filter simply lists the query parameter(s) whose values should be +scrubbed before any logging occurs. This prevents sensitive information from +being logged by fields which include the query string portion of the request +URL. It can also be useful to remove things like cache-busting or +inconsequentially variable parameters that might otherwise obfuscate the +reporting from log analyzers. + +Multiple query parameters may be listed, separated by spaces, though only the +first occurence of each will be wiped from the query string if any individual +parameter appears more than once in the URL. + +.. _admin-custom-logs-logs: + +Logs +---- + +Up to this point, we've only described what events should be logged and what +they should look like in the logging output. Now we define where those logs +should be sent. Three options currently exist for the type of logging output, +and each is selected by invoking the appropriate function. All three functions +take a single Lua table as their argument, with the same set of key/value +pairs. + +log.ascii(table) + Creates an ASCII logging object. + +log.binary(table) + Creates a binaryy logging object. + +log.pipe(table) + Creates a logging object that logs to a pipe. + +There is no need to capture the return values of these functions. Which type of +logging output you choose depends largely on how you intend to process the logs +with other tools, and a discussion of the merits of each is covered elsewhere, +in :ref:`admin-logging-binary-v-ascii`. + +The following subsections cover the contents of the table which should be +passed when creating your logging object. Only ``Filename`` and ``Format`` are +required. + +=================== =========== =============================================== +Name Type Description +=================== =========== =============================================== +Filename string The name of the logfile relative to the default + logging directory (set with + :ts:cv:`proxy.config.log.logfile_dir`). +Format string or Either a format object created earlier, or a + format obj string with a valid format specification. +Header string If present, emitted as the first line of each + new log file. +RollingEnabled *see below* Determines the type of log rolling to use (or + whether to disable rolling). Overrides + :ts:cv:`proxy.config.log.rolling_enabled`. +RollingIntervalSec number Interval in seconds between log file rolling. + Overrides + :ts:cv:`proxy.config.log.rolling_interval_sec`. +RollingOffsetHr number Specifies an hour (from 0 to 23) at which log + rolling is guaranteed to align. Only has an + effect if RollingIntervalSec is set to greater + than one hour. Overrides + :ts:cv:`proxy.config.log.rolling_offset_hr`. +RollingSizeMb number Size, in megabytes, at which log files are + rolled. +Filters array of The optional list of filter objects which + filters restrict the individual events logged. +CollationHosts array of If present, one or more strings specifying the + strings log collation hosts to which logs should be + delivered, each in the form of "<ip>:<port>". + :ref:`admin-monitoring-logging-log-collation` + for more information. +=================== =========== =============================================== + +Enabling log rolling may be done globally in :file:`records.config`, or on a +per-log basis by passing appropriate values for the ``RollingEnabled`` key. The +latter method may also be used to effect different rolling settings for +individual logs. The numeric values that may be passed are the same as used by +:ts:cv:`proxy.config.log.rolling_enabled`. For convenience and readability, +the following predefined variables may also be used in :file:`logging.config`: + +log.roll.none + Disable log rolling. + +log.roll.time + Roll at a certain time frequency, specified by RollingIntervalSec and + RollingOffsetHr. + +log.roll.size + Roll when the size exceeds RollingSizeMb. + +log.roll.both + Roll when either the specified rolling time is reached or the specified + file size is reached. + +log.roll.any + Roll the log file when the specified rolling time is reached if the size of + the file equals or exceeds the specified size. + +Examples +======== + +The following is an example of a format that collects information using three +common fields: + +.. code:: lua + + minimalfmt = format { + Format = '%<chi> : %<cqu> : %<pssc>' + } + +The following is an example of a format that uses aggregate operators to +produce a summary log: + +.. code:: lua + + summaryfmt = format { + Format = '%<LAST(cqts)> : %<COUNT(*)> : %<SUM(psql)>', + Interval = 10 + } + +The following is an example of a filter that will cause only REFRESH_HIT events +to be logged: + +.. code:: lua + + refreshhitfilter = filter.accept('pssc MATCH REFRESH_HIT') + +The following is an example of a filter that will cause the value of the first +query parameter named ``passwd`` to be wiped. + +.. code:: lua + + passwdfilter = filter.wipe('passwd') + +The following is an example of a log specification that creates a local log +file for the minimal format defined earlier. The log filename will be +``minimal.log`` because we select the ASCII logging format. + +.. code:: lua + + log.ascii { + Filename = 'minimal', + Format = minimalfmt + } + +The following is an example of a log specification that creates a local log +file using the summary format from earlier, and only includes events that +matched the REFRESH_HIT filter we created. + +.. code:: lua + + log.ascii { + Filename = 'refreshhit_summary', + Format = summaryfmt, + Filters = { refreshhitfilter } + } + +Further Reading +=============== + +As the :file:`logging.config` configuration file is just a Lua script (with a +handful of predefined functions and variables), general Lua references may be +handy for those not already familiar with the language. + +* `Lua Documentation <https://www.lua.org/docs.html>`_ diff --git a/doc/admin-guide/files/records.config.en.rst b/doc/admin-guide/files/records.config.en.rst index 608adb6..afdaf6f 100644 --- a/doc/admin-guide/files/records.config.en.rst +++ b/doc/admin-guide/files/records.config.en.rst @@ -2651,7 +2651,7 @@ Logging Configuration standard and traditional custom formats to the collation server. ===== ====================================================================== - For information on sending XML-based custom formats to the collation server, + For information on sending custom formats to the collation server, refer to :ref:`admin-monitoring-logging-formats` and :file:`logging.config`. .. note:: diff --git a/doc/admin-guide/monitoring/logging/log-collation.en.rst b/doc/admin-guide/monitoring/logging/log-collation.en.rst index 5849dc6..3334e7f 100644 --- a/doc/admin-guide/monitoring/logging/log-collation.en.rst +++ b/doc/admin-guide/monitoring/logging/log-collation.en.rst @@ -17,6 +17,8 @@ .. include:: ../../../common.defs +.. _admin-monitoring-logging-log-collation: + Log Collation ************* @@ -103,9 +105,8 @@ restart |TS|. #. In the :file:`records.config` file, edit the following variables: - :ts:cv:`proxy.local.log.collation_mode`: ``2`` to configure this node as - log collation client and sen standard formatted log entries to the - collation server. For XML-based formatted log entries, see - :file:`logging.config` file. + log collation client and send standard formatted log entries to the + collation server. For custom log entries, see :file:`logging.config`. - :ts:cv:`proxy.config.log.collation_host` - :ts:cv:`proxy.config.log.collation_port` - :ts:cv:`proxy.config.log.collation_secret` @@ -124,19 +125,16 @@ in addition to configuring a collation server and collation clients. To collate custom event log files: #. On each collation client, edit :file:`logging.config` and add the - :ref:`CollationHosts <logs-xml-logobject-collationhost>` attribute to the - :ref:`LogObject` specification:: - - <LogObject> - <Format = "squid"/> - <Filename = "squid"/> - <CollationHosts="ipaddress:port"/> - </LogObject> - - Where ``ipaddress`` is the hostname or IP address of the collation - server to which all log entries (for this object) are forwarded, and - ``port`` is the port number for communication between the collation - server and collation clients. + ``CollationHosts`` attribute to the relevant logs. For example, adding two + collation hosts to an ASCII log that uses the Squid format would look like: + + .. code:: lua + + log.ascii { + Format = squid, + Filename = 'squid', + CollationHosts = { '192.168.1.100:4567', '192.168.1.101:4567' } + } #. Run the command :option:`traffic_ctl config reload` to restart Traffic Server on the local node or :option:`traffic_ctl config reload` to restart Traffic Server on all diff --git a/doc/admin-guide/monitoring/logging/log-formats.en.rst b/doc/admin-guide/monitoring/logging/log-formats.en.rst index 0654764..b1bfbc9 100644 --- a/doc/admin-guide/monitoring/logging/log-formats.en.rst +++ b/doc/admin-guide/monitoring/logging/log-formats.en.rst @@ -68,7 +68,7 @@ Defining Log Objects ==================== To perform any logging at all on your |TS| nodes, you must have at least one -:ref:`LogObject` defined in :file:`logging.config`. These definitions configure +log defined in :file:`logging.config`. These definitions configure what logs will be created, the format they will use (covered in the sections :ref:`admin-monitoring-logging-format-standard` and :ref:`admin-monitoring-logging-format-custom`), any filters which may be @@ -77,11 +77,10 @@ applied to events before logging them, and when or how the logs will be rolled. Log Filters =========== -:ref:`LogFilter` objects, configured in :file:`logging.config` allow you to -create filters, which may be applied to :ref:`LogObject` definitions, limiting -the types of entries which will be included in the log output. This may be -useful if your |TS| nodes receive many events which you have no need to log or -analyze. +Filters, configured in :file:`logging.config` allow you to create filters, +which may be applied to log definitions, limiting the types of entries which +will be included in the log output. This may be useful if your |TS| nodes +receive many events which you have no need to log or analyze. .. _admin-monitoring-logging-format-standard: @@ -94,7 +93,7 @@ variety of off-the-shelf log-analysis packages. You should use one of the standard event log formats unless you need information that these formats do not provide. -These formats may be used by editing the ``LogObject`` entry in :file:`logging.config`. +These formats may be used by editing :file:`logging.config`. .. _admin-logging-format-squid: @@ -248,38 +247,42 @@ Defining a Format ----------------- Custom logging formats in |TS| are defined by editing :file:`logging.config` -and adding new :ref:`LogFormat` entries for each format you wish to define. The -syntax is fairly simple: every :ref:`LogFormat` element should contain at least -two child elements (additional elements are used for features such as log -summarization and are covered elsewhere): +and adding new format entries for each format you wish to define. The syntax is +fairly simple: every format must contain a ``Format`` attribute, which is the +string defining the format of each line in the log, and may also contain an +optional ``Interval`` attribute defining the log aggregation interval for +any logs which use the format (see :ref:`admin-monitoring-logging-summary-logs` +for more information). -- A ``<Name>`` which contains an arbitrary string (using only the allowed - characters: ``[a-z0-9]``) naming your custom format. +The return value from the ``format`` function is the log format object which +may then be supplied to the appropriate ``log.*`` functions that define your +logging destinations. -- A ``<Format>`` which defines the fields that will populate each entry in the - custom logs, as well as the order in which they appear. +A very simple exampe, which contains only the timestamp of when the event began +and the canonical URL of the request, would look like: -A very simple example format, which contains only the timestamp of when the -event began and the canonical URL of the request, and named *myformat* would -be written as follows:: +.. code:: lua - <LogFormat> - <Name = "myformat"/> - <Format = "%<cqtq> %<cauc>"/> - </LogFormat> + myformat = format { + Format = "%<cqtq> %<cauc>" + } You may include as many custom field codes as you wish. The full list of codes available can be found in :ref:`custom-logging-fields`. You may also include any literal characters in your format. For example, if we wished to separate the timestamp and canonical URL in our customer format above with a slash instead of a space, or even a slash surrounded by spaces, we could do so by -just adding the desired characters to the format string:: +just adding the desired characters to the format string: - %<cqtq> / %<cauc> +.. code:: lua + + myformat = format { + Format = "%<cqtq> / %<cauc>" + } You may define as many custom formats as you wish. To apply changes to custom -formats, you will need to run the command :option:`traffic_ctl config reload` after -saving your changes to :file:`logging.config`. +formats, you will need to run the command :option:`traffic_ctl config reload` +after saving your changes to :file:`logging.config`. .. _custom-logging-fields: diff --git a/doc/admin-guide/monitoring/logging/managing-logs.en.rst b/doc/admin-guide/monitoring/logging/managing-logs.en.rst index 26af4fb..b5d8ecc 100644 --- a/doc/admin-guide/monitoring/logging/managing-logs.en.rst +++ b/doc/admin-guide/monitoring/logging/managing-logs.en.rst @@ -228,10 +228,10 @@ they reach a certain size, adjust the following settings in changes. You can fine-tune log file rolling settings for a custom log file in the -:ref:`LogObject` specification in :file:`logging.config`. The custom log file -uses the rolling settings in its :ref:`LogObject`, which override the default -settings you specify in Traffic Manager or :file:`records.config` described -above. +``log.*`` specification in :file:`logging.config`. The custom log file uses the +rolling settings provided in the relevant ``log`` function call, which override +the default settings you specify in Traffic Manager or :file:`records.config` +described above. .. _admin-monitoring-logging-host-split: diff --git a/doc/admin-guide/monitoring/logging/summary-logs.en.rst b/doc/admin-guide/monitoring/logging/summary-logs.en.rst index 949fa71..4e51b9e 100644 --- a/doc/admin-guide/monitoring/logging/summary-logs.en.rst +++ b/doc/admin-guide/monitoring/logging/summary-logs.en.rst @@ -28,8 +28,8 @@ SQL-like aggregate operators, you can configure |TS| to create summary log files that summarize a set of log entries over a specified period of time. This can significantly reduce the size of the log files generated. -To generate a summary log file, create a :ref:`LogFormat` object in the -XML-based logging configuration :file:`logging.config` using the SQL-like +To generate a summary log file, create a ``format`` object in the +custom logging configuration :file:`logging.config` using the SQL-like aggregate operators below. You can apply each of these operators to specific fields, over a specified interval. @@ -39,42 +39,63 @@ fields, over a specified interval. - ``FIRST`` - ``LAST`` +Creating Summary Log Formats +============================ + To create a summary log file format: -#. Define the format of the log file in :file:`logging.config` as follows:: +#. Define the format of the log file in :file:`logging.config` as follows: + + .. code:: lua - <LogFormat> - <Name = "summary"/> - <Format = "%<operator(field)> : %<operator(field)>"/> - <Interval = "n"/> - </LogFormat> + mysummary = format { + Format = "%<operator(field)> : %<operator(field)>", + Interval = "n" + } - Where ``operator`` is one of the five aggregate operators (``COUNT``, - ``SUM``, ``AVERAGE``, ``FIRST``, ``LAST``); ``field`` is the logging field - you want to aggregate; and ``n`` is the interval (in seconds) between - summary log entries. + Where ``operator`` is one of the five aggregate operators detailed earlier + and ``field`` is the logging field you want to aggregate. For the interval, + ``n`` is the interval (in seconds) between summary log entries. You can specify more than one ``operator`` in the format line. For more information, refer to :file:`logging.config`. -#. Run the command :option:`traffic_ctl config reload` to apply configuration changes . +#. Run the command :option:`traffic_ctl config reload` to apply the changes. + +Mixing Normal Fields and Aggregates +=================================== + +It is important to note that you cannot create custom formats for summary logs +which mix both normal, unadorned fields and aggregate functions. Your custom +format must be all of one, or all of the other. In other words, the following +attempt would not work: + +.. code:: lua + + wontwork = format { + Format = "%<LAST(cqts)> : %<COUNT(*)> : %<SUM(psql)> : %<cqu>", + Interval = "10" + } + +Because it attempts to use the last, count, and sum aggregate functions at the +same time it also includes the bare, unaggregated ``cqu`` field. As each line +of the summary log may be summarized across many individual events, |TS| would +have no way of knowing which individual event's ``cqu`` should be emitted to +the log. + +Examples +======== The following example format generates one entry every 10 seconds. Each entry contains the timestamp of the last entry of the interval, a count of the number of entries seen within that 10-second interval, and the sum of all bytes sent -to the client:: - - <LogFormat> - <Name = "summary"/> - <Format = "%<LAST(cqts)> : %<COUNT(*)> : %<SUM(psql)>"/> - <Interval = "10"/> - </LogFormat> +to the client: -.. important:: - You cannot create a format specification that contains - both aggregate operators and regular fields. For example, the following - specification would be invalid: :: +.. code:: lua - <Format = "%<LAST(cqts)> : %<COUNT(*)> : %<SUM(psql)> : %<cqu>"/> + mysummary = format { + Format = "%<LAST(cqts)> : %<COUNT(*)> : %<SUM(psql)>", + Interval = "10" + } -- To stop receiving notification emails like this one, please contact "[email protected]" <[email protected]>.
