Hi,
I am able to parse IIS access logs now *BUT *STUCK in the next step where
the Telegraf 1.2.1 is unable to parse the CUSTOM_LOG Patterns, below is the
GROK pattern which works but telegraf is not able to parse the IIS logs..
any help please?
*Working GROK pattern for IIS logs is below*
----------------------------------------------------------------
%{DATESTAMP:EventTime} %{WORD:sitename} %{HOSTNAME:computername}
%{IP:hostip} %{WORD:verb} (%{URIPATH:request}|%{NOTSPACE:request}|/)
(?:%{NOTSPACE:queryparam}|-) %{NUMBER:port} (?:%{WORD:username}|-)
%{IP:clientip} %{NOTSPACE:httpversion} %{NOTSPACE:user-agent}
(?:%{NOTSPACE:cookie}|-) (?:%{NOTSPACE:referer}|-)
(?:%{HOSTNAME:host}:%{NUMBER:port}|%{HOSTNAME:host}|-) %{NUMBER:resp_code}
%{NUMBER:sub-status} %{NUMBER:win32-status} %{NUMBER:bytes-received}
%{NUMBER:bytes-sent} %{NUMBER:time-taken}
*Telegraf 1.2.1 config file updated below*
---------------------------------------------------------------
# Telegraf configuration
# Telegraf is entirely plugin driven. All metrics are gathered from the
# declared inputs, and sent to the declared outputs.
# Plugins must be declared in here to be active.
# To deactivate a plugin, comment out the name and any variables.
# Use 'telegraf -config telegraf.conf -test' to see what metrics a config
# file would generate.
# Global tags can be specified here in key="value" format.
[global_tags]
# dc = "us-east-1" # will tag all metrics with dc=us-east-1
# rack = "1a"
# Configuration for telegraf agent
[agent]
## Default data collection interval for all inputs
interval = "10s"
## Rounds collection interval to 'interval'
## ie, if interval="10s" then always collect on :00, :10, :20, etc.
round_interval = true
## Telegraf will cache metric_buffer_limit metrics for each output, and
will
## flush this buffer on a successful write.
metric_buffer_limit = 1000
## Flush the buffer whenever full, regardless of flush_interval.
flush_buffer_when_full = true
## Collection jitter is used to jitter the collection by a random amount.
## Each plugin will sleep for a random time within jitter before
collecting.
## This can be used to avoid many plugins querying things like sysfs at
the
## same time, which can have a measurable effect on the system.
collection_jitter = "0s"
## Default flushing interval for all outputs. You shouldn't set this below
## interval. Maximum flush_interval will be flush_interval + flush_jitter
flush_interval = "10s"
## Jitter the flush interval by a random amount. This is primarily to
avoid
## large write spikes for users running a large number of telegraf
instances.
## ie, a jitter of 5s and interval 10s means flushes will happen every
10-15s
flush_jitter = "0s"
## Logging configuration:
## Run telegraf in debug mode
debug = true
## Run telegraf in quiet mode
quiet = false
## Specify the log file name. The empty string means to log to stdout.
logfile = ""
## Override default hostname, if empty use os.Hostname()
hostname = ""
###############################################################################
# OUTPUTS
#
###############################################################################
# Configuration for influxdb server to send metrics to
[[outputs.influxdb]]
# The full HTTP or UDP endpoint URL for your InfluxDB instance.
# Multiple urls can be specified but it is assumed that they are part of
the same
# cluster, this means that only ONE of the urls will be written to each
interval.
# urls = ["udp://localhost:8089"] # UDP endpoint example
urls = ["http://localhost:8086";] # required
# The target database for metrics (telegraf will create it if not exists)
database = "telegraf" # required
# Precision of writes, valid values are "ns", "us" (or "µs"), "ms", "s",
"m", "h".
# note: using second precision greatly helps InfluxDB compression
precision = "s"
## Write timeout (for the InfluxDB client), formatted as a string.
## If not provided, will default to 5s. 0s means no timeout (not
recommended).
timeout = "5s"
# username = "telegraf"
# password = "metricsmetricsmetricsmetrics"
# Set the user agent for HTTP POSTs (can be useful for log
differentiation)
# user_agent = "telegraf"
# Set UDP payload size, defaults to InfluxDB UDP Client default (512
bytes)
# udp_payload = 512
###############################################################################
# INPUTS
#
###############################################################################
# Stream and parse log file(s).
[[inputs.logparser]]
## Log files to parse.
## These accept standard unix glob matching rules, but with the addition of
## ** as a "super asterisk". ie:
## /var/log/**.log ->recursively find all .log files in /var/log
## /var/log/*/*.log ->find all .log files with a parent dir in /var/log
## /var/log/apache.log -> only tail the apache log file
files = ["C:/Temp/test.log"]
## Read file from beginning.
from_beginning = true
## Parse logstash-style "grok" patterns:
## Telegraf built-in parsing patterns: https://goo.gl/dkay10
[inputs.logparser.grok]
## This is a list of patterns to check the given log file(s) for.
## Note that adding patterns here increases processing time. The most
## efficient configuration is to have one pattern per logparser.
## Other common built-in patterns are:
patterns = ["%{CUSTOM_LOG}"]
## Name of the outputted measurement name.
measurement = "CWS_logs"
## Full path(s) to custom pattern files.
## custom_pattern_files = []
## Custom patterns can also be defined here. Put one pattern per line.
custom_patterns = '''CUSTOM_LOG %{DATESTAMP:EventTime} %{WORD:sitename}
%{HOSTNAME:computername} %{IP:hostip} %{WORD:verb:tag}
(%{URIPATH:request}|%{NOTSPACE:request}|/) (?:%{NOTSPACE:queryparam}|-)
%{NUMBER:port} (?:%{WORD:username}|-) %{IP:clientip}
%{NOTSPACE:httpversion} %{NOTSPACE:user-agent} (?:%{NOTSPACE:cookie}|-)
(?:%{NOTSPACE:referer}|-)
(?:%{HOSTNAME:host}:%{NUMBER:port}|%{HOSTNAME:host}|-)
%{NUMBER:resp_code:tag} %{NUMBER:sub-status:int} %{NUMBER:win32-status:int}
%{NUMBER:bytes-received:int} %{NUMBER:bytes-sent:int}
%{NUMBER:time-taken:int}'''
*After executing it does not parse the logs and output looks like below..
does not throw any error as well*
----------------------------------------------------------------------------------------------------------------------------------------------------------
C:\telegraf-1.2>telegraf -config LogParseCWSIIS.conf
2017-02-23T13:05:48Z D! Attempting connection to output: influxdb
2017-02-23T13:05:48Z D! Successfully connected to output: influxdb
2017-02-23T13:05:48Z I! Starting Telegraf (version 1.2.1)
2017-02-23T13:05:48Z I! Loaded outputs: influxdb
2017-02-23T13:05:48Z I! Loaded inputs: inputs.logparser
2017-02-23T13:05:48Z I! Tags enabled: host=
2017-02-23T13:05:48Z I! Agent Config: Interval:10s, Quiet:false,
Hostname:"", Flush Interval:10s
2017/02/23 08:05:48 Seeked C:/Temp/test.log - &{Offset:0 Whence:0}
2017-02-23T13:06:00Z D! Output [influxdb] buffer fullness: 0 / 1000 metrics.
2017-02-23T13:06:10Z D! Output [influxdb] buffer fullness: 0 / 1000 metrics.
2017-02-23T13:06:20Z D! Output [influxdb] buffer fullness: 0 / 1000 metrics.
2017-02-23T13:06:30Z D! Output [influxdb] buffer fullness: 0 / 1000 metrics.
2017-02-23T13:06:40Z D! Output [influxdb] buffer fullness: 0 / 1000 metrics.
2017-02-23T13:06:50Z D! Output [influxdb] buffer fullness: 0 / 1000 metrics.
Please help if I am missing something?
Thanks,
Kamal
On Thu, Feb 23, 2017 at 1:10 PM, <[email protected]> wrote:
> I'm having exactly the same issue. Any way to make this work?
--
Remember to include the version number!
---
You received this message because you are subscribed to the Google Groups
"InfluxData" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/influxdb.
To view this discussion on the web visit
https://groups.google.com/d/msgid/influxdb/CAPFArmqP00pp69Ns%2BR2oXFvBWavbuJO%2BA7kuRRwCL7CsHFYQQA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
