Hi,
some weeks ago we suddenly had some parsing issues with nxlog that we never
had before. Nxlog is constantly logging this error:
*ERROR HTTP response status is not OK: 400 Bad Request* (which refers that
the json we try to send to Elasticsearch is not well formed)
In Elasticsearch we found this exception flooding the logs:
*[2017-01-18 08:58:21,822][DEBUG][action.index ]
[ATVP6WIMMS001] failed to execute [index
{[logstash-2017.01.18][nx_iis][AVmwzGL-zYWou6-64pmC],
source[{"EventReceivedTime":"2017-01-11
10:33:26","csUser-Agent":"Mozilla/5.0+(iPhone;+CPU+iPhone+OS+10_0_2+like+Mac+OS+X)+AppleWebKit/602.1.50+(KHTML,+like+Gecko)+Mobile/14A456+[FBAN/FBIOS;FBAV/68.0.0.49.70;FBBV/41924288;FBRV/0;FBDV/iPhone7,2;FBMD/iPhone;FBSN/iOS;FBSV/10.0.2;FBSS/2;FBCR/OrangeEspa?a;FBID/phone;FBLC/en_US;FBOP/5]",
"sc-status":200}]MapperParsingException[*
*failed to parse [csUser-Agent]]; nested: JsonParseException[Invalid UTF-8
middle byte 0x61*
The error apparently is due to Spanish language character ñ which appears
in ES logs as a symbol.
Given that Spanish should be ISO 8859-1 encoding, I tried these 2
configurations without any success:
<Extension w3c>
Module xm_csv
Fields $date, $time, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query,
$s-port, $cs-username, $c-ip, $csUser-Agent, $csReferer, $sc-status,
$sc-substatus, $sc-win32-status, $sc-bytes, $cs-bytes, $time-taken
FieldTypes string, string, string, string, string, string, integer,
string, string, string, string, integer, integer, integer, integer,
integer, integer
Delimiter ' '
QuoteChar '"'
EscapeControl FALSE
UndefValue -
</Extension>
<Extension charconv>
Module xm_charconv
AutodetectCharsets utf-8, euc-jp, utf-16, utf-32, iso8859-2,
windows-1252, ansi, cp850, cp1252, windows850, iso-8859-1, iso8859-1
</Extension>
define PARSE_IIS_LOG \
if $raw_event =~ /^#/ drop(); \
else \
{ \
w3c->parse_csv(); \
$Hostname = hostname(); \
$DateEventTime = strftime(parsedate($date + " " + $time),
"%Y-%m-%dT%H:%M:%S+00:00"); \
}
<Input one>
Module im_file
File "D:\\LogFiles\\W3SVC19\\u_ex*.log"
SavePos True
ReadFromLast True
Exec convert_fields("AUTO", "UTF-8");
Exec %PARSE_IIS_LOG%;
</Input>
------------------------------------------------------------------------------------------------------------------------------
The second attempt was with this configuration instead:
<Input one>
Module im_file
File "D:\\LogFiles\\W3SVC19\\u_ex*.log"
SavePos True
ReadFromLast True
Exec %PARSE_IIS_LOG%;
Exec $raw_event = convert($raw_event, "iso8859-1", "UTF-8");
</Input>
Does anyone know what might be the issue why nxlog is not parsing properly
this character? Maybe I am using the wrong encodings?
Thanks and kind regards
Ruth
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
nxlog-ce-users mailing list
nxlog-ce-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
