Hi,
You could dump the json into a file and check if the encoding is wrong
and to help figure out how to convert it correctly:
Exec file_write("tmp/output.log",to_json() + "\n");
BTW, there is an enhancement for xm_json to ensure that the generated
JSON is valid UTF-8. This is currently in the EE only but will be added
to the NXLog CE soon.
Regards,
Botond
On Wed, 18 Jan 2017 10:58:53 +0100
Ruth Garzón <[email protected]> wrote:
> Hi,
>
> some weeks ago we suddenly had some parsing issues with nxlog that we never
> had before. Nxlog is constantly logging this error:
>
> *ERROR HTTP response status is not OK: 400 Bad Request* (which refers that
> the json we try to send to Elasticsearch is not well formed)
>
> In Elasticsearch we found this exception flooding the logs:
>
>
>
> *[2017-01-18 08:58:21,822][DEBUG][action.index ]
> [ATVP6WIMMS001] failed to execute [index
> {[logstash-2017.01.18][nx_iis][AVmwzGL-zYWou6-64pmC],
> source[{"EventReceivedTime":"2017-01-11
> 10:33:26","csUser-Agent":"Mozilla/5.0+(iPhone;+CPU+iPhone+OS+10_0_2+like+Mac+OS+X)+AppleWebKit/602.1.50+(KHTML,+like+Gecko)+Mobile/14A456+[FBAN/FBIOS;FBAV/68.0.0.49.70;FBBV/41924288;FBRV/0;FBDV/iPhone7,2;FBMD/iPhone;FBSN/iOS;FBSV/10.0.2;FBSS/2;FBCR/OrangeEspa?a;FBID/phone;FBLC/en_US;FBOP/5]",
> "sc-status":200}]MapperParsingException[*
>
>
> *failed to parse [csUser-Agent]]; nested: JsonParseException[Invalid UTF-8
> middle byte 0x61*
> The error apparently is due to Spanish language character ñ which appears
> in ES logs as a symbol.
>
> Given that Spanish should be ISO 8859-1 encoding, I tried these 2
> configurations without any success:
>
>
> <Extension w3c>
> Module xm_csv
> Fields $date, $time, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query,
> $s-port, $cs-username, $c-ip, $csUser-Agent, $csReferer, $sc-status,
> $sc-substatus, $sc-win32-status, $sc-bytes, $cs-bytes, $time-taken
> FieldTypes string, string, string, string, string, string, integer,
> string, string, string, string, integer, integer, integer, integer,
> integer, integer
> Delimiter ' '
> QuoteChar '"'
> EscapeControl FALSE
> UndefValue -
> </Extension>
>
> <Extension charconv>
> Module xm_charconv
> AutodetectCharsets utf-8, euc-jp, utf-16, utf-32, iso8859-2,
> windows-1252, ansi, cp850, cp1252, windows850, iso-8859-1, iso8859-1
> </Extension>
>
> define PARSE_IIS_LOG \
> if $raw_event =~ /^#/ drop(); \
> else \
> { \
> w3c->parse_csv(); \
> $Hostname = hostname(); \
> $DateEventTime = strftime(parsedate($date + " " + $time),
> "%Y-%m-%dT%H:%M:%S+00:00"); \
> }
>
> <Input one>
> Module im_file
> File "D:\\LogFiles\\W3SVC19\\u_ex*.log"
> SavePos True
> ReadFromLast True
> Exec convert_fields("AUTO", "UTF-8");
> Exec %PARSE_IIS_LOG%;
> </Input>
>
>
> ------------------------------------------------------------------------------------------------------------------------------
> The second attempt was with this configuration instead:
>
> <Input one>
> Module im_file
> File "D:\\LogFiles\\W3SVC19\\u_ex*.log"
> SavePos True
> ReadFromLast True
> Exec %PARSE_IIS_LOG%;
> Exec $raw_event = convert($raw_event, "iso8859-1", "UTF-8");
> </Input>
>
>
> Does anyone know what might be the issue why nxlog is not parsing properly
> this character? Maybe I am using the wrong encodings?
> Thanks and kind regards
> Ruth
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
nxlog-ce-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
