hi!
thanks. I created a dump but I was not able to find there the same log that
is causing me issues.
I think at this moment the error comes because it is trying to reload an
old file which contains that character.
The encoding in IIS for the Logging is UTF-8 and in the IISLogs the
character is written fine: *OrangeEspaña*
Is there a way to drop the character similar to if $raw_event *=~ /^#/
drop(); *?
Thanks and best regards,
Ruth
On Thu, Jan 19, 2017 at 11:35 AM, Botond Botyanszki <b...@nxlog.org> wrote:
> Hi,
>
> You could dump the json into a file and check if the encoding is wrong
> and to help figure out how to convert it correctly:
>
> Exec file_write("tmp/output.log",to_json() + "\n");
>
> BTW, there is an enhancement for xm_json to ensure that the generated
> JSON is valid UTF-8. This is currently in the EE only but will be added
> to the NXLog CE soon.
>
> Regards,
> Botond
>
> On Wed, 18 Jan 2017 10:58:53 +0100
> Ruth Garzón <abir...@gmail.com> wrote:
>
> > Hi,
> >
> > some weeks ago we suddenly had some parsing issues with nxlog that we
> never
> > had before. Nxlog is constantly logging this error:
> >
> > *ERROR HTTP response status is not OK: 400 Bad Request* (which refers
> that
> > the json we try to send to Elasticsearch is not well formed)
> >
> > In Elasticsearch we found this exception flooding the logs:
> >
> >
> >
> > *[2017-01-18 08:58:21,822][DEBUG][action.index ]
> > [ATVP6WIMMS001] failed to execute [index
> > {[logstash-2017.01.18][nx_iis][AVmwzGL-zYWou6-64pmC],
> > source[{"EventReceivedTime":"2017-01-11
> > 10:33:26","csUser-Agent":"Mozilla/5.0+(iPhone;+CPU+
> iPhone+OS+10_0_2+like+Mac+OS+X)+AppleWebKit/602.1.50+(
> KHTML,+like+Gecko)+Mobile/14A456+[FBAN/FBIOS;FBAV/68.0.
> 0.49.70;FBBV/41924288;FBRV/0;FBDV/iPhone7,2;FBMD/iPhone;
> FBSN/iOS;FBSV/10.0.2;FBSS/2;FBCR/OrangeEspa?a;FBID/phone;
> FBLC/en_US;FBOP/5]",
> > "sc-status":200}]MapperParsingException[*
> >
> >
> > *failed to parse [csUser-Agent]]; nested: JsonParseException[Invalid
> UTF-8
> > middle byte 0x61*
> > The error apparently is due to Spanish language character ñ which appears
> > in ES logs as a symbol.
> >
> > Given that Spanish should be ISO 8859-1 encoding, I tried these 2
> > configurations without any success:
> >
> >
> > <Extension w3c>
> > Module xm_csv
> > Fields $date, $time, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query,
> > $s-port, $cs-username, $c-ip, $csUser-Agent, $csReferer, $sc-status,
> > $sc-substatus, $sc-win32-status, $sc-bytes, $cs-bytes, $time-taken
> > FieldTypes string, string, string, string, string, string, integer,
> > string, string, string, string, integer, integer, integer, integer,
> > integer, integer
> > Delimiter ' '
> > QuoteChar '"'
> > EscapeControl FALSE
> > UndefValue -
> > </Extension>
> >
> > <Extension charconv>
> > Module xm_charconv
> > AutodetectCharsets utf-8, euc-jp, utf-16, utf-32, iso8859-2,
> > windows-1252, ansi, cp850, cp1252, windows850, iso-8859-1, iso8859-1
> > </Extension>
> >
> > define PARSE_IIS_LOG \
> > if $raw_event =~ /^#/ drop(); \
> > else \
> > { \
> > w3c->parse_csv(); \
> > $Hostname = hostname(); \
> > $DateEventTime = strftime(parsedate($date + " " + $time),
> > "%Y-%m-%dT%H:%M:%S+00:00"); \
> > }
> >
> > <Input one>
> > Module im_file
> > File "D:\\LogFiles\\W3SVC19\\u_ex*.log"
> > SavePos True
> > ReadFromLast True
> > Exec convert_fields("AUTO", "UTF-8");
> > Exec %PARSE_IIS_LOG%;
> > </Input>
> >
> >
> > ------------------------------------------------------------
> ------------------------------------------------------------------
> > The second attempt was with this configuration instead:
> >
> > <Input one>
> > Module im_file
> > File "D:\\LogFiles\\W3SVC19\\u_ex*.log"
> > SavePos True
> > ReadFromLast True
> > Exec %PARSE_IIS_LOG%;
> > Exec $raw_event = convert($raw_event, "iso8859-1", "UTF-8");
> > </Input>
> >
> >
> > Does anyone know what might be the issue why nxlog is not parsing
> properly
> > this character? Maybe I am using the wrong encodings?
> > Thanks and kind regards
> > Ruth
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
nxlog-ce-users mailing list
nxlog-ce-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
