Hi,
Yes, another option is to remove the offending byte:
Exec $raw_event = replace($raw_event, "\x61", "");
Regards,
Botond
On Fri, 27 Jan 2017 11:23:45 +0100
Ruth Garzón <abir...@gmail.com> wrote:
> hi!
>
> thanks. I created a dump but I was not able to find there the same log that
> is causing me issues.
> I think at this moment the error comes because it is trying to reload an
> old file which contains that character.
> The encoding in IIS for the Logging is UTF-8 and in the IISLogs the
> character is written fine: *OrangeEspaña*
>
> Is there a way to drop the character similar to if $raw_event *=~ /^#/
> drop(); *?
>
> Thanks and best regards,
> Ruth
>
>
> On Thu, Jan 19, 2017 at 11:35 AM, Botond Botyanszki <b...@nxlog.org> wrote:
>
> > Hi,
> >
> > You could dump the json into a file and check if the encoding is wrong
> > and to help figure out how to convert it correctly:
> >
> > Exec file_write("tmp/output.log",to_json() + "\n");
> >
> > BTW, there is an enhancement for xm_json to ensure that the generated
> > JSON is valid UTF-8. This is currently in the EE only but will be added
> > to the NXLog CE soon.
> >
> > Regards,
> > Botond
> >
> > On Wed, 18 Jan 2017 10:58:53 +0100
> > Ruth Garzón <abir...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > some weeks ago we suddenly had some parsing issues with nxlog that we
> > never
> > > had before. Nxlog is constantly logging this error:
> > >
> > > *ERROR HTTP response status is not OK: 400 Bad Request* (which refers
> > that
> > > the json we try to send to Elasticsearch is not well formed)
> > >
> > > In Elasticsearch we found this exception flooding the logs:
> > >
> > >
> > >
> > > *[2017-01-18 08:58:21,822][DEBUG][action.index ]
> > > [ATVP6WIMMS001] failed to execute [index
> > > {[logstash-2017.01.18][nx_iis][AVmwzGL-zYWou6-64pmC],
> > > source[{"EventReceivedTime":"2017-01-11
> > > 10:33:26","csUser-Agent":"Mozilla/5.0+(iPhone;+CPU+
> > iPhone+OS+10_0_2+like+Mac+OS+X)+AppleWebKit/602.1.50+(
> > KHTML,+like+Gecko)+Mobile/14A456+[FBAN/FBIOS;FBAV/68.0.
> > 0.49.70;FBBV/41924288;FBRV/0;FBDV/iPhone7,2;FBMD/iPhone;
> > FBSN/iOS;FBSV/10.0.2;FBSS/2;FBCR/OrangeEspa?a;FBID/phone;
> > FBLC/en_US;FBOP/5]",
> > > "sc-status":200}]MapperParsingException[*
> > >
> > >
> > > *failed to parse [csUser-Agent]]; nested: JsonParseException[Invalid
> > UTF-8
> > > middle byte 0x61*
> > > The error apparently is due to Spanish language character ñ which appears
> > > in ES logs as a symbol.
> > >
> > > Given that Spanish should be ISO 8859-1 encoding, I tried these 2
> > > configurations without any success:
> > >
> > >
> > > <Extension w3c>
> > > Module xm_csv
> > > Fields $date, $time, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query,
> > > $s-port, $cs-username, $c-ip, $csUser-Agent, $csReferer, $sc-status,
> > > $sc-substatus, $sc-win32-status, $sc-bytes, $cs-bytes, $time-taken
> > > FieldTypes string, string, string, string, string, string, integer,
> > > string, string, string, string, integer, integer, integer, integer,
> > > integer, integer
> > > Delimiter ' '
> > > QuoteChar '"'
> > > EscapeControl FALSE
> > > UndefValue -
> > > </Extension>
> > >
> > > <Extension charconv>
> > > Module xm_charconv
> > > AutodetectCharsets utf-8, euc-jp, utf-16, utf-32, iso8859-2,
> > > windows-1252, ansi, cp850, cp1252, windows850, iso-8859-1, iso8859-1
> > > </Extension>
> > >
> > > define PARSE_IIS_LOG \
> > > if $raw_event =~ /^#/ drop(); \
> > > else \
> > > { \
> > > w3c->parse_csv(); \
> > > $Hostname = hostname(); \
> > > $DateEventTime = strftime(parsedate($date + " " + $time),
> > > "%Y-%m-%dT%H:%M:%S+00:00"); \
> > > }
> > >
> > > <Input one>
> > > Module im_file
> > > File "D:\\LogFiles\\W3SVC19\\u_ex*.log"
> > > SavePos True
> > > ReadFromLast True
> > > Exec convert_fields("AUTO", "UTF-8");
> > > Exec %PARSE_IIS_LOG%;
> > > </Input>
> > >
> > >
> > > ------------------------------------------------------------
> > ------------------------------------------------------------------
> > > The second attempt was with this configuration instead:
> > >
> > > <Input one>
> > > Module im_file
> > > File "D:\\LogFiles\\W3SVC19\\u_ex*.log"
> > > SavePos True
> > > ReadFromLast True
> > > Exec %PARSE_IIS_LOG%;
> > > Exec $raw_event = convert($raw_event, "iso8859-1", "UTF-8");
> > > </Input>
> > >
> > >
> > > Does anyone know what might be the issue why nxlog is not parsing
> > properly
> > > this character? Maybe I am using the wrong encodings?
> > > Thanks and kind regards
> > > Ruth
> >
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
nxlog-ce-users mailing list
nxlog-ce-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
