[
https://issues.apache.org/jira/browse/FLUME-2089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687986#comment-13687986
]
Allan Feid commented on FLUME-2089:
-----------------------------------
This probably has something to do with character encoding, but as far as I know
all logs are utf-8 encoded. The problem is that there are 100s of log sources
which generate logs based on web traffic, among other things. I can't
absolutely be sure there isn't some random encoded characters floating around.
It is interesting that the xContentType method in ES detects some of my events
as YAML though, I can be fairly confident there's no YAML in my log messages.
I'll look into adding a patch on reviews.apache.org, and be sure to remove the
0.90.1 upgrade.
> ElasticSearchSink raises YAMLException when event body has unexpected
> encoding.
> -------------------------------------------------------------------------------
>
> Key: FLUME-2089
> URL: https://issues.apache.org/jira/browse/FLUME-2089
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.4.0, v1.3.1
> Reporter: Edward Sargisson
>
> Detected by Allan Feid and documented on the user list
> http://mail-archives.apache.org/mod_mbox/flume-user/201306.mbox/%3CCAN94UWe6UvcOKT1S%2BXANC-sy0qFsxet3RJY9PVkj-eSfO5fk6Q%40mail.gmail.com%3E
> Steps:
> Send an event with the body as follows:
> foo¤data¤1371126476.436¤0.005¤555¤10.1.1.1¤HTTP/1.1¤GET¤http¤vhost¤/path/url¤¤-¤200¤
> referrer.com/search/?query=\x8D\x91\x89\xEF\x8Bc\x8E\x96\x93\xB0¤-¤-¤-
> Expected Results:
> The event is stored in elasticsearch.
> Actual Results:
> >> 10 Jun 2013 09:52:34,360 ERROR
> >> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> >> (org.apache.flume.SinkRunner$PollingRunner.run:160) - Unable to deliver
> >> event. Exception follows.
> >> org.elasticsearch.common.jackson.dataformat.yaml.snakeyaml.error.YAMLException:
> >> java.io.CharConversionException: Invalid UTF-8 start byte 0xfc (at char
> >> #81, byte #-1)
> >> at
> >> org.elasticsearch.common.jackson.dataformat.yaml.snakeyaml.reader.StreamReader.update(StreamReader.java:198)
> >> at
> >> org.elasticsearch.common.jackson.dataformat.yaml.snakeyaml.reader.StreamReader.<init>(StreamReader.java:62)
> >> at
> >> org.elasticsearch.common.jackson.dataformat.yaml.YAMLParser.<init>(YAMLParser.java:147)
> >> at
> >> org.elasticsearch.common.jackson.dataformat.yaml.YAMLFactory._createParser(YAMLFactory.java:530)
> >> at
> >> org.elasticsearch.common.jackson.dataformat.yaml.YAMLFactory.createJsonParser(YAMLFactory.java:420)
> >> at
> >> org.elasticsearch.common.xcontent.yaml.YamlXContent.createParser(YamlXContent.java:83)
> >> at
> >> org.apache.flume.sink.elasticsearch.ContentBuilderUtil.addComplexField(ContentBuilderUtil.java:61)
> >> at
> >> org.apache.flume.sink.elasticsearch.ContentBuilderUtil.appendField(ContentBuilderUtil.java:47)
> >> at
> >> org.apache.flume.sink.elasticsearch.ElasticSearchLogStashEventSerializer.appendBody(ElasticSearchLogStashEventSerializer.java:87)
> >> at
> >> org.apache.flume.sink.elasticsearch.ElasticSearchLogStashEventSerializer.getContentBuilder(ElasticSearchLogStashEventSerializer.java:79)
> >> at
> >> org.apache.flume.sink.elasticsearch.ElasticSearchSink.process(ElasticSearchSink.java:178)
> >> at
> >> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> >> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> >> at java.lang.Thread.run(Thread.java:662)
> >> Caused by: java.io.CharConversionException: Invalid UTF-8 start byte 0xfc
> >> (at char #81, byte #-1)
> >> at
> >> org.elasticsearch.common.jackson.dataformat.yaml.UTF8Reader.reportInvalidInitial(UTF8Reader.java:395)
> >> at
> >> org.elasticsearch.common.jackson.dataformat.yaml.UTF8Reader.read(UTF8Reader.java:247)
> >> at
> >> org.elasticsearch.common.jackson.dataformat.yaml.UTF8Reader.read(UTF8Reader.java:157)
> >> at
> >> org.elasticsearch.common.jackson.dataformat.yaml.snakeyaml.reader.StreamReader.update(StreamReader.java:182)
> >> ... 13 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
