[
https://issues.apache.org/jira/browse/FLUME-2126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095200#comment-14095200
]
Andrew Sammut commented on FLUME-2126:
--------------------------------------
I'm interested in this functionality making it into an upcoming release,
however, as stated earlier it must contain some form of exception handling. The
incoming field type must match the existing field, and if not drop the record
(or alter it on the fly). I've seen this fail first hand and have resolved it
by either changing the Elasticsearch mapping or using morphline to adjust the
field type. Neither of those situations is ideal.
> Problem in elasticsearch sink when the event body is a complex field
> --------------------------------------------------------------------
>
> Key: FLUME-2126
> URL: https://issues.apache.org/jira/browse/FLUME-2126
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Environment: 1.3.1 and 1.4
> Reporter: Massimo Paladin
> Assignee: Ashish Paliwal
>
> I have found a bug in the elasticsearch sink, the problem is in the
> {{ContentBuilderUtil.addComplexField}} method, when it does
> {{builder.field(fieldName, tmp);}} the {{tmp}} object is taken as {{Object}}
> with the result of being serialized with the {{toString}} method in the
> {{XContentBuilder}}. In the end you get the object reference as content.
> The following change workaround the problem for me, the bad point is that it
> has to parse the content twice, I guess there is a better way to solve the
> problem but I am not an elasticsearch api expert.
> {code}
> ---
> a/flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ContentBuilderUtil.java
> +++
> b/flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ContentBuilderUtil.java
> @@ -61,7 +61,12 @@ public class ContentBuilderUtil {
> parser = XContentFactory.xContent(contentType).createParser(data);
> parser.nextToken();
> tmp.copyCurrentStructure(parser);
> - builder.field(fieldName, tmp);
> +
> + // if it is a valid structure then we include it
> + parser = XContentFactory.xContent(contentType).createParser(data);
> + parser.nextToken();
> + builder.field(fieldName);
> + builder.copyCurrentStructure(parser);
> } catch (JsonParseException ex) {
> // If we get an exception here the most likely cause is nested JSON
> that
> // can't be figured out in the body. At this point just push it through
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
