[
https://issues.apache.org/jira/browse/FLUME-2126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14320108#comment-14320108
]
Francis edited comment on FLUME-2126 at 2/13/15 2:09 PM:
---------------------------------------------------------
I'm not sure to understand your point. The sink is already doing this content
type detection. Look at the ContentBuilderUtil:
{code}
XContentType contentType = XContentFactory.xContentType(data);
if (contentType == null) {
addSimpleField(builder, field, data);
} else {
addComplexField(builder, field, contentType, data);
}
{code}
The ES XContentFactory.xContentType() method tries to detect the data format.
Then, in the addComplexField() method, a parser corresponding to the detected
type is instantiated. The XContentFactory supports the following content types:
JSON, YAML and Smile. Everything else will be added as-is as a text field.
I'm not proposing to handle any content type. I just want the current code to
work as I think it was initially supposed to work. The sink handles the content
types that are already supported by the ES SDK and that's it. Nothing more.
was (Author: faelenor):
I'm not sure to understand your point. The sink is already doing this content
type detection. Look at the ContentBuilderUtil:
XContentType contentType = XContentFactory.xContentType(data);
if (contentType == null) {
addSimpleField(builder, field, data);
} else {
addComplexField(builder, field, contentType, data);
}
The ES XContentFactory.xContentType() method tries to detect the data format.
Then, in the addComplexField() method, a parser corresponding to the detected
type is instantiated. The XContentFactory supports the following content types:
JSON, YAML and Smile. Everything else will be added as-is as a text field.
I'm not proposing to handle any content type. I just want the current code to
work as I think it was initially supposed to work. The sink handles the content
types that are already supported by the ES SDK and that's it. Nothing more.
> Problem in elasticsearch sink when the event body is a complex field
> --------------------------------------------------------------------
>
> Key: FLUME-2126
> URL: https://issues.apache.org/jira/browse/FLUME-2126
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Environment: 1.3.1 and 1.4
> Reporter: Massimo Paladin
> Assignee: Ashish Paliwal
> Attachments: FLUME-2126-0.patch
>
>
> I have found a bug in the elasticsearch sink, the problem is in the
> {{ContentBuilderUtil.addComplexField}} method, when it does
> {{builder.field(fieldName, tmp);}} the {{tmp}} object is taken as {{Object}}
> with the result of being serialized with the {{toString}} method in the
> {{XContentBuilder}}. In the end you get the object reference as content.
> The following change workaround the problem for me, the bad point is that it
> has to parse the content twice, I guess there is a better way to solve the
> problem but I am not an elasticsearch api expert.
> {code}
> ---
> a/flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ContentBuilderUtil.java
> +++
> b/flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ContentBuilderUtil.java
> @@ -61,7 +61,12 @@ public class ContentBuilderUtil {
> parser = XContentFactory.xContent(contentType).createParser(data);
> parser.nextToken();
> tmp.copyCurrentStructure(parser);
> - builder.field(fieldName, tmp);
> +
> + // if it is a valid structure then we include it
> + parser = XContentFactory.xContent(contentType).createParser(data);
> + parser.nextToken();
> + builder.field(fieldName);
> + builder.copyCurrentStructure(parser);
> } catch (JsonParseException ex) {
> // If we get an exception here the most likely cause is nested JSON
> that
> // can't be figured out in the body. At this point just push it through
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
