[
https://issues.apache.org/jira/browse/FLUME-2015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644371#comment-13644371
]
Tim Bacon commented on FLUME-2015:
----------------------------------
Thanks for the positive feedback! I've attached a link to the [Code Review
Board|https://reviews.apache.org/r/10835/] at the top of the page.
In reply to your comments:
* My (non-logging-related) use case has no need for the timestamp to be
provided as a header on every event, which is why I didn't add it. If that
makes it unusable for your LogStash use case I can always add it back in such a
way as to avoid the warnings in my own code path...
* There were 2 reasons why I wrapped the incoming event rather than adding the
timestamp as another param
** I was striving to keep backwards-compatibility (speaking as someone who has
been burned by the lack of it in the past!) and the contract with existing
clients is based only on the event param
** The timestamp used to build the index name should match the timestamp value
that goes into the index, and it's only at the level above
{{ElasticSearchLogStashEventSerializer}} that consistency can be ensured
* Having said that I understand your concern about GC -- and, if it helps, I
haven't noticed any performance hit while running the new code in my own flume
test environment (with several million events per day)
Rgds,
Tim
> ElasticSearchSink: need access to IndexRequestBuilder instance during flume
> event processing
> --------------------------------------------------------------------------------------------
>
> Key: FLUME-2015
> URL: https://issues.apache.org/jira/browse/FLUME-2015
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Affects Versions: v1.3.0, v1.3.1
> Reporter: Tim Bacon
> Attachments: FLUME-2015-1.4.0-0.patch, FLUME-2015-doc-1.4.0-0.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> I need more control over the indexing performed by the {{ElasticSearchSink}}
> -- in particular I need access to an {{IndexRequestBuilder}} instance during
> flume event processing. The interactions between {{ElasticSearchSink}} and
> {{ElasticSearchEventSerializer}} currently do not make this possible.
> I have authored a patch that meets my needs and maintains
> backwards-compatibility for existing users of the sink. It is available at
> [https://github.com/prime8/flume/commit/1056c129b10c95cee50e0c3f77e309668f82bfc6]
> -- please let me know if it can be pulled into the main ASF source tree!
> Thanks :-)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
