[
https://issues.apache.org/jira/browse/FLUME-2921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15374038#comment-15374038
]
Mike Percy commented on FLUME-2921:
-----------------------------------
Moving some discussion back to this JIRA from the code review. Based on your
investigation in the code review, Flume exposes Guava as part of its public
API, particularly in Context.java, in public ImmutableMap<String, String>
getSubProperties(String prefix). This is really unfortunate as it means that it
is not possible to shade Guava in Flume.
That means, because Flume has HDFS, Hive, and HBase sinks with transitive deps
on Guava 11, Flume core is stuck on Guava 11. We cannot upgrade Guava to
version 18, which is apparently required by the ES 2.0 client.
Short of cutting a Flume 2.0 to break API compatibility (we follow the
www.semver.org versioning standard), which we should probably do at some point
(I do not think we should do it instead of 1.7.0), I think we have all bad
options.
One bad, but potentially feasible option for people who need ES 2.x is the
following:
* Create an out-of-tree ES 2.x sink on GitHub. It would depend on Guava 18 and
pull in that dep
* When deploying the ES2 sink with Flume, ensure that the Guava 18 JAR appears
on the classpath ahead of the Guava 11 JAR
* Note that the above will likely break other sinks, like the HDFS sink, so
they could not be used in the same agent
This is quite hacky, and I'm not sure it would work without testing it, but I
think it could be made to work since ImmutableMap appears in Guava 18 and that
is the only absolutely required Guava dependency in Flume core.
I am open to other options that do not break backcompat in 1.x
On a side note, for Flume 2.x, if someone wants to tackle the custom class
loader work, that would be very interesting to help avoid these problems. I
hope we could maintain backcompat for most Flume 1.x plugins in such a world,
we would only have to break the plugins that call APIs "tainted" by Guava.
> Support Elasticsearch 2.0+
> --------------------------
>
> Key: FLUME-2921
> URL: https://issues.apache.org/jira/browse/FLUME-2921
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Reporter: Lior Zeno
> Assignee: Lior Zeno
> Fix For: v1.7.0
>
> Attachments: FLUME-2921-0.patch, FLUME-2921-1.patch
>
>
> Elasticsearch sink supports an ancient version of ES. We should make the sink
> work with newer versions of Elasticsearch.
> I attached a patch for that. Please note that this involves upgrading netty
> and guava.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
