[
https://issues.apache.org/jira/browse/FLINK-5013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15656541#comment-15656541
]
Tzu-Li (Gordon) Tai commented on FLINK-5013:
--------------------------------------------
Results of my tests:
Shading {{com.amazonaws.*}} to a different namespace solves the reported
dependency issue for 4.x (randomly tested on 4.3.0, 4.4.0, 4.7.2, 4.8.0). It
also works without issue for 5.0.3.
However, for the native Flink support in EMR 5.1.0, the {{NoSuchMethodError}}
for {{HttpConnectionParams.setSoKeepalive}} still remains, even with the
shading. It seems like somehow the connector is catching HttpClient 4.2.6
(Flink probably uses this version of HttpClient somewhere else), while the AWS
Kinesis libraries we are using uses 4.3.6. I've tried packaging the user jar
with newer HttpClient versions, but still doesn't work. Neither does packaging
it with the Flink Kinesis Connector.
Any clues what might be going on? The above was tested with Maven 3.0.5.
> Flink Kinesis connector doesn't work on old EMR versions
> --------------------------------------------------------
>
> Key: FLINK-5013
> URL: https://issues.apache.org/jira/browse/FLINK-5013
> Project: Flink
> Issue Type: Bug
> Components: Kinesis Connector
> Reporter: Robert Metzger
> Assignee: Tzu-Li (Gordon) Tai
> Priority: Critical
>
> A user reported on the mailing list that our Kinesis connector doesn't work
> with EMR 4.4.0:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kinesis-Connector-Dependency-Problems-td9790.html
> The problem seems to be that Flink is loading older libraries from the "YARN
> container classpath", which on EMR contains the default Amazon libraries.
> We should try to shade kinesis and its amazon dependencies into a different
> namespace.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
