[
https://issues.apache.org/jira/browse/FLINK-5013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15657832#comment-15657832
]
ASF GitHub Bot commented on FLINK-5013:
---------------------------------------
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/2787
I've actually found out the reason for
`HttpConnectionParams.setSoKeepalive` error, noted in the last comment in JIRA.
Cross-posting it here so we don't need to jump around two places:
It seems like that the `flink-dist` jar that comes natively with EMR 5.1.0
hasn't shaded `httpclient` / `httpcore` properly (it contains the wrong
unshaded classes). On other EMR versions where we need to download the Flink
binaries ourselves, the class `HttpConnectionParams` was correctly loaded from
user jars (this is the expected behaviour because our distributed Flink
binaries has the classes shaded in the Hadoop dependency; see [1]). However, on
EMR 5.1.0, it was loading the wrong unshaded class from their own shipped
`flink-dist` jar.
Is it possible that AWS have built Flink themselves for EMR 5.1.0, and used
Maven 3.3.x which lead to incorrect shading of the apache http classes?
[1]
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kinesis-connector-classpath-issue-when-running-Flink-1-1-SNAPSHOT-on-YARN-td7611.html
> Flink Kinesis connector doesn't work on old EMR versions
> --------------------------------------------------------
>
> Key: FLINK-5013
> URL: https://issues.apache.org/jira/browse/FLINK-5013
> Project: Flink
> Issue Type: Bug
> Components: Kinesis Connector
> Reporter: Robert Metzger
> Assignee: Tzu-Li (Gordon) Tai
> Priority: Critical
>
> A user reported on the mailing list that our Kinesis connector doesn't work
> with EMR 4.4.0:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kinesis-Connector-Dependency-Problems-td9790.html
> The problem seems to be that Flink is loading older libraries from the "YARN
> container classpath", which on EMR contains the default Amazon libraries.
> We should try to shade kinesis and its amazon dependencies into a different
> namespace.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
