[
https://issues.apache.org/jira/browse/FLINK-5013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15657042#comment-15657042
]
Tzu-Li (Gordon) Tai edited comment on FLINK-5013 at 11/11/16 1:15 PM:
----------------------------------------------------------------------
Update:
I've found the problem with the {{NoSuchMethodError - HttpConnectionParams}}.
It seems like that the {{flink-dist}} jar that comes natively with EMR 5.1.0
hasn't shaded {{httpclient}} / {{httpcore}} properly. On other EMR versions
where we need to download the Flink binaries ourselves, the class
{{HttpConnectionParams}} was correctly loaded from user jars (this is the
expected behaviour; see [1]). However, on EMR 5.1.0, it was loading the class
from the shipped {{flink-dist}} jar.
Is it possible that AWS have built Flink themselves for EMR 5.1.0, and used
Maven 3.3.x which lead to incorrect shading?
So, to be clear, users reported two different kinds of dependency issues:
(A) NoSuchMethodError:
com.amazonaws.SDKGlobalConfiguration.isInRegionOptimizedModeEnabled
(B) NoSuchMethodError:
org.apache.http.params.HttpConnectionParams.setSoKeepalive
(A) is verified to be fixed by shading all AWS dependencies on our Kinesis
connection pom (PR fixes this).
For (B), it only happens on EMR 5.1.0, regardless of whether we build the
connector with Maven 3.0.5 or 3.3.x because the Flink dist jar wasn't properly
shaded, for the suspected reason I explained above.
[1]
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kinesis-connector-classpath-issue-when-running-Flink-1-1-SNAPSHOT-on-YARN-td7611.html
was (Author: tzulitai):
Update:
I've found the problem with the {{NoSuchMethodError - HttpConnectionParams}}.
It seems like that the {{flink-dist}} jar that comes natively with EMR 5.1.0
hasn't shaded {{HttpClient}} / {{HttpCore}} properly. On other EMR versions
where we need to download the Flink binaries ourselves, the class
{{HttpConnectionParams}} was correctly loaded from user jars. However, on EMR
5.1.0, it was loading from the shipped {{flink-dist}} jar. Is it possible that
AWS have built Flink themselves for EMR 5.1.0, and used Maven 3.3.x which lead
to incorrect shading?
So, to be clear, users reported two different kinds of dependency issues:
(A) NoSuchMethodError:
com.amazonaws.SDKGlobalConfiguration.isInRegionOptimizedModeEnabled
(B) NoSuchMethodError:
org.apache.http.params.HttpConnectionParams.setSoKeepalive
(A) is verified to be fixed by shading all AWS dependencies on our Kinesis
connection pom (PR fixes this). For (B), it only happens on EMR 5.1.0,
regardless of whether we build the connector with Maven 3.0.5 or 3.3.x, for the
suspected reason I explained above.
> Flink Kinesis connector doesn't work on old EMR versions
> --------------------------------------------------------
>
> Key: FLINK-5013
> URL: https://issues.apache.org/jira/browse/FLINK-5013
> Project: Flink
> Issue Type: Bug
> Components: Kinesis Connector
> Reporter: Robert Metzger
> Assignee: Tzu-Li (Gordon) Tai
> Priority: Critical
>
> A user reported on the mailing list that our Kinesis connector doesn't work
> with EMR 4.4.0:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kinesis-Connector-Dependency-Problems-td9790.html
> The problem seems to be that Flink is loading older libraries from the "YARN
> container classpath", which on EMR contains the default Amazon libraries.
> We should try to shade kinesis and its amazon dependencies into a different
> namespace.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
