[
https://issues.apache.org/jira/browse/FLINK-5013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15647135#comment-15647135
]
Till Rohrmann commented on FLINK-5013:
--------------------------------------
Another user reported that on EMR 5.1.0 he gets the following error when using
the {{FlinkKinesisConsumer}}:
{code}
java.lang.NoSuchMethodError:
org.apache.http.params.HttpConnectionParams.setSoKeepalive(Lorg/apache/http/params/HttpParams;Z)V
at
com.amazonaws.http.HttpClientFactory.createHttpClient(HttpClientFactory.java:96)
at
com.amazonaws.http.AmazonHttpClient.<init>(AmazonHttpClient.java:187)
at
com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:136)
at
com.amazonaws.services.kinesis.AmazonKinesisClient.<init>(AmazonKinesisClient.java:221)
at
com.amazonaws.services.kinesis.AmazonKinesisClient.<init>(AmazonKinesisClient.java:197)
at
org.apache.flink.streaming.connectors.kinesis.util.AWSUtil.createKinesisClient(AWSUtil.java:56)
at
org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.<init>(KinesisProxy.java:118)
at
org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.create(KinesisProxy.java:176)
at
org.apache.flink.streaming.connectors.kinesis.internals.KinesisDataFetcher.<init>(KinesisDataFetcher.java:188)
at
org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumer.run(FlinkKinesisConsumer.java:198)
at
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:80)
at
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:53)
at
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:56)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:266)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:585)
at java.lang.Thread.run(Thread.java:745)
{code}
This seems to be also a dependency issue.
In general it would be great if we tested connectors for these kind of
platforms for different versions.
> Flink Kinesis connector doesn't work on old EMR versions
> --------------------------------------------------------
>
> Key: FLINK-5013
> URL: https://issues.apache.org/jira/browse/FLINK-5013
> Project: Flink
> Issue Type: Bug
> Components: Kinesis Connector
> Reporter: Robert Metzger
> Priority: Critical
>
> A user reported on the mailing list that our Kinesis connector doesn't work
> with EMR 4.4.0:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kinesis-Connector-Dependency-Problems-td9790.html
> The problem seems to be that Flink is loading older libraries from the "YARN
> container classpath", which on EMR contains the default Amazon libraries.
> We should try to shade kinesis and its amazon dependencies into a different
> namespace.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
