Hi Josh,
I assume that you build the SNAPSHOT version yourself. I had similar
version conflicts for Apache HttpCore with Flink SNAPSHOT versions on EMR.
The problem is cause by a changed behavior in Maven 3.3 and following
versions.
Due to these changes, the dependency shading is not working correctly.
That's why we use Maven 3.2 to build the Flink release artifacts.
Can you check whether you used Maven 3.3 and try to downgrade to 3.2 if
that was the case?
Cheers, Fabian
2016-06-17 8:12 GMT+02:00 Tai Gordon <tzuli...@gmail.com>:
> Hi Josh,
>
> I’m looking into the problem. Seems like the connector is somehow using
> older versions of httpclient.
> Can you print the loaded class path at runtime, and check the path &
> version of the loaded httpclient / httpcore dependency?
> i.e. `classOf[HttpConnectionParams].getResource("
> HttpConnectionParams.class").toString`
>
> Also, on which commit was your kinesis connector built?
>
> Regards,
> Gordon
>
>
> On June 17, 2016 at 1:08:37 AM, Josh (jof...@gmail.com) wrote:
>
> Hey,
>
> I've been running the Kinesis connector successfully now for a couple of
> weeks, on a Flink cluster running Flink 1.0.3 on EMR 2.7.1/YARN.
>
> Today I've been trying to get it working on a cluster running the current
> Flink master (1.1-SNAPSHOT) but am running into a classpath issue when
> starting the job. This only happens when running on EMR/YARN (it's fine
> when running 1.1-SNAPSHOT locally, and when running 1.0.3 on EMR)
>
> ----
> The program finished with the following exception:
>
> java.lang.NoSuchMethodError:
> org.apache.http.params.HttpConnectionParams.setSoKeepalive(Lorg/apache/http/params/HttpParams;Z)V
> at
> com.amazonaws.http.HttpClientFactory.createHttpClient(HttpClientFactory.java:96)
> at com.amazonaws.http.AmazonHttpClient.<init>(AmazonHttpClient.java:187)
> at
> com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:136)
> at
> com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:120)
> at
> com.amazonaws.services.kinesis.AmazonKinesisClient.<init>(AmazonKinesisClient.java:157)
> at
> com.amazonaws.services.kinesis.AmazonKinesisClient.<init>(AmazonKinesisClient.java:137)
> at
> org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.<init>(KinesisProxy.java:76)
> at
> org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumer.<init>(FlinkKinesisConsumer.java:166)
> at
> org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumer.<init>(FlinkKinesisConsumer.java:140)
> at
> org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumer.<init>(FlinkKinesisConsumer.java:123)
> ---
>
> Any ideas what's going on?
>
> The job I'm deploying has httpclient 4.3.6 and httpcore 4.3.3 which I
> believe are the libraries with the HttpConnectionParams class.
>
> Thanks,
> Josh
>
>
