[
https://issues.apache.org/jira/browse/FLINK-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15938990#comment-15938990
]
Scott Kidder commented on FLINK-6176:
-------------------------------------
The ordering of {{FLINK_CLASSPATH}} entries affected classloader
prioritization, which is the reason for the {{NoClassDefFoundError}} I hit.
I'll provide some more specifics.
*Old hosts* that *work* have the following profile:
|Base OS|Ubuntu 14.04.3 LTS (GNU/Linux 3.13.0-74-generic x86_64)|
|Kernel|Linux ip-10-55-2-175 3.13.0-74-generic #118-Ubuntu SMP Thu Dec 17
22:52:10 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux|
|Docker Version| Docker version 1.11.1, build 5604cbe|
|Calculated
{{FLINK_CLASSPATH}}|/usr/local/flink-1.2.0/lib/egads-0.1.jar:/usr/local/flink-1.2.0/lib/flink-metrics-statsd-1.2-SNAPSHOT.jar:/usr/local/flink-1.2.0/lib/log4j-1.2.17.jar:/usr/local/flink-1.2.0/lib/flink-python_2.11-1.2-SNAPSHOT.jar:/usr/local/flink-1.2.0/lib/slf4j-log4j12-1.7.7.jar:/usr/local/flink-1.2.0/lib/flink-connector-rabbitmq_2.11-1.2-SNAPSHOT.jar:/usr/local/flink-1.2.0/lib/flink-connector-kinesis_2.11-1.2-SNAPSHOT.jar:/usr/local/flink-1.2.0/lib/flink-dist_2.11-1.2-SNAPSHOT.jar:::|
*New Hosts* that *do not work* have the following profile:
|Base OS|Ubuntu 14.04.5 LTS (GNU/Linux 3.13.0-112-generic x86_64)|
|Kernel|Linux ip-10-55-3-137 3.13.0-112-generic #159-Ubuntu SMP Fri Mar 3
15:26:07 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux|
|Docker Version| Docker version 17.03.0-ce, build 3a232c8|
|Calculated
{{FLINK_CLASSPATH}}|/usr/local/flink-1.2.0/lib/egads-0.1.jar:/usr/local/flink-1.2.0/lib/log4j-1.2.17.jar:/usr/local/flink-1.2.0/lib/flink-connector-rabbitmq_2.11-1.2-SNAPSHOT.jar:/usr/local/flink-1.2.0/lib/flink-python_2.11-1.2-SNAPSHOT.jar:/usr/local/flink-1.2.0/lib/flink-dist_2.11-1.2-SNAPSHOT.jar:/usr/local/flink-1.2.0/lib/flink-metrics-statsd-1.2-SNAPSHOT.jar:/usr/local/flink-1.2.0/lib/slf4j-log4j12-1.7.7.jar:/usr/local/flink-1.2.0/lib/flink-connector-kinesis_2.11-1.2-SNAPSHOT.jar:::|
The sizes & timestamps for all JARs were identical. But note the difference in
ordering of Classpath entries. The Kinesis JAR file contains shaded
dependencies, including a newer version of Apache HTTP Client than what's
included in the Flink distribution JAR.
The *new host* produced a {{FLINK_CLASSPATH}} with the `flink-dist` JAR in the
middle of the classpath, ahead of the Kinesis JAR. This led to the older HTTP
Client bundled with the Flink distribution JAR taking precedence, and then
being unable to tie back to the AWS classes. This difference in ordering led to
the following exception being thrown when my application that uses the Flink
Kinesis Streaming Connector:
{code}
java.lang.NoClassDefFoundError: Could not initialize class
com.amazonaws.http.conn.ssl.SdkTLSSocketFactory
at
com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.getPreferredSocketFactory(ApacheConnectionManagerFactory.java:87)
at
com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.create(ApacheConnectionManagerFactory.java:65)
at
com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.create(ApacheConnectionManagerFactory.java:58)
at
com.amazonaws.http.apache.client.impl.ApacheHttpClientFactory.create(ApacheHttpClientFactory.java:51)
at
com.amazonaws.http.apache.client.impl.ApacheHttpClientFactory.create(ApacheHttpClientFactory.java:39)
at
com.amazonaws.http.AmazonHttpClient.<init>(AmazonHttpClient.java:319)
at
com.amazonaws.http.AmazonHttpClient.<init>(AmazonHttpClient.java:303)
at
com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:165)
at
com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:154)
at
com.amazonaws.services.kinesis.AmazonKinesisClient.<init>(AmazonKinesisClient.java:243)
at
com.amazonaws.services.kinesis.AmazonKinesisClient.<init>(AmazonKinesisClient.java:218)
at
org.apache.flink.streaming.connectors.kinesis.util.AWSUtil.createKinesisClient(AWSUtil.java:56)
at
org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.<init>(KinesisProxy.java:121)
at
org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.create(KinesisProxy.java:179)
at
org.apache.flink.streaming.connectors.kinesis.internals.KinesisDataFetcher.<init>(KinesisDataFetcher.java:188)
at
org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumer.run(FlinkKinesisConsumer.java:198)
at
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:78)
at
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:55)
at
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:56)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:262)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
at java.lang.Thread.run(Thread.java:745)
{code}
Here's a link to the commit I made to my fork to order additional JAR files
alphabetically, then append the Flink distribution JAR at the end:
https://github.com/muxinc/flink/commit/39a769464ada9bf481033e8889beb9bae41fb100
> Add JARs to CLASSPATH deterministically
> ---------------------------------------
>
> Key: FLINK-6176
> URL: https://issues.apache.org/jira/browse/FLINK-6176
> Project: Flink
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2.0
> Reporter: Scott Kidder
>
> The {{config.sh}} script uses the following shell-script function to build
> the {{FLINK_CLASSPATH}} variable from a listing of JAR files in the
> {{$FLINK_LIB_DIR}} directory:
> {code}
> constructFlinkClassPath() {
> while read -d '' -r jarfile ; do
> if [[ $FLINK_CLASSPATH = "" ]]; then
> FLINK_CLASSPATH="$jarfile";
> else
> FLINK_CLASSPATH="$FLINK_CLASSPATH":"$jarfile"
> fi
> done < <(find "$FLINK_LIB_DIR" ! -type d -name '*.jar' -print0)
> echo $FLINK_CLASSPATH
> }
> {code}
> The {{find}} command as specified will return files in directory-order, which
> varies by OS and filesystem.
> The inconsistent ordering of directory contents caused problems for me when
> installing a Flink Docker image onto new machine with a newer version of
> Docker and different filesystem (UFS). The differences in the Docker
> filesystem implementation led to different ordering of the directory
> contents; this affected the {{FLINK_CLASSPATH}} ordering and generated very
> puzzling {{NoClassNotFoundException}} errors when running my Flink
> application.
> This should be addressed by deterministically ordering JAR files added to the
> {{FLINK_CLASSPATH}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
