[
https://issues.apache.org/jira/browse/FLINK-9540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16592584#comment-16592584
]
Stephan Ewen commented on FLINK-9540:
-------------------------------------
Here is still some confusion. The shaded s3 file systems do not need to be
aligned with any Hadoop version, that is the whole point of having them ;-)
What has happened in your case is that want to use the {{flink-fs-s3-hadoop}},
which registers itself under **s3://**. Then you use **s3a://** i your program
which does not use that shaded file system, but searches through the regular
Hadoop dependencies in your classpath to find an s3a implementation. That
fails, because your Hadoop classpath has not all dependencies for hadoop's
normal s3a.
> Apache Flink 1.4.2 S3 Hadoop library for Hadoop 2.7 is built for Hadoop 2.8
> and fails
> -------------------------------------------------------------------------------------
>
> Key: FLINK-9540
> URL: https://issues.apache.org/jira/browse/FLINK-9540
> Project: Flink
> Issue Type: Bug
> Reporter: Razvan
> Priority: Blocker
>
> Download Apache Flink 1.4.2 for Hadoop 2.7 or earlier. Go to opt/ inside
> flink-s3-fs-hadoop-1.4.2.jar, open common version info .properties inside you
> get:
>
> #
> # Licensed to the Apache Software Foundation (ASF) under one
> # or more contributor license agreements. See the NOTICE file
> # distributed with this work for additional information
> # regarding copyright ownership. The ASF licenses this file
> # to you under the Apache License, Version 2.0 (the
> # "License"); you may not use this file except in compliance
> # with the License. You may obtain a copy of the License at
> #
> # [http://www.apache.org/licenses/LICENSE-2.0]
> #
> # Unless required by applicable law or agreed to in writing, software
> # distributed under the License is distributed on an "AS IS" BASIS,
> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> # See the License for the specific language governing permissions and
> # limitations under the License.
> #
> *version=2.8.1*
> revision=1e6296df38f9cd3d9581c8af58a2a03a6e4312be
> *branch=branch-2.8.1-private*
> user=vinodkv
> date=2017-06-07T21:22Z
> *url=[https://git-wip-us.apache.org/repos/asf/hadoop.git*]
> srcChecksum=60125541c2b3e266cbf3becc5bda666
> protocVersion=2.5.0
>
> This will fail to work with dependencies described in
> ([https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/deployment/aws.html)]
>
> "Depending on which file system you use, please add the following
> dependencies. You can find these as part of the Hadoop binaries in
> {{hadoop-2.7/share/hadoop/tools/lib}}:
> * {{S3AFileSystem}}:
> ** {{hadoop-aws-2.7.3.jar}}
> ** {{aws-java-sdk-s3-1.11.183.jar}} and its dependencies:
> *** {{aws-java-sdk-core-1.11.183.jar}}
> *** {{aws-java-sdk-kms-1.11.183.jar}}
> *** {{jackson-annotations-2.6.7.jar}}
> *** {{jackson-core-2.6.7.jar}}
> *** {{jackson-databind-2.6.7.jar}}
> *** {{joda-time-2.8.1.jar}}
> *** {{httpcore-4.4.4.jar}}
> *** {{httpclient-4.5.3.jar}}
> * {{NativeS3FileSystem}}:
> ** {{hadoop-aws-2.7.3.jar}}
> ** {{guava-11.0.2.jar"}}{{}}
>
> {{ I presume the build task is flawed, it should be built for Apache Hadoop
> 2.7 can someone please check it?}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
