[
https://issues.apache.org/jira/browse/FLINK-9540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16505955#comment-16505955
]
Razvan commented on FLINK-9540:
-------------------------------
Hi,
This is the first exception:
2018-05-18 17:37:20.986 [main] WARN
org.apache.flink.configuration.Configuration - Config uses deprecated
configuration key 'high-availability.zookeeper.storageDir' instead of proper
key 'high-availability.storageDir'
2018-05-18 17:37:21.239 [main] ERROR
org.apache.flink.runtime.taskmanager.TaskManager - Failed to run TaskManager.
java.io.IOException: Could not create FileSystem for highly available storage
(high-availability.storageDir)
at
org.apache.flink.runtime.blob.BlobUtils.createFileSystemBlobStore(BlobUtils.java:119)
at
org.apache.flink.runtime.blob.BlobUtils.createBlobStoreFromConfig(BlobUtils.java:92)
at
org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:103)
at
org.apache.flink.runtime.taskmanager.TaskManager$.selectNetworkInterfaceAndRunTaskManager(TaskManager.scala:1681)
at
org.apache.flink.runtime.taskmanager.TaskManager$$anon$2.call(TaskManager.scala:1592)
at
org.apache.flink.runtime.taskmanager.TaskManager$$anon$2.call(TaskManager.scala:1590)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
at
org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at
org.apache.flink.runtime.taskmanager.TaskManager$.main(TaskManager.scala:1590)
at org.apache.flink.runtime.taskmanager.TaskManager.main(TaskManager.scala)
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could
not find a file system implementation for scheme 's3a'. The scheme is not
directly supported by Flink and no Hadoop file system to support this scheme
could be loaded.
at
org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:405)
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:320)
at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293)
at
org.apache.flink.runtime.blob.BlobUtils.createFileSystemBlobStore(BlobUtils.java:116)
... 11 common frames omitted
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException:
Cannot support file system for 's3a' via Hadoop, because Hadoop is not in the
classpath, or some classes are missing from the classpath.
at
org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:179)
at
org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:401)
... 14 common frames omitted
Caused by: java.lang.NoClassDefFoundError:
org/apache/http/HttpEntityEnclosingRequest
at com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:119)
at com.amazonaws.services.s3.AmazonS3Client.<init>(AmazonS3Client.java:389)
at com.amazonaws.services.s3.AmazonS3Client.<init>(AmazonS3Client.java:371)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:235)
at
org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:159)
... 15 common frames omitted
Caused by: java.lang.ClassNotFoundException:
org.apache.http.HttpEntityEnclosingRequest
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 20 common frames omitted
Then I started to add HttpClient and other libs
> Apache Flink 1.4.2 S3 Hadoop library for Hadoop 2.7 is built for Hadoop 2.8
> and fails
> -------------------------------------------------------------------------------------
>
> Key: FLINK-9540
> URL: https://issues.apache.org/jira/browse/FLINK-9540
> Project: Flink
> Issue Type: Bug
> Reporter: Razvan
> Priority: Blocker
>
> Download Apache Flink 1.4.2 for Hadoop 2.7 or earlier. Go to opt/ inside
> flink-s3-fs-hadoop-1.4.2.jar, open common version info .properties inside you
> get:
>
> #
> # Licensed to the Apache Software Foundation (ASF) under one
> # or more contributor license agreements. See the NOTICE file
> # distributed with this work for additional information
> # regarding copyright ownership. The ASF licenses this file
> # to you under the Apache License, Version 2.0 (the
> # "License"); you may not use this file except in compliance
> # with the License. You may obtain a copy of the License at
> #
> # [http://www.apache.org/licenses/LICENSE-2.0]
> #
> # Unless required by applicable law or agreed to in writing, software
> # distributed under the License is distributed on an "AS IS" BASIS,
> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> # See the License for the specific language governing permissions and
> # limitations under the License.
> #
> *version=2.8.1*
> revision=1e6296df38f9cd3d9581c8af58a2a03a6e4312be
> *branch=branch-2.8.1-private*
> user=vinodkv
> date=2017-06-07T21:22Z
> *url=[https://git-wip-us.apache.org/repos/asf/hadoop.git*]
> srcChecksum=60125541c2b3e266cbf3becc5bda666
> protocVersion=2.5.0
>
> This will fail to work with dependencies described in
> ([https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/deployment/aws.html)]
>
> "Depending on which file system you use, please add the following
> dependencies. You can find these as part of the Hadoop binaries in
> {{hadoop-2.7/share/hadoop/tools/lib}}:
> * {{S3AFileSystem}}:
> ** {{hadoop-aws-2.7.3.jar}}
> ** {{aws-java-sdk-s3-1.11.183.jar}} and its dependencies:
> *** {{aws-java-sdk-core-1.11.183.jar}}
> *** {{aws-java-sdk-kms-1.11.183.jar}}
> *** {{jackson-annotations-2.6.7.jar}}
> *** {{jackson-core-2.6.7.jar}}
> *** {{jackson-databind-2.6.7.jar}}
> *** {{joda-time-2.8.1.jar}}
> *** {{httpcore-4.4.4.jar}}
> *** {{httpclient-4.5.3.jar}}
> * {{NativeS3FileSystem}}:
> ** {{hadoop-aws-2.7.3.jar}}
> ** {{guava-11.0.2.jar"}}{{}}
>
> {{ I presume the build task is flawed, it should be built for Apache Hadoop
> 2.7 can someone please check it?}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
