[
https://issues.apache.org/jira/browse/FLINK-8628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362680#comment-16362680
]
Aljoscha Krettek commented on FLINK-8628:
-----------------------------------------
I think the issues mentioned in those mail threads have been fixed as part of
FLINK-8406. Your issue seems to be different? How are you building your user
jar? Does it include Hadoop classes? Do you have the shaded hadoop jar still in
the lib folder of Flink? I'm guessing you're running on YARN on EMR? If yes,
can you try removing the shaded hadoop jar from the lib folder of Flink?
> BucketingSink does not work with S3
> -----------------------------------
>
> Key: FLINK-8628
> URL: https://issues.apache.org/jira/browse/FLINK-8628
> Project: Flink
> Issue Type: Bug
> Components: FileSystem, Streaming
> Affects Versions: 1.4.0
> Reporter: dejan miljkovic
> Priority: Critical
> Fix For: 1.5.0
>
>
> BucketingSink does not work wit S3. Followed instructions provided on
> [https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/deployment/aws.html]
> but got below exception. Several people are complaining on the same issue.
> [http://mail-archives.apache.org/mod_mbox/flink-user/201801.mbox/%3CCADAFrT9T6WQa25HXR1z1NaL=n8wP9s7aSXxZWxHy=hubo93...@mail.gmail.com%3E]
> [https://lists.apache.org/thread.html/%3CCADAFrT9T6WQa25HXR1z1NaL=n8wP9s7aSXxZWxHy=hubo93...@mail.gmail.com%3E]
> [http://mail-archives.apache.org/mod_mbox/flink-user/201801.mbox/%3CCADAFrT-i+vGe64e__=-dnu4pmpxhvyzvkfqzrhgxbeyhnwa...@mail.gmail.com%3E]
> I don't see any specific bug related to this.
>
> java.lang.RuntimeException: Error while creating FileSystem when initializing
> the state of the BucketingSink.
> at
> org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.initializeState(BucketingSink.java:358)
> at
> org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
> at
> org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
> at
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:259)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:694)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:682)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:253)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: Cannot instantiate file system for URI:
> hdfs://localhost:12345/
> at
> org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:187)
> at
> org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:401)
> at
> org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.createHadoopFileSystem(BucketingSink.java:1154)
> at
> org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.initFileSystem(BucketingSink.java:411)
> at
> org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.initializeState(BucketingSink.java:355)
> ... 9 more
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.ipc.ProtobufRpcEngine cannot be cast to
> org.apache.hadoop.ipc.RpcEngine
> at org.apache.hadoop.ipc.RPC.getProtocolEngine(RPC.java:211)
> at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:583)
> at
> org.apache.hadoop.hdfs.NameNodeProxiesClient.createNonHAProxyWithClientProtocol(NameNodeProxiesClient.java:343)
> at
> org.apache.hadoop.hdfs.NameNodeProxiesClient.createProxyWithClientProtocol(NameNodeProxiesClient.java:131)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:343)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:287)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:156)
> at
> org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:159)
> ... 13 more
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
