[
https://issues.apache.org/jira/browse/FLINK-9075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16630192#comment-16630192
]
Aljoscha Krettek commented on FLINK-9075:
-----------------------------------------
[[email protected]] The BucketingSink does not work with the s3-fs-hadoop jar.
This jar provides a self-contained implementation for the Flink {{FileSystem}}
based on Hadoop code. The BucktingSink, on the other hand, directly uses the
Hadoop {{FileSystem}} so it cannot use this one.
What exactly did you copy into the lib folder? To me it looks like there are
some other Hadoop jars in there and you get version clashes.
> BucketingSink S3 does not work on local cluster
> -----------------------------------------------
>
> Key: FLINK-9075
> URL: https://issues.apache.org/jira/browse/FLINK-9075
> Project: Flink
> Issue Type: Bug
> Components: Streaming Connectors
> Affects Versions: 1.4.2
> Reporter: dejan miljkovic
> Priority: Major
> Fix For: 1.7.0
>
>
> Trying to write to S3 using BucketingSink. Got below error when code is
> executed on local Flink 1.4.2 cluster. Code works from InteliJ. I followed
> procedure for S3 connection from documentation (copied
> flink-s3-fs-hadoop-1.4.2.jar to lib). I reported similar issues before. It
> looks that they were all related to class loading issues.
> On [https://github.com/dmiljkovic/test-flink-bucketingsink-s3] I provided
> code that produces below error. pom.xm contains more stuff than is needed. I
> just copied pom from project that need to write to S3.
>
> javax.xml.parsers.FactoryConfigurationError: Provider for class
> javax.xml.parsers.DocumentBuilderFactory cannot be created
> at
> javax.xml.parsers.FactoryFinder.findServiceProvider(FactoryFinder.java:311)
> at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:267)
> at
> javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:120)
> at
> org.apache.flink.fs.s3hadoop.shaded.org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2567)
> at
> org.apache.flink.fs.s3hadoop.shaded.org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2543)
> at
> org.apache.flink.fs.s3hadoop.shaded.org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2426)
> at
> org.apache.flink.fs.s3hadoop.shaded.org.apache.hadoop.conf.Configuration.get(Configuration.java:1240)
> at
> org.apache.flink.fs.s3hadoop.S3FileSystemFactory.create(S3FileSystemFactory.java:98)
> at
> org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:397)
> at
> org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.createHadoopFileSystem(BucketingSink.java:1126)
> at
> org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.initFileSystem(BucketingSink.java:411)
> at
> org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.initializeState(BucketingSink.java:355)
> at
> org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
> at
> org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
> at
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:258)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:694)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:682)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:253)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718)
> at java.lang.Thread.run(Thread.java:748)
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
