[
https://issues.apache.org/jira/browse/FLINK-17989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17118757#comment-17118757
]
Israel Ekpo commented on FLINK-17989:
-------------------------------------
[~maguowei], thanks for the follow up and letting me know. So, in summary, it
seems that the Azure Filesystems are not yet supported in the
StreamingFileSink. I need to discuss with the community to see what is needed
to add support for it.
> java.lang.NoClassDefFoundError:
> org/apache/flink/fs/azure/common/hadoop/HadoopRecoverableWriter
> -----------------------------------------------------------------------------------------------
>
> Key: FLINK-17989
> URL: https://issues.apache.org/jira/browse/FLINK-17989
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, FileSystems
> Affects Versions: 1.9.3, 1.10.1, 1.11.0, 1.11.1
> Environment: Ubuntu 18
> Java 1.8
> Flink 1.9.x and 1.10.x
> Reporter: Israel Ekpo
> Priority: Critical
>
> In the POM.xml classes from certain packages are relocated and filtered out
> of the final jar In the POM.xml classes from certain packages are relocated
> and filtered out of the final jar
> This is causing errors for customers and users that are using the
> StreamingFileSink with Azure Blob Storage in Flink versions 1.9.x, 1.10.x and
> possibly 1.11.x
> [https://github.com/apache/flink/blob/release-1.9/flink-filesystems/flink-azure-fs-hadoop/pom.xml#L170|https://github.com/apache/flink/blob/release-1.9/flink-filesystems/flink-azure-fs-hadoop/pom.xml#L170https://github.com/apache/flink/blob/release-1.9/flink-filesystems/flink-fs-hadoop-shaded/pom.xml#L233]
> [https://github.com/apache/flink/blob/release-1.9/flink-filesystems/flink-fs-hadoop-shaded/pom.xml#L233|https://github.com/apache/flink/blob/release-1.9/flink-filesystems/flink-azure-fs-hadoop/pom.xml#L170https://github.com/apache/flink/blob/release-1.9/flink-filesystems/flink-fs-hadoop-shaded/pom.xml#L233]
>
> I would like to know why the relocation is happening and the reasoning behind
> the exclusion and filtering of the classes
> It seems to affect just the Azure file systems in my sample implementations
>
> {code:java}
> String outputPath =
> "wasbs://[email protected]/streaming-output/";
> final StreamingFileSink<Integer> sink = StreamingFileSink .forRowFormat(new
> Path(outputPath), new SimpleStringEncoder<Integer>("UTF-8")) .build();
> stream.addSink(sink);
> // execute programenv.execute(StreamingJob.class.getCanonicalName());
> {code}
> {code:java}
> //Exception Details Below
> 2020-05-27 17:23:16java.lang.NoClassDefFoundError:
> org/apache/flink/fs/azure/common/hadoop/HadoopRecoverableWriter at
> org.apache.flink.fs.azure.common.hadoop.HadoopFileSystem.createRecoverableWriter(HadoopFileSystem.java:202)
> at
> org.apache.flink.core.fs.SafetyNetWrapperFileSystem.createRecoverableWriter(SafetyNetWrapperFileSystem.java:69)
> at
> org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.<init>(Buckets.java:112)
> at
> org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink$RowFormatBuilder.createBuckets(StreamingFileSink.java:242)
> at
> org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.initializeState(StreamingFileSink.java:327)
> at
> org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
> at
> org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
> at
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:281)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:881)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:395)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) at
> org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) at
> java.lang.Thread.run(Thread.java:748)Caused by:
> java.lang.ClassNotFoundException:
> org.apache.flink.fs.azure.common.hadoop.HadoopRecoverableWriter at
> java.net.URLClassLoader.findClass(URLClassLoader.java:382) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:418) at
> org.apache.flink.util.ChildFirstClassLoader.loadClass(ChildFirstClassLoader.java:60)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
