Israel Ekpo created FLINK-17989:
-----------------------------------
Summary: java.lang.NoClassDefFoundError:
org/apache/flink/fs/azure/common/hadoop/HadoopRecoverableWriter
Key: FLINK-17989
URL: https://issues.apache.org/jira/browse/FLINK-17989
Project: Flink
Issue Type: Bug
Components: Connectors / FileSystem, FileSystems
Affects Versions: 1.10.1, 1.9.3
Environment: Ubuntu 18
Java 1.8
Flink 1.9.x and 1.10.x
Reporter: Israel Ekpo
Fix For: 1.10.1, 1.9.3
In the POM.xml classes from certain packages are relocated and filtered out of
the final jar In the POM.xml classes from certain packages are relocated and
filtered out of the final jar
This is causing errors for customers and users that are using the
StreamingFileSink with Azure Blob Storage in Flink version 1.9.x, 1.10.x and
possibly 1.11.x
https://github.com/apache/flink/blob/release-1.9/flink-filesystems/flink-azure-fs-hadoop/pom.xml#L170https://github.com/apache/flink/blob/release-1.9/flink-filesystems/flink-fs-hadoop-shaded/pom.xml#L233
I would like to know why the relocation is happening and the reasoning behind
the exclusion and filtering of the classes
It seems to affect just the Azure file systems in my sample implementations
{code:java}
String outputPath =
"wasbs://[email protected]/streaming-output/";
final StreamingFileSink<Integer> sink = StreamingFileSink .forRowFormat(new
Path(outputPath), new SimpleStringEncoder<Integer>("UTF-8")) .build();
stream.addSink(sink);
// execute programenv.execute(StreamingJob.class.getCanonicalName());
{code}
2020-05-27 17:23:16java.lang.NoClassDefFoundError:
org/apache/flink/fs/azure/common/hadoop/HadoopRecoverableWriter at
org.apache.flink.fs.azure.common.hadoop.HadoopFileSystem.createRecoverableWriter(HadoopFileSystem.java:202)
at
org.apache.flink.core.fs.SafetyNetWrapperFileSystem.createRecoverableWriter(SafetyNetWrapperFileSystem.java:69)
at
org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.<init>(Buckets.java:112)
at
org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink$RowFormatBuilder.createBuckets(StreamingFileSink.java:242)
at
org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.initializeState(StreamingFileSink.java:327)
at
org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
at
org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
at
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:281)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:881)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:395)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) at
org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) at
java.lang.Thread.run(Thread.java:748)Caused by:
java.lang.ClassNotFoundException:
org.apache.flink.fs.azure.common.hadoop.HadoopRecoverableWriter at
java.net.URLClassLoader.findClass(URLClassLoader.java:382) at
java.lang.ClassLoader.loadClass(ClassLoader.java:418) at
org.apache.flink.util.ChildFirstClassLoader.loadClass(ChildFirstClassLoader.java:60)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
