[
https://issues.apache.org/jira/browse/FLINK-31875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17714852#comment-17714852
]
Martijn Visser commented on FLINK-31875:
----------------------------------------
I don't think that's the problem here. Flink has a minimum version of Hadoop
2.10.2, but the Flink filesystems have their own implementations. I believe
that the issue is because OSS doesn't shade anything in with Hadoop (like is
done with S3 Hadoop and S3 Presto), so if you would like to use OSS you also
need to have Hadoop artifacts on the classpath.
> OSS throwns NoClassDefFoundError due to old hadoop-common version
> -----------------------------------------------------------------
>
> Key: FLINK-31875
> URL: https://issues.apache.org/jira/browse/FLINK-31875
> Project: Flink
> Issue Type: Bug
> Components: FileSystems
> Affects Versions: 1.16.0, 1.17.0
> Reporter: Hangxiang Yu
> Assignee: Hangxiang Yu
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.16.2, 1.17.1
>
>
> h2. Problem
> When using OSS in 1.17, an exception will be thrown:
> {code:java}
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/thirdparty/com/google/common/base/Preconditions
> at
> org.apache.hadoop.fs.aliyun.oss.AliyunOSSUtils.longOption(AliyunOSSUtils.java:221)
> at
> org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.initialize(AliyunOSSFileSystem.java:343)
> at
> org.apache.flink.fs.osshadoop.OSSFileSystemFactory.create(OSSFileSystemFactory.java:147)
> at
> org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:508)
> at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:409)
> at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274){code}
> It could be reproduced in ITCASE of OSS if some envs has been configured.
> h2. Why
> After https://issues.apache.org/jira/browse/FLINK-27308 and
> https://issues.apache.org/jira/browse/FLINK-29502 ，hadoop-aliyun has also be
> upgraded to 3.3.4 which relys on the newest version of hadoop-common.
> OSS still uses the old version (2.10.2) extended from flink-parent so that
> some classes cannot be found.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
