[
https://issues.apache.org/jira/browse/FLINK-39033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
CHANHAE OH updated FLINK-39033:
-------------------------------
Attachment: (was: image-2026-02-05-14-59-51-374.png)
> File Connector does not support WebHDFS protocol
> ------------------------------------------------
>
> Key: FLINK-39033
> URL: https://issues.apache.org/jira/browse/FLINK-39033
> Project: Flink
> Issue Type: Bug
> Components: API / Core
> Affects Versions: 2.0.1
> Reporter: CHANHAE OH
> Priority: Minor
> Fix For: 2.0.1
>
>
> We encountered an issue where the File Connector does not support the WebHDFS
> protocol.
>
> >flink/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableWriter.java
>
> {code:java}
> protected void checkSupportedFSSchemes(org.apache.hadoop.fs.FileSystem fs) {
> // This writer is only supported on a subset of file systems
> if (!("hdfs".equalsIgnoreCase(fs.getScheme())
> || "viewfs".equalsIgnoreCase(fs.getScheme()))) {
> throw new UnsupportedOperationException(
> "Recoverable writers on Hadoop are only supported for HDFS");
> }
> // Part of functionality depends on specific versions. We check these
> schemes and versions
> // eagerly for
> // better error messages.
> if (!HadoopUtils.isMinHadoopVersion(2, 7)) {
> LOG.warn(
> "WARNING: You are running on hadoop version "
> + VersionInfo.getVersion()
> + "."
> + " If your RollingPolicy does not roll on every
> checkpoint/savepoint, the StreamingFileSink will throw an exception upon
> recovery.");
> }
> } {code}
>
>
> Interestingly, when using the Iceberg Connector in the same environment, data
> can be
> successfully written to WebHDFS paths without any issues. This suggests that
> WebHDFS
> itself is a supported protocol, and upon further investigation, it may be
> possible for
> the File Connector to support WebHDFS as well.
>
> Would it be possible for me to submit a PR to improve this? If so, could you
> provide
> any guidance on what areas I should examine?
> Thank you!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
