CHANHAE OH created FLINK-39033:
----------------------------------
Summary: File Connector does not support WebHDFS protocol
Key: FLINK-39033
URL: https://issues.apache.org/jira/browse/FLINK-39033
Project: Flink
Issue Type: Bug
Components: API / Core
Affects Versions: 2.0.1
Reporter: CHANHAE OH
Fix For: 2.0.1
Attachments: image-2026-02-05-14-59-51-374.png
We encountered an issue where the File Connector does not support the WebHDFS
protocol.
>flink/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableWriter.java
{code:java}
protected void checkSupportedFSSchemes(org.apache.hadoop.fs.FileSystem fs) {
// This writer is only supported on a subset of file systems
if (!("hdfs".equalsIgnoreCase(fs.getScheme())
|| "viewfs".equalsIgnoreCase(fs.getScheme()))) {
throw new UnsupportedOperationException(
"Recoverable writers on Hadoop are only supported for HDFS");
}
// Part of functionality depends on specific versions. We check these
schemes and versions
// eagerly for
// better error messages.
if (!HadoopUtils.isMinHadoopVersion(2, 7)) {
LOG.warn(
"WARNING: You are running on hadoop version "
+ VersionInfo.getVersion()
+ "."
+ " If your RollingPolicy does not roll on every
checkpoint/savepoint, the StreamingFileSink will throw an exception upon
recovery.");
}
} {code}
Interestingly, when using the Iceberg Connector in the same environment, data
can be
successfully written to WebHDFS paths without any issues. This suggests that
WebHDFS
itself is a supported protocol, and upon further investigation, it may be
possible for
the File Connector to support WebHDFS as well.
Would it be possible for me to submit a PR to improve this? If so, could you
provide
any guidance on what areas I should examine?
Thank you!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
