Jetiaime opened a new issue, #4518:
URL: https://github.com/apache/streampark/issues/4518
### Search before asking
Searched existing issues for `hdfs:/`, checkpoint deletion and HDFS path
handling. #1017 concerns a different filesystem-authority mismatch; #2210
discusses deleting checkpoint records and files. I did not find a report of
this specific single-slash URI conversion.
### Java Version
Real-HDFS reproduction and module regression tests: Eclipse Temurin 11.0.27
in an isolated Linux container. The module tests also pass on OpenJDK 17 with
Java 11 compilation targets.
### Scala Version
2.12.x (not involved in this Java filesystem path conversion).
### StreamPark Version
Current `dev` / 3.0.0 at `829466b5470d749773793193f1fc1d46e8613d61`.
### Flink Version
The deletion failure is reproducible without running Flink, by passing a
stored checkpoint path to `HdfsOperator.delete`. A live Flink job is not needed
for the filesystem reproduction.
### Deploy mode
yarn-application
### What happened
Deleting a checkpoint/savepoint whose stored path is
`hdfs:/tmp/checkpoints/chk-1` fails to remove its HDFS directory. This is a
valid Hadoop path without an authority: Hadoop can resolve it against
`fs.defaultFS`.
`HdfsOperator.toHdfsPath` only recognizes the `hdfs://` prefix. For a
single-slash URI, it prepends the default filesystem to a path that already has
the HDFS scheme. For example:
```text
fs.defaultFS = hdfs://namenode:8020
stored path = hdfs:/tmp/checkpoints/chk-1
converted = hdfs://namenode:8020hdfs:/tmp/checkpoints/chk-1
```
In the YARN Application deletion flow, the Console loads the stored
savepoint path and passes it to this operator. The input should be preserved
and resolved by Hadoop rather than prefixed again.
#### Minimal reproduction
1. Configure an isolated HDFS instance as `fs.defaultFS`.
2. Create `/tmp/checkpoints/chk-1/_metadata` and an adjacent file to retain.
3. Call `HdfsOperator.delete("hdfs:/tmp/checkpoints/chk-1")`.
4. Verify that the checkpoint directory still exists on unpatched `dev`; the
call throws `IllegalArgumentException` in the real-HDFS test.
5. Repeat with `hdfs:///tmp/checkpoints/chk-1` or a fully qualified HDFS
URI: those paths delete correctly.
Expected: the single-slash HDFS URI deletes the same directory as the
triple-slash URI, and the adjacent file remains intact.
#### Verified scope
An isolated Hadoop 3.3.5 `MiniDFSCluster` (real NameNode + one DataNode, not
a filesystem mock) reproduced the bug using the upstream `HdfsOperator`. Tested
four input forms (`hdfs:/...`, `hdfs:///...`, fully qualified HDFS URI,
absolute path) with `fs.defaultFS` both with and without a trailing slash. The
two single-slash cases failed before the fix; all eight cases passed with the
fix. The adjacent file's content was checked after each deletion.
### Error Exception
`IllegalArgumentException` while attempting the malformed HDFS path; the
intended HDFS directory remains present.
### Are you willing to submit PR?
- [x] Yes, a minimal fix and a 12-case parameterized regression test are
prepared.
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
