peterxcli opened a new issue, #4867:
URL: https://github.com/apache/datafusion-comet/issues/4867
## Background
Comet currently has two HDFS implementations:
- `native/core` default feature `hdfs-opendal`, which uses OpenDAL's HDFS
backend through `object_store_opendal`
- legacy Comet-maintained crates:
- `native/hdfs`
- `native/fs-hdfs`
The default native build already uses `hdfs-opendal`, and the legacy
implementation is only selected when the `hdfs` feature is enabled without
`hdfs-opendal`.
## Proposal
Remove the legacy Comet-maintained HDFS client/object_store implementation
and standardize HDFS support on OpenDAL.
This should include:
- Remove `native/hdfs` and `native/fs-hdfs` from the Rust workspace.
- Remove the legacy `hdfs` implementation path from
`native/core/src/parquet/parquet_support.rs`.
- Either remove the `hdfs` feature or keep it as a compatibility alias to
`hdfs-opendal`.
- Update user docs that still recommend `COMET_FEATURES=hdfs`.
- Keep `spark.hadoop.fs.comet.libhdfs.schemes` behavior so configured
schemes still route through the native HDFS backend.
- Verify HDFS read/write behavior through the existing OpenDAL path.
## Motivation
Maintaining a separate libhdfs wrapper and object_store implementation
increases Comet's maintenance burden. OpenDAL already provides HDFS support and
is the default path in Comet, so keeping the legacy crates creates duplicate
code paths and feature confusion.
## Notes
OpenDAL HDFS still depends on the Hadoop/JDK/libhdfs runtime environment, so
this is not intended to remove those runtime requirements. The goal is to stop
maintaining Comet's own HDFS client code.
## Acceptance Criteria
- Default HDFS read path continues to work through OpenDAL.
- Native HDFS write path continues to work through OpenDAL.
- Existing fake-HDFS/native scan tests remain valid or are updated to
reflect the OpenDAL-only path.
- Documentation no longer points users at the legacy `hdfs` feature or
`native/hdfs` README.
- `cargo build` and relevant JVM tests pass with the final feature layout.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
