andygrove opened a new pull request, #3749: URL: https://github.com/apache/datafusion-comet/pull/3749
## Which issue does this PR close? Closes #. ## Rationale for this change Continue the crate extraction effort started in #3667 (which extracted `common` and `jni-bridge`). The shuffle module has a nearly one-directional dependency graph, making it an ideal candidate for extraction into its own crate. This improves code modularity and separation of concerns. ## What changes are included in this PR? **Move shared utilities to `common` crate:** - `bytes_to_i128` function moved to `common/src/utils.rs` (re-exported from core for backward compat) - `tracing.rs` module moved to `common/src/tracing.rs` (re-exported from core). Upgraded `once_cell::sync::Lazy` to `std::sync::LazyLock` (stable since Rust 1.80) **Create `datafusion-comet-shuffle` crate (`native/shuffle/`):** - All 16 shuffle source files moved from `native/core/src/execution/shuffle/` - Public API unchanged: `CompressionCodec`, `ShuffleBlockWriter`, `read_ipc_compressed`, `CometPartitioning`, `ShuffleWriterExec`, `spark_unsafe` module - `shuffle_writer` and `row_columnar` benchmarks moved to the new crate **Update core:** - Core re-exports the shuffle crate as `pub use datafusion_comet_shuffle as shuffle` in `execution/mod.rs`, so existing `crate::execution::shuffle::*` import paths in planner.rs, jni_api.rs, and parquet_writer.rs continue to work unchanged **Housekeeping:** - Added README files for `common`, `jni-bridge`, and `shuffle` crates - All crates have `publish = false` ## How are these changes tested? - `cargo build --workspace` passes - `cargo clippy --all-targets --workspace -- -D warnings` passes - `cargo bench -p datafusion-comet-shuffle --no-run` compiles benchmarks - No functional changes — pure refactoring, verified by workspace build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
