This is an automated email from the ASF dual-hosted git repository.
yihua pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hudi-rs.git
The following commit(s) were added to refs/heads/main by this push:
new 53a403c8 chore: let consumers opt out of the merge map's on-disk tier
(#681)
53a403c8 is described below
commit 53a403c8ce7d4f5ee5d1931e220be431501d27e9
Author: Lin Liu <[email protected]>
AuthorDate: Tue Sep 1 18:39:35 2026 -0700
chore: let consumers opt out of the merge map's on-disk tier (#681)
---
.github/workflows/ci.yml | 47 ++++++++++++++++++++++++++++++++++++++++
AGENTS.md | 24 ++++++++++++++------
crates/core/src/schema/delete.rs | 16 ++++++++++++++
crates/datafusion/Cargo.toml | 12 +++++++++-
crates/hudi/Cargo.toml | 15 +++++++++++--
5 files changed, 104 insertions(+), 10 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 2b951027..44d0ec67 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -105,6 +105,53 @@ jobs:
run: cargo clippy -p hudi-core --lib --no-default-features -- -D
warnings
- name: Unit tests without the spill backend
run: cargo test -p hudi-core --lib --no-default-features --no-fail-fast
+ # The umbrella crate is where a consumer actually opts out, and it
reaches
+ # hudi-core twice: directly, and through hudi-datafusion. Either edge
taking
+ # default features hands the tier back without the steps above noticing,
+ # since they only build hudi-core. Both shapes are built here, where
there
+ # is no libclang to build rocksdb with.
+ - name: Check the umbrella crate without the spill backend
+ run: cargo check -p hudi --no-default-features
+ - name: Check the umbrella crate without the spill backend, with
datafusion
+ run: cargo check -p hudi --no-default-features --features datafusion
+ # The steps above prove the opt-out COMPILES. This proves it actually
drops
+ # the dependency, and unlike them it does not rest on libclang being
absent,
+ # so it keeps working if this job ever gains a C++ toolchain.
+ - name: Assert the opt-out drops rocksdb from the dependency graph
+ run: |
+ # A graph assertion has two ways to look like it passed: the tier
+ # really is gone, or cargo tree never ran. `-i` exits non-zero with
an
+ # empty stdout in BOTH cases, so neither the output nor the status
+ # alone tells them apart, and discarding stderr makes a broken
command
+ # indistinguishable from a working opt-out. Each check below
therefore
+ # pins the specific reason rather than inferring from emptiness.
+
+ # Under defaults the tier must be reachable. If this ever stops
+ # holding, the absence checks below are asserting nothing.
+ for args in "" "--features datafusion"; do
+ if ! out=$(cargo tree -p hudi $args -i rocksdb 2>&1) \
+ || ! printf '%s\n' "$out" | grep -q '^rocksdb'; then
+ echo "cargo tree did not find rocksdb under default features:
'$args'"
+ echo "so the absence checks below cannot tell an opt-out from a
broken command"
+ printf '%s\n' "$out"
+ exit 1
+ fi
+ done
+
+ # With the opt-out it must be gone — and gone because the package is
+ # absent from the graph, which cargo says in those words, not because
+ # the command failed for some other reason.
+ for args in "--no-default-features" "--no-default-features
--features datafusion"; do
+ if out=$(cargo tree -p hudi $args -i rocksdb 2>&1); then
+ echo "rocksdb is still reachable from hudi with: $args"
+ printf '%s\n' "$out"
+ exit 1
+ elif ! printf '%s\n' "$out" | grep -q 'did not match any
packages'; then
+ echo "cargo tree failed for a reason other than rocksdb being
absent: '$args'"
+ printf '%s\n' "$out"
+ exit 1
+ fi
+ done
rust-tests-windows:
runs-on: windows-2025
diff --git a/AGENTS.md b/AGENTS.md
index 881031b3..cdf5f24d 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -45,13 +45,23 @@ cargo test -p hudi-core --lib --no-default-features
### Features
-`hudi-core` has one default-on feature, `spill-rocksdb`, carrying the
merge-on-read merge map's
-on-disk tier. It is separable because `rocksdb` bundles RocksDB and runs
`bindgen`, so leaving it
-unconditional puts libclang and a C++ toolchain in front of every consumer —
the Python wheel and
-the cxx bridge included — for a tier that only engages when a merge exceeds
-`hoodie.memory.merge.max.size`. A test that needs the disk tier must be
-`#[cfg(feature = "spill-rocksdb")]`; the `--no-default-features` CI leg
installs no libclang, so a
-change that reintroduces the native dependency fails to compile there rather
than passing quietly.
+`spill-rocksdb` carries the merge-on-read merge map's on-disk tier. It is
separable because
+`rocksdb` bundles RocksDB and runs `bindgen`, so leaving it unconditional puts
libclang and a C++
+toolchain in front of every consumer, for a tier that only engages when a
merge exceeds
+`hoodie.memory.merge.max.size`.
+
+It is **default-on in `hudi-core`, `hudi-datafusion` and `hudi`**, and the
latter two forward it
+rather than taking `hudi-core` with its defaults: a consumer can only opt out
of a feature its
+direct dependency exposes, and unification through `hudi-datafusion` would
otherwise hand the tier
+back. The opt-out is therefore `default-features = false` on `hudi`, and it
holds with the
+`datafusion` feature on. **The Python wheel and the cxx bridge do not opt
out** — both take `hudi`
+with defaults and build RocksDB. Dropping the feature removes the tier, and a
merge past the budget
+then fails with a `CoreError::Unsupported` naming the config rather than
spilling, which is why the
+published artifacts keep it.
+
+A test that needs the disk tier must be `#[cfg(feature = "spill-rocksdb")]`;
the
+`--no-default-features` CI leg installs no libclang, so a change that
reintroduces the native
+dependency fails to compile there rather than passing quietly.
### Dependencies
diff --git a/crates/core/src/schema/delete.rs b/crates/core/src/schema/delete.rs
index e62a3c79..883bb236 100644
--- a/crates/core/src/schema/delete.rs
+++ b/crates/core/src/schema/delete.rs
@@ -27,6 +27,19 @@ use once_cell::sync::Lazy;
use serde_json::Value as JsonValue;
use std::sync::Arc;
+/// One delete record on its own, as it looks once unwrapped out of the list.
+///
+/// The wire schema is [`DELETE_RECORD_LIST_AVRO_SCHEMA_STR`]
+/// (`HoodieDeleteRecordList.avsc`) — that is what a delete log block is
written
+/// as and what the live decode reads. This file is its post-unwrap companion,
+/// kept as the reference spelling of a single record's shape.
+///
+/// The two helpers over it, [`unwrap_ordering_value`] and
+/// [`avro_schema_for_delete_record`], have no callers beyond this module's own
+/// tests. Their plural namesakes are the live path and are different
functions:
+/// `unwrap_ordering_values` and [`avro_schema_for_delete_record_list`], both
in
+/// `file_group::log_file::content`'s decode. Singular here means one record;
+/// plural means the block.
static DELETE_RECORD_AVRO_SCHEMA_STR: &str = include_str!(concat!(
env!("CARGO_MANIFEST_DIR"),
"/schemas/HoodieDeleteRecord.avsc"
@@ -159,6 +172,9 @@ pub fn avro_schema_for_delete_record(delete_record_value:
&AvroValue) -> Result<
AvroSchema::parse(&json).map_err(CoreError::AvroError)
}
+/// The wire schema: what a delete log block is written as, and what the live
+/// decode reads. [`DELETE_RECORD_AVRO_SCHEMA_STR`] is the single-record
+/// companion to this one.
static DELETE_RECORD_LIST_AVRO_SCHEMA_STR: &str = include_str!(concat!(
env!("CARGO_MANIFEST_DIR"),
"/schemas/HoodieDeleteRecordList.avsc"
diff --git a/crates/datafusion/Cargo.toml b/crates/datafusion/Cargo.toml
index 6e6bb333..5a9a8b7d 100644
--- a/crates/datafusion/Cargo.toml
+++ b/crates/datafusion/Cargo.toml
@@ -27,8 +27,18 @@ description.workspace = true
homepage.workspace = true
repository.workspace = true
+[features]
+default = ["spill-rocksdb"]
+# Forwarded so a consumer reaching hudi-core through this crate can still drop
+# the merge map's on-disk tier. Taking hudi-core with its defaults here would
+# re-enable that tier through feature unification whatever the umbrella crate
+# asked for.
+spill-rocksdb = ["hudi-core/spill-rocksdb"]
+
[dependencies]
-hudi-core = { version = "0.5.0-dev", path = "../core", features =
["datafusion"] }
+hudi-core = { version = "0.5.0-dev", path = "../core", default-features =
false, features = [
+ "datafusion",
+] }
# arrow
arrow = { workspace = true }
arrow-arith = { workspace = true }
diff --git a/crates/hudi/Cargo.toml b/crates/hudi/Cargo.toml
index 67c4cfd5..77c6a151 100644
--- a/crates/hudi/Cargo.toml
+++ b/crates/hudi/Cargo.toml
@@ -28,10 +28,21 @@ homepage.workspace = true
repository.workspace = true
[dependencies]
-hudi-core = { version = "0.5.0-dev", path = "../core" }
-hudi-datafusion = { version = "0.5.0-dev", path = "../datafusion", optional =
true }
+hudi-core = { version = "0.5.0-dev", path = "../core", default-features =
false }
+hudi-datafusion = { version = "0.5.0-dev", path = "../datafusion", optional =
true, default-features = false }
[features]
+default = ["spill-rocksdb"]
+# On by default, so this crate and everything downstream of it keep the merge
+# map's on-disk tier unless they say otherwise. Forwarded rather than left to
+# hudi-core's own default because a consumer can only opt out of a feature its
+# direct dependency exposes: without this, `default-features = false` here
would
+# still get the tier back through hudi-core. Dropping it removes the tier, and
a
+# merge exceeding `hoodie.memory.merge.max.size` then fails rather than
spilling.
+spill-rocksdb = [
+ "hudi-core/spill-rocksdb",
+ "hudi-datafusion?/spill-rocksdb",
+]
datafusion = [
"hudi-datafusion",
]
