This is an automated email from the ASF dual-hosted git repository.
danny0405 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new 82640e56e66f feat(cli): add table set-meta-fields-mode command (#19206)
82640e56e66f is described below
commit 82640e56e66f7dcf0389a55b990e3e8aabaf4afa
Author: Sivabalan Narayanan <[email protected]>
AuthorDate: Tue Aug 18 23:59:20 2026 -0700
feat(cli): add table set-meta-fields-mode command (#19206)
* feat(cli): add table set-meta-fields-mode command
Adds a hudi-cli command to toggle hoodie.meta.fields.mode on an existing
table:
table set-meta-fields-mode --target-mode <MODE> [--force true|false]
Where <MODE> is one of ALL, NONE, COMMIT_TIME_ONLY, FILE_NAME_ONLY,
COMMIT_TIME_AND_FILE_NAME. This is the sanctioned way to change the
property outside of
table creation — hoodie.meta.fields.mode is immutable at runtime because it
is a
physical-storage decision baked into files at write time.
Safety guard:
- On a table that already has commits, the command refuses to change the
mode by default.
Existing files were written under the current mode and are not rewritten
by this command;
new commits would be written under the new mode, producing mixed-mode
files whose
incremental / file-pruning semantics differ between old and new data.
- Pass --force to override the guard. A warning is logged describing the
specific data
correctness impact.
Behavior:
- ALL and NONE are persisted implicitly (via
populate.meta.fields=true/false); the
hoodie.meta.fields.mode property is cleared when transitioning to
ALL/NONE.
- Selective modes (COMMIT_TIME_ONLY, FILE_NAME_ONLY,
COMMIT_TIME_AND_FILE_NAME) are
persisted explicitly on hoodie.properties alongside
populate.meta.fields=false.
- No-op when target matches current mode.
Test coverage (TestTableCommand):
- All 5 modes settable on a fresh table.
- Selective → ALL clears the property.
- No-op when target matches current mode.
- Rejects unknown enum values.
- Refuses on a populated table without --force; the mode does not change.
- Accepts on a populated table with --force; the mode changes.
Follow-up to PR #19205 (hoodie.meta.fields.mode). PR-C (MoR support) is the
next follow-up.
* fix(cli): apply the one-way meta-fields rule to set-meta-fields-mode
Rebased onto #19205's current head, which establishes that
hoodie.meta.fields.mode
is a table property changeable only through this command or an upgrade.
Three gaps
between what the command did and what that rule requires:
1. Nothing stopped a *widening*. The --force gate keyed only on commit
count, so
`--target-mode ALL --force` on a populated COMMIT_TIME_ONLY table was
allowed.
Existing files are not rewritten, so the table would advertise a meta
column that
is null for every earlier row, and incremental queries would admit the
table and
silently skip exactly those rows. Widening is now refused outright and
--force
does not override it -- there is no consequence for an operator to
knowingly
accept. Narrowing keeps the --force gate, since it only leaves values
nothing
reads. Uses MetaFieldsMode#isWiderThan, the same predicate
BaseHoodieWriteClient#validateAgainstTableProperties uses, so the CLI
and the
writer cannot disagree about which transitions are legal.
2. ALL and NONE deleted the mode property instead of writing it. That left
the table
resolving through the legacy fallback -- indistinguishable from a table
predating
the property -- made the command's effect invisible in
hoodie.properties, and
removed the value TenToNineDowngradeHandler reads to derive the boolean
it writes
back. Both properties are now always written, with the boolean derived
from the
mode, matching the invariant TableBuilder enforces at creation.
3. valueOf rejected lowercase input and produced a bare "No enum constant".
Routed
through MetaFieldsMode.parse, so `commit_time_only` works and the
message lists
the allowed values like every other path.
Tests: widening refused in three forms including --force; narrowing refused
without
--force and accepted with it; lowercase accepted; both properties in
agreement
across all five modes. testSetMetaFieldsModeToAllClearsProperty asserted the
deleted-property behavior and is rewritten as
testSetMetaFieldsModeToAllWritesTheModeExplicitly.
* docs(cli): clarify that --force permits narrowing only
The --force help implied it could override any mode change on a populated
table.
It cannot override a widening -- that is refused outright, since existing
files are
never rewritten and the table would advertise a column null for every
earlier row.
* test(cli): pin the full meta-fields-mode migration matrix
The existing tests spot-checked "narrowing works, widening does not". That
misses
the shape of the relation: COMMIT_TIME_ONLY and FILE_NAME_ONLY are mutually
wider --
each populates a column the other does not -- so neither can migrate to the
other in
either direction. It is a lattice, not a chain, and a spot-check cannot
show that.
Adds an exhaustive test over all 20 ordered pairs on a populated table,
asserting
each is either a legal narrowing (succeeds with --force, mode and derived
boolean
both updated) or a refused widening (fails even with --force, mode
untouched). Plus
a focused test naming both sibling directions explicitly, since that is the
case a
reader is most likely to get wrong.
Verified the matrix independently against MetaFieldsMode#isWiderThan before
writing
the test, so the expectations are derived from the predicate rather than
from my
reading of it.
Note: these cannot be executed locally -- hudi-cli compilation is blocked
by a stale
hudi-utilities-bundle shadowing hudi-common, and rebuilding that bundle
fails on a
pre-existing KafkaAvroSchemaDeserializer error that reproduces on master.
Compiled
against a classpath excluding the stale bundle, and checkstyle passes.
* fix(cli): adapt meta-fields command after rebase
---------
Co-authored-by: danny0405 <[email protected]>
---
.../org/apache/hudi/cli/commands/TableCommand.java | 105 +++++++
.../apache/hudi/cli/commands/TestTableCommand.java | 323 ++++++++++++++++++++-
.../HoodieTestCommitMetadataGenerator.java | 6 +-
3 files changed, 429 insertions(+), 5 deletions(-)
diff --git
a/hudi-cli/src/main/java/org/apache/hudi/cli/commands/TableCommand.java
b/hudi-cli/src/main/java/org/apache/hudi/cli/commands/TableCommand.java
index a091111952cc..134a28df8e3e 100644
--- a/hudi-cli/src/main/java/org/apache/hudi/cli/commands/TableCommand.java
+++ b/hudi-cli/src/main/java/org/apache/hudi/cli/commands/TableCommand.java
@@ -25,6 +25,7 @@ import org.apache.hudi.cli.TableHeader;
import org.apache.hudi.common.config.HoodieTimeGeneratorConfig;
import org.apache.hudi.common.fs.ConsistencyGuardConfig;
import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.MetaFieldsMode;
import org.apache.hudi.common.schema.HoodieSchema;
import org.apache.hudi.common.table.HoodieTableConfig;
import org.apache.hudi.common.table.HoodieTableMetaClient;
@@ -256,6 +257,110 @@ public class TableCommand {
return renderOldNewProps(newProps, oldProps);
}
+ @ShellMethod(key = "table set-meta-fields-mode",
+ value = "Set hoodie.meta.fields.mode on an existing table. This is the
sanctioned way to change "
+ + "the property — a write never can, since it is a physical-storage
decision baked into "
+ + "files at write time. Two guards apply on a table that already has
commits: widening the "
+ + "mode (adding a populated meta column) is refused outright,
because existing files are "
+ + "not rewritten and the table would advertise a column that is null
for every earlier "
+ + "row; narrowing is refused unless --force is passed, since it
leaves mixed-mode files "
+ + "whose incremental / file-pruning semantics differ between old and
new data.")
+ public String setMetaFieldsMode(
+ @ShellOption(value = {"--target-mode"},
+ help = "One of ALL, NONE, COMMIT_TIME_ONLY, FILE_NAME_ONLY,
COMMIT_TIME_AND_FILE_NAME")
+ final String targetModeStr,
+ @ShellOption(value = {"--force"}, defaultValue = "false",
+ help = "Allow NARROWING the mode on a table that already has
commits. Does not permit "
+ + "widening, which is refused regardless. Existing files are not
rewritten — new "
+ + "commits use the new mode, old commits keep the old one.
Incremental queries and "
+ + "file-name-based lookups will silently drop rows from commits
written under the "
+ + "incompatible mode.")
+ final boolean force) throws IOException {
+ MetaFieldsMode targetMode;
+ try {
+ // Resolve through the public configuration API rather than valueOf: it
is case-insensitive,
+ // trims, and lists the allowed values in its message, so an operator
typing
+ // `commit_time_only` is not rejected and the error text matches what a
bad value in
+ // hoodie.properties or a write option produces.
+ Properties targetProps = new Properties();
+ targetProps.setProperty(HoodieTableConfig.META_FIELDS_MODE.key(),
targetModeStr);
+ targetMode = MetaFieldsMode.resolve(targetProps);
+ } catch (IllegalArgumentException e) {
+ throw new HoodieException("Invalid --target-mode: " + e.getMessage(), e);
+ }
+
+ HoodieCLI.refreshTableMetadata();
+ HoodieTableMetaClient client = HoodieCLI.getTableMetaClient();
+ Map<String, String> oldProps = client.getTableConfig().propsMap();
+ MetaFieldsMode currentMode = client.getTableConfig().getMetaFieldsMode();
+
+ if (currentMode == targetMode) {
+ return String.format("Table is already in %s mode; nothing to change.",
targetMode);
+ }
+
+ int commitCount =
client.getActiveTimeline().getCommitsTimeline().countInstants();
+
+ // Meta-field population is one-way for a table that already holds data,
and --force does not
+ // override it: dropping a column leaves earlier files carrying values
nothing reads, which a
+ // reader can ignore, but *adding* one leaves later files claiming a
column earlier files do not
+ // physically have. Nothing distinguishes the two sets, so a widened table
advertises a column
+ // that is silently null for every pre-existing row — incremental queries
would then admit the
+ // table and drop exactly those rows. There is no consequence for an
operator to accept here, so
+ // this is a hard failure rather than a --force gate.
+ //
+ // Same predicate the write path uses
(BaseHoodieWriteClient#validateAgainstTableProperties), so
+ // the CLI and the writer cannot disagree about which transitions are
legal.
+ if (commitCount > 0 && targetMode.isWiderThan(currentMode)) {
+ throw new HoodieException(String.format(
+ "Refusing to widen hoodie.meta.fields.mode from %s to %s on a table
with %d commit(s): "
+ + "%s populates meta columns that %s does not, and this command
does not rewrite "
+ + "existing files. The table would advertise a column that is
null for every row "
+ + "written so far, which incremental queries and file-name
lookups silently skip. "
+ + "Narrowing the mode is allowed; to widen, recreate the table.",
+ currentMode, targetMode, commitCount, targetMode, currentMode));
+ }
+
+ // Safety check: refuse to change the mode on a table with commits unless
--force.
+ if (commitCount > 0 && !force) {
+ throw new HoodieException(String.format(
+ "Refusing to change hoodie.meta.fields.mode on a table that already
has %d commit(s). "
+ + "Existing files were written under %s and will not be
rewritten by this command; "
+ + "new commits would be written under %s, producing mixed-mode
files whose "
+ + "incremental / file-pruning semantics differ between old and
new data. "
+ + "Pass --force if you accept the consequences, or recreate the
table to change "
+ + "the mode cleanly.",
+ commitCount, currentMode, targetMode));
+ }
+ if (commitCount > 0) {
+ log.warn("--force passed: changing hoodie.meta.fields.mode from {} to {}
on a table with "
+ + "{} commit(s). Existing files retain the old layout; new
commits use the new "
+ + "layout. Incremental queries and file-name lookups may
silently drop rows written "
+ + "under the incompatible mode.",
+ currentMode, targetMode, commitCount);
+ }
+
+ // Persist both properties, always, and derive the legacy boolean from the
mode rather than
+ // letting the caller supply it — the same invariant
HoodieTableMetaClient.TableBuilder enforces
+ // at creation. hoodie.properties can then never contradict itself, which
is what lets a
+ // pre-1.3.0 reader (which sees only the boolean) treat a selective table
as NONE instead of
+ // assuming meta columns that are physically null.
+ //
+ // The mode is written even for ALL and NONE rather than deleted. Deleting
it would leave the
+ // table resolving through the legacy fallback, which is indistinguishable
from a table that
+ // predates the property — and it would make this command's effect
invisible to anyone reading
+ // hoodie.properties. Writing it explicitly also keeps the value present
for the v10 -> v9
+ // downgrade handler, which reads the mode to derive the boolean it writes
back.
+ Properties toUpdate = new Properties();
+ toUpdate.setProperty(HoodieTableConfig.META_FIELDS_MODE.key(),
targetMode.name());
+ toUpdate.setProperty(HoodieTableConfig.POPULATE_META_FIELDS.key(),
+ String.valueOf(targetMode.toLegacyPopulateMetaFields()));
+ HoodieTableConfig.update(client.getStorage(), client.getMetaPath(),
toUpdate);
+
+ HoodieCLI.refreshTableMetadata();
+ Map<String, String> newProps =
HoodieCLI.getTableMetaClient().getTableConfig().propsMap();
+ return renderOldNewProps(newProps, oldProps);
+ }
+
@ShellMethod(key = "table change-table-type", value = "Change hudi table
type to target type: COW or MOR. "
+ "Note: before changing to COW, by default this command will execute
all the pending compactions and execute a full compaction if needed.")
public String changeTableType(
diff --git
a/hudi-cli/src/test/java/org/apache/hudi/cli/commands/TestTableCommand.java
b/hudi-cli/src/test/java/org/apache/hudi/cli/commands/TestTableCommand.java
index 67f60662eee9..d0a73785106d 100644
--- a/hudi-cli/src/test/java/org/apache/hudi/cli/commands/TestTableCommand.java
+++ b/hudi-cli/src/test/java/org/apache/hudi/cli/commands/TestTableCommand.java
@@ -26,6 +26,7 @@ import
org.apache.hudi.common.config.HoodieTimeGeneratorConfig;
import org.apache.hudi.common.fs.ConsistencyGuardConfig;
import org.apache.hudi.common.model.HoodieCommitMetadata;
import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.MetaFieldsMode;
import org.apache.hudi.common.schema.HoodieSchema;
import org.apache.hudi.common.schema.HoodieSchemaUtils;
import org.apache.hudi.common.table.HoodieTableConfig;
@@ -57,6 +58,7 @@ import java.util.List;
import java.util.Map;
import static
org.apache.hudi.common.table.HoodieTableMetaClient.METAFOLDER_NAME;
+import static
org.apache.hudi.common.table.timeline.versioning.TimelineLayoutVersion.CURR_VERSION;
import static org.apache.hudi.common.util.StringUtils.fromUTF8Bytes;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
@@ -86,7 +88,8 @@ public class TestTableCommand extends
CLIFunctionalTestHarness {
tableName = tableName();
tablePath = tablePath(tableName);
metaPath = Paths.get(tablePath, METAFOLDER_NAME).toString();
- archivePath = Paths.get(metaPath,
HoodieTableConfig.TIMELINE_HISTORY_PATH.defaultValue()).toString();
+ archivePath = Paths.get(metaPath,
HoodieTableConfig.TIMELINE_PATH.defaultValue(),
+ HoodieTableConfig.TIMELINE_HISTORY_PATH.defaultValue()).toString();
}
/**
@@ -136,11 +139,11 @@ public class TestTableCommand extends
CLIFunctionalTestHarness {
// Test meta
HoodieTableMetaClient client = HoodieCLI.getTableMetaClient();
- assertEquals(archivePath, client.getArchivePath());
+ assertEquals(archivePath, client.getArchivePath().toString());
assertEquals(tablePath, client.getBasePath().toString());
assertEquals(metaPath, client.getMetaPath().toString());
assertEquals(HoodieTableType.COPY_ON_WRITE, client.getTableType());
- assertEquals(new Integer(1),
client.getTimelineLayoutVersion().getVersion());
+ assertEquals(CURR_VERSION, client.getTimelineLayoutVersion().getVersion());
HoodieTimeGeneratorConfig timeGeneratorConfig =
HoodieCLI.timeGeneratorConfig;
assertEquals(tablePath, timeGeneratorConfig.getBasePath());
@@ -251,7 +254,7 @@ public class TestTableCommand extends
CLIFunctionalTestHarness {
+ " \"name\" : \"val\",\n"
+ " \"type\" : [ \"null\", \"string\" ],\n"
+ " \"default\" : null\n"
- + " }]};";
+ + " }]}";
generateData(schemaStr);
@@ -304,4 +307,316 @@ public class TestTableCommand extends
CLIFunctionalTestHarness {
fis.close();
return fromUTF8Bytes(data);
}
+
+ //
---------------------------------------------------------------------------
+ // set-meta-fields-mode
+ //
---------------------------------------------------------------------------
+
+ @Test
+ public void testSetMetaFieldsModeOnFreshTableToCommitTimeOnly() {
+ assertTrue(prepareTable());
+ // Default table is ALL — no commits yet, so the safety check must let
this through.
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ // Rendered diff must surface the changed properties so the operator can
confirm the write.
+ String rendered = result.toString();
+ assertTrue(rendered.contains(HoodieTableConfig.POPULATE_META_FIELDS.key()),
+ "expected rendered diff to mention populate.meta.fields, got: " +
rendered);
+ assertTrue(rendered.contains(HoodieTableConfig.META_FIELDS_MODE.key()),
+ "expected rendered diff to mention meta.fields.mode, got: " +
rendered);
+ HoodieTableMetaClient client = HoodieCLI.getTableMetaClient();
+ assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY,
client.getTableConfig().getMetaFieldsMode());
+ assertFalse(client.getTableConfig().populateMetaFields());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeOnFreshTableToFileNameOnly() {
+ assertTrue(prepareTable());
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode FILE_NAME_ONLY");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ assertEquals(MetaFieldsMode.FILE_NAME_ONLY,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeOnFreshTableToCombinedMode() {
+ assertTrue(prepareTable());
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_AND_FILE_NAME");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ assertEquals(MetaFieldsMode.COMMIT_TIME_AND_FILE_NAME,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeOnFreshTableToNone() {
+ assertTrue(prepareTable());
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode NONE");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ HoodieTableMetaClient client = HoodieCLI.getTableMetaClient();
+ assertEquals(MetaFieldsMode.NONE,
client.getTableConfig().getMetaFieldsMode());
+ assertFalse(client.getTableConfig().populateMetaFields());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeToAllWritesTheModeExplicitly() throws
IOException {
+ assertTrue(prepareTable());
+ // First move to a selective mode, then back to ALL. Legal here only
because the table has no
+ // commits — on a populated table this would be a widening and refused
outright.
+ shell.evaluate(() -> "table set-meta-fields-mode --target-mode
COMMIT_TIME_ONLY");
+ Object result = shell.evaluate(() -> "table set-meta-fields-mode
--target-mode ALL");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+
+ HoodieTableMetaClient client = HoodieCLI.getTableMetaClient();
+ assertEquals(MetaFieldsMode.ALL,
client.getTableConfig().getMetaFieldsMode());
+ assertTrue(client.getTableConfig().populateMetaFields());
+ // The mode is written explicitly rather than deleted. Deleting it would
leave the table
+ // resolving through the legacy fallback -- indistinguishable from a table
predating the
+ // property -- and would make this command's effect invisible in
hoodie.properties. The v10->v9
+ // downgrade handler also reads the mode to derive the boolean it writes
back.
+ assertEquals(MetaFieldsMode.ALL.name(),
+ client.getTableConfig().getString(HoodieTableConfig.META_FIELDS_MODE));
+ }
+
+ @Test
+ public void
testSetMetaFieldsModeRefusesWideningOnPopulatedTableEvenWithForce() throws
Exception {
+ assertTrue(prepareTable());
+ shell.evaluate(() -> "table set-meta-fields-mode --target-mode
COMMIT_TIME_ONLY");
+ createDummyCommitFile("20260101000000000");
+ HoodieCLI.refreshTableMetadata();
+
+ // Widening is one-way-forbidden and --force does not override it:
existing files are not
+ // rewritten, so the table would advertise a column that is null for every
row written so far,
+ // and incremental queries would then admit the table and silently skip
those rows. There is no
+ // consequence for an operator to knowingly accept, so this is a hard
failure.
+ for (String command : new String[] {
+ "table set-meta-fields-mode --target-mode ALL",
+ "table set-meta-fields-mode --target-mode ALL --force true",
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_AND_FILE_NAME
--force true"}) {
+ Object result = shell.evaluate(() -> command);
+ assertFalse(ShellEvaluationResultUtil.isSuccess(result), "expected
refusal for: " + command);
+ assertTrue(result.toString().contains("widen"),
+ "expected a widening refusal for '" + command + "', got: " + result);
+ assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode(),
+ "mode must be unchanged after a refused widening");
+ }
+ }
+
+ @Test
+ public void testSetMetaFieldsModeAllowsNarrowingOnPopulatedTableWithForce()
throws Exception {
+ assertTrue(prepareTable());
+ shell.evaluate(() -> "table set-meta-fields-mode --target-mode
COMMIT_TIME_AND_FILE_NAME");
+ createDummyCommitFile("20260101000000000");
+ HoodieCLI.refreshTableMetadata();
+
+ // Narrowing is the direction the CLI exists to allow. It still needs
--force, because it leaves
+ // mixed-mode files, but it is not refused outright the way widening is.
+ Object refused = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ assertFalse(ShellEvaluationResultUtil.isSuccess(refused));
+
+ Object forced = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY --force
true");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(forced), "narrowing with
--force must succeed: " + forced);
+ assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeAcceptsLowercaseTargetMode() {
+ assertTrue(prepareTable());
+ // Routed through MetaFieldsMode.resolve rather than valueOf, so an
operator typing the mode in
+ // lower case is not rejected.
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode commit_time_only");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result), "expected
lowercase to be accepted: " + result);
+ assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeKeepsBothPropertiesInAgreement() {
+ assertTrue(prepareTable());
+ // hoodie.properties must never contradict itself: the legacy boolean is
derived from the mode,
+ // never taken from the caller, so a pre-1.3.0 reader that sees only the
boolean treats a
+ // selective table as NONE rather than assuming meta columns that are
physically null.
+ for (MetaFieldsMode mode : MetaFieldsMode.values()) {
+ shell.evaluate(() -> "table set-meta-fields-mode --target-mode " +
mode.name() + " --force true");
+ HoodieTableConfig tableConfig =
HoodieCLI.getTableMetaClient().getTableConfig();
+ if (tableConfig.getMetaFieldsMode() == mode) {
+ assertEquals(mode.toLegacyPopulateMetaFields(),
tableConfig.populateMetaFields(),
+ "populate.meta.fields must be the derived value for mode " + mode);
+ }
+ }
+ }
+
+ @Test
+ public void testSetMetaFieldsModeNoOpWhenAlreadyInTargetMode() {
+ assertTrue(prepareTable());
+ Object first = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(first));
+ // Second call — same target — should be a no-op message.
+ Object second = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(second));
+ assertTrue(second.toString().contains("already in COMMIT_TIME_ONLY"),
+ "expected no-op message, got: " + second);
+ }
+
+ @Test
+ public void testSetMetaFieldsModeRejectsUnknownValue() {
+ assertTrue(prepareTable());
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode BOGUS_MODE");
+ // Shell evaluate returns the exception object on failure.
+ assertFalse(ShellEvaluationResultUtil.isSuccess(result));
+ assertTrue(result.toString().contains("BOGUS_MODE"),
+ "expected error message to name the rejected value, got: " + result);
+ }
+
+ @Test
+ public void testSetMetaFieldsModeRefusesOnPopulatedTable() throws Exception {
+ assertTrue(prepareTable());
+ createDummyCommitFile("20260101000000000");
+ HoodieCLI.refreshTableMetadata();
+
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ assertFalse(ShellEvaluationResultUtil.isSuccess(result));
+ assertTrue(result.toString().contains("Refusing to change") ||
result.toString().contains("--force"),
+ "expected refusal message, got: " + result);
+
+ // Mode must not have changed.
+ assertEquals(MetaFieldsMode.ALL,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeWithForceOnPopulatedTable() throws
Exception {
+ assertTrue(prepareTable());
+ createDummyCommitFile("20260101000000000");
+ HoodieCLI.refreshTableMetadata();
+
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY --force
true");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+ }
+
+ /**
+ * Pins the whole migration matrix rather than spot-checking it. Twenty
ordered pairs, each either
+ * a legal narrowing or a refused widening, on a table that already has
commits.
+ *
+ * <p>The subtle entries are the mutually-wider siblings: {@code
COMMIT_TIME_ONLY} and
+ * {@code FILE_NAME_ONLY} each populate a column the other does not, so
neither can migrate to the
+ * other in either direction. A spot-check of "narrowing works, widening
does not" would miss that
+ * the relation is a lattice rather than a chain.
+ *
+ * <p>Uses a fresh table per pair so the starting mode can be set without
tripping the very guard
+ * under test -- setting the initial mode happens before any commit exists.
+ */
+ @Test
+ public void testSetMetaFieldsModeMigrationMatrixOnPopulatedTable() throws
Exception {
+ for (MetaFieldsMode from : MetaFieldsMode.values()) {
+ for (MetaFieldsMode to : MetaFieldsMode.values()) {
+ if (from == to) {
+ continue;
+ }
+ // Fresh table per pair; connect to it so HoodieCLI points at the
right one.
+ String pairName = tableName + "_" + from.name() + "_to_" + to.name();
+ String pairPath = tablePath(pairName);
+ assertTrue(ShellEvaluationResultUtil.isSuccess(
+ shell.evaluate(() -> "create --path " + pairPath + " --tableName "
+ pairName)));
+
+ // Establish the starting mode while the table is still empty, then
make it "populated".
+ assertTrue(ShellEvaluationResultUtil.isSuccess(
+ shell.evaluate(() -> "table set-meta-fields-mode --target-mode " +
from.name())),
+ "setting the initial mode on an empty table must succeed: " +
from);
+ createDummyCommitFileAt(pairPath, "20260101000000000");
+ HoodieCLI.refreshTableMetadata();
+ assertEquals(from,
HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+
+ boolean widening = to.isWiderThan(from);
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode " + to.name() + "
--force true");
+
+ if (widening) {
+ assertFalse(ShellEvaluationResultUtil.isSuccess(result),
+ from + " -> " + to + " adds a meta column and must be refused
even with --force");
+ assertTrue(result.toString().contains("widen"),
+ "expected a widening refusal for " + from + " -> " + to + ",
got: " + result);
+ assertEquals(from,
HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode(),
+ "a refused migration must leave the mode untouched: " + from + "
-> " + to);
+ } else {
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result),
+ from + " -> " + to + " drops meta columns and must be allowed
with --force, got: " + result);
+ HoodieTableConfig tableConfig =
HoodieCLI.getTableMetaClient().getTableConfig();
+ assertEquals(to, tableConfig.getMetaFieldsMode(), from + " -> " +
to);
+ assertEquals(to.toLegacyPopulateMetaFields(),
tableConfig.populateMetaFields(),
+ "the derived boolean must follow the new mode for " + from + "
-> " + to);
+ }
+ }
+ }
+ }
+
+ /** Both mutually-wider directions between the two single-column modes are
refused. */
+ @Test
+ public void testSetMetaFieldsModeRefusesBothSiblingDirections() throws
Exception {
+ assertTrue(prepareTable());
+ shell.evaluate(() -> "table set-meta-fields-mode --target-mode
COMMIT_TIME_ONLY");
+ createDummyCommitFile("20260101000000000");
+ HoodieCLI.refreshTableMetadata();
+
+ Object toSibling = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode FILE_NAME_ONLY --force
true");
+ assertFalse(ShellEvaluationResultUtil.isSuccess(toSibling),
+ "COMMIT_TIME_ONLY -> FILE_NAME_ONLY adds _hoodie_file_name and must be
refused");
+ assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+
+ // ...and the reverse, on a table that starts the other way round.
+ String otherName = tableName + "_sibling_reverse";
+ String otherPath = tablePath(otherName);
+ assertTrue(ShellEvaluationResultUtil.isSuccess(
+ shell.evaluate(() -> "create --path " + otherPath + " --tableName " +
otherName)));
+ shell.evaluate(() -> "table set-meta-fields-mode --target-mode
FILE_NAME_ONLY");
+ createDummyCommitFileAt(otherPath, "20260101000000000");
+ HoodieCLI.refreshTableMetadata();
+
+ Object toOther = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY --force
true");
+ assertFalse(ShellEvaluationResultUtil.isSuccess(toOther),
+ "FILE_NAME_ONLY -> COMMIT_TIME_ONLY adds _hoodie_commit_time and must
be refused");
+ assertEquals(MetaFieldsMode.FILE_NAME_ONLY,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+ }
+
+ private void createDummyCommitFileAt(String tableBasePath, String
instantTime) throws IOException {
+ java.nio.file.Path timelineDir =
+ Paths.get(tableBasePath, METAFOLDER_NAME, "timeline");
+ if (!timelineDir.toFile().exists()) {
+ timelineDir.toFile().mkdirs();
+ }
+ String completionTime = instantTime + "1";
+ java.nio.file.Files.createFile(timelineDir.resolve(instantTime + "_" +
completionTime + ".commit"));
+ }
+
+ private void createDummyCommitFile(String instantTime) throws IOException {
+ // Timeline v2 layout: files live under .hoodie/timeline/. Writing an
empty completed commit
+ // is enough to make countInstants > 0 for the safety check.
+ java.nio.file.Path timelineDir = Paths.get(metaPath, "timeline");
+ if (!timelineDir.toFile().exists()) {
+ timelineDir.toFile().mkdirs();
+ }
+ // Completed-commit filename in v2 uses <requested>_<completed>.commit;
the completion time is
+ // used for range queries. Any monotonically-later value works for a
synthetic commit.
+ String completionTime = instantTime + "1";
+ java.nio.file.Files.createFile(timelineDir.resolve(instantTime + "_" +
completionTime + ".commit"));
+ }
}
diff --git
a/hudi-cli/src/test/java/org/apache/hudi/cli/testutils/HoodieTestCommitMetadataGenerator.java
b/hudi-cli/src/test/java/org/apache/hudi/cli/testutils/HoodieTestCommitMetadataGenerator.java
index 83d1af7dde10..3a15b231ddcf 100644
---
a/hudi-cli/src/test/java/org/apache/hudi/cli/testutils/HoodieTestCommitMetadataGenerator.java
+++
b/hudi-cli/src/test/java/org/apache/hudi/cli/testutils/HoodieTestCommitMetadataGenerator.java
@@ -114,7 +114,11 @@ public class HoodieTestCommitMetadataGenerator extends
HoodieTestDataGenerator {
}
static <T> void createFileWithMetadata(String basePath,
StorageConfiguration<?> configuration, String name, T metadata) throws
IOException {
- Path commitFilePath = new Path(basePath + "/" +
HoodieTableMetaClient.METAFOLDER_NAME + "/" + name);
+ HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder()
+ .setConf(configuration)
+ .setBasePath(basePath)
+ .build();
+ Path commitFilePath = new Path(metaClient.getTimelinePath().toString(),
name);
try (OutputStream os = HadoopFSUtils.getFs(basePath,
configuration).create(commitFilePath, true)) {
COMMIT_METADATA_SER_DE.getInstantWriter(metadata).get().writeToStream(os);
}
