This is an automated email from the ASF dual-hosted git repository.
voonhous pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new 739ab7696979 fix(spark): make slash-separated date partitioning work
on the row writer path (#19648)
739ab7696979 is described below
commit 739ab7696979150304faf221e3344fe4949e6adc
Author: Sepuri Sai Krishna <[email protected]>
AuthorDate: Mon Aug 24 21:14:23 2026 +0530
fix(spark): make slash-separated date partitioning work on the row writer
path (#19648)
* fix(spark): make slash-separated date partitioning work on the row writer
path
* test(spark): clarify partition path formatter test naming and hive-style
case
* address review feedback on slash-separated date partitioning
- assert the InternalRow path for a null partition value at keygen level.
[[KeyGeneratorTestUtilities#getInternalRow]] builds a flat
GenericInternalRow,
so the nested "nested_col.prop1" value stays a Row and reading it back as
a
struct fails; the conversion goes through Spark's CatalystTypeConverters
instead. A null on a top-level field is not an option either -- every
other
field of the example schema is non-nullable and HoodieUnsafeRowUtils
rejects
it before the formatter is reached.
- assert Row + InternalRow for CustomKeyGenerator, which builds one
single-field
sub-key-generator per partition field. The test partitioned on
"timestamp",
a long, which cannot survive the conversion into a Row, so it now uses the
string-typed "ts_ms".
- pin the url-encoding behaviour: encoding runs before the substitution, so
an
already slash-separated value is escaped rather than turned into
directories.
- cover a DATE typed partition column across insert and bulk_insert.
- project datestr in the row-writer assertion so the read-back and the null
row
are covered, and drop the deprecated bulk-insert configs for
hoodie.spark.sql.insert.into.operation.
- drop two formatter assertions already pinned by TestComplexKeyGenerator.
- correct the NOTE on the single-field fast path: CustomKeyGenerator is not
an
exception to it, and getPartitionPath governs the single-field Avro case.
- correct the hive-style comment: the mutual exclusion is documented on
SLASH_SEPARATED_DATE_PARTITIONING but only enforced by HoodieCatalogTable
for
SQL options, so df.write and HoodieStreamer still accept the combination.
* fix(spark): don't slash-separate partition values with a leading dash
A leading dash in a partition value turned into a leading slash, making the
relative partition path absolute. FSUtils#constructAbsolutePath(String,
String)
chops that leading slash while the StoragePath overload used by
AbstractTableFileSystemView lets it URI-resolve the table base path away, so
the writer and the file-system view disagreed on where the partition lives:
"-5" wrote to <base>/5 but was looked up under /5, and "-" collapsed the
partition onto the table root.
Suppress the substitution for such values on all three write paths --
PartitionPathFormatterBase#combine for the Row/InternalRow formatters and
KeyGenUtils#getPartitionPath/#getRecordPartitionPath for Avro.
Also reword the multi-field NOTE: the three write paths agreeing on the
CustomKeyGenerator layout does not make it usable, since it cannot be read
back (HUDI issue #19666).
* docs(spark): move partition path formatter rationale into javadoc
Move the two NOTE blocks out of PartitionPathFormatterBase#combine and onto
the methods they explain -- the single-field/CustomKeyGenerator rationale
onto
replaceDashesWithSlashes, the leading-dash rationale onto startsWithDash --
leaving a two-line pointer at the call site. Comments only, no behavior
change.
* fix(spark): reject slash-separated date partitioning with multiple
partition fields
The multi-field slash layout writes extra path fragments that
doParsePartitionColumnValues cannot line up with the partition columns, so
the write commits cleanly and every subsequent read of the table fails
under the default lazy listing. On master the row-writer leg threw a
ClassCastException, so no table could reach that state; with the CCE fixed
the write succeeded silently. Fail fast at the write instead:
- HoodieWriterUtils#validateTableConfig rejects slash partitioning with
more than one partition field, covering df.write and SQL writes on new
and existing tables
- HoodieCatalogTable#extraTableConfig rejects it at CREATE TABLE, next to
the existing hive-style mutual-exclusion check
- the multi-part substitution in PartitionPathFormatterBase stays: new
writes cannot reach it, and it keeps composeRelativePartitionPath naming
the directory legacy CustomKeyGenerator tables hold on disk
- the multi-field CustomKeyGenerator test becomes rejection asserts at
create and at write time
* test(spark): pin the multi-field slash rejection firing off the table
config alone
An existing table holding slash partitioning with two partition fields can
no longer be created through SQL or df.write, so the write-time rejection
for that legacy shape (e.g. bulk_insert into such a table) had no coverage;
validateTableConfig is exercised with empty write params against a table
config built directly.
* fix(spark): widen the slash-partitioning dash guard to dot segments and
fix the rejection remedy
Review round-4 majors:
- hasPathBreakingDash missed dash-delimited dot segments: "..-a" passed
the leading/trailing/doubled checks, substituted to "../a", and
URI-resolved outside the table base path entirely ("..-..-x" lands in
/tmp/x); escapePathName leaves dots and dashes alone, so url-encoding
does not neutralize it. The guard now rejects any dash-delimited token
that is empty, "." or "..", one rule subsuming the previous three,
char-wise in KeyGenUtils and byte-wise in the UTF8String formatter
('-' and '.' are ASCII, so a byte scan cannot collide with UTF-8
continuation bytes).
- The multi-field rejection message told users to "disable slash-separated
date partitioning", which is not actionable on an existing table: ALTER
TABLE SET TBLPROPERTIES never rewrites hoodie.properties, and an
explicit =false write option trips the config-diff rejection instead.
The message now says to recreate the table.
* chore(spark): address round-4 review minors on slash-separated date
partitioning
- Read the slash flag via equalsIgnoreCase("true") at both validation sites,
matching the Boolean.parseBoolean semantics every other reader of the
config uses (toBoolean threw on values like "1")
- Correct the HoodieCatalogTable comment: the !tableExists gate only avoids
a duplicate error, since validateTableConfig already rejects an existing
multi-field table on that path
- Soften the formatter comments: a HoodieStreamer first write to a
not-yet-existing table bypasses the validation
- Tests: pin the rejection under SaveMode.Overwrite (params alone, ahead
of the overwrite gate); parameterize the SimpleKeyGenerator row-writing
test over url-encoding so encode-before-substitute is pinned on the Avro
path too, and assert the Avro arm of the null case; assert fragments
unique to each rejection site; drop the DATE pruning asserts duplicated
by TestTypedPartitionValues and the upsert dir/MDT asserts duplicated by
the insert test; SAM lambda for checkExceptionContain
---------
Co-authored-by: voon <[email protected]>
---
.../java/org/apache/hudi/keygen/KeyGenUtils.java | 74 ++++-
.../hudi/keygen/PartitionPathFormatterBase.java | 72 ++++-
.../hudi/keygen/StringPartitionPathFormatter.java | 10 +
.../keygen/UTF8StringPartitionPathFormatter.java | 28 ++
.../hudi/keygen/TestComplexKeyGenerator.java | 18 ++
.../apache/hudi/keygen/TestCustomKeyGenerator.java | 15 +-
.../hudi/keygen/TestPartitionPathFormatter.java | 135 +++++++++
.../apache/hudi/keygen/TestSimpleKeyGenerator.java | 77 ++++++
.../scala/org/apache/hudi/HoodieWriterUtils.scala | 28 ++
.../sql/catalyst/catalog/HoodieCatalogTable.scala | 13 +
.../org/apache/hudi/TestHoodieWriterUtils.java | 32 +++
.../common/TestSlashSeparatedPartitionValue.scala | 304 +++++++++++++++------
12 files changed, 714 insertions(+), 92 deletions(-)
diff --git
a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java
b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java
index 6586e0ee385f..37dc5c8a2e4b 100644
---
a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java
+++
b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java
@@ -254,12 +254,15 @@ public class KeyGenUtils {
if (encodePartitionPath) {
fieldVal = PartitionPathEncodeUtils.escapePathName(fieldVal);
}
+ // NOTE: See [[slashSeparateDateValue]] on which dashes suppress the
substitution. It runs
+ // before the hive-style prefix so that the guard inspects the
bare value, matching
+ // [[PartitionPathFormatterBase#combine]] -- on "dt=-5" no guard
could ever fire
+ if (partitionPathFields.size() == 1 && slashSeparatedDatePartitioning)
{
+ fieldVal = slashSeparateDateValue(fieldVal);
+ }
if (hiveStylePartitioning) {
fieldVal = partitionPathField + "=" + fieldVal;
}
- if (partitionPathFields.size() == 1 && slashSeparatedDatePartitioning)
{
- fieldVal = fieldVal.replace('-', '/');
- }
partitionPath.append(fieldVal);
}
if (i != partitionPathFields.size() - 1) {
@@ -288,15 +291,74 @@ public class KeyGenUtils {
if (encodePartitionPath) {
partitionPath = PartitionPathEncodeUtils.escapePathName(partitionPath);
}
+ // NOTE: See [[slashSeparateDateValue]] on which dashes suppress the
substitution. It runs
+ // before the hive-style prefix so that the guard inspects the bare
value, matching
+ // [[PartitionPathFormatterBase#combine]] -- on "dt=-5" no guard
could ever fire
+ if (slashSeparatedDatePartitioning) {
+ partitionPath = slashSeparateDateValue(partitionPath);
+ }
if (hiveStylePartitioning) {
partitionPath = partitionPathField + "=" + partitionPath;
}
- if (slashSeparatedDatePartitioning) {
- partitionPath = partitionPath.replace('-', '/');
- }
return partitionPath;
}
+ /**
+ * Turns a {@code yyyy-MM-dd} formatted date value into the {@code
yyyy/MM/dd} directory structure
+ * requested by {@code
hoodie.datasource.write.slash.separated.date.partitioning}.
+ *
+ * <p>A value whose dash-delimited tokens would produce a path-breaking
segment -- an empty
+ * token, {@code "."} or {@code ".."} -- is returned as-is, because the
resulting path does not
+ * survive the round trip back from storage:
+ *
+ * <ul>
+ * <li>empty token, leading -- {@code "-5"} becomes {@code "/5"}, and an
absolute
+ * relative-partition-path is resolved inconsistently:
+ * {@code FSUtils#constructAbsolutePath(String, String)} chops the leading
{@code "/"} while the
+ * {@link org.apache.hudi.storage.StoragePath} overload used by
+ * {@code AbstractTableFileSystemView} lets it URI-resolve away the table
base path (the writer
+ * lands in {@code "<base>/5"} but the file-system view in {@code "/5"},
and {@code "-"}
+ * resolves to the base path itself)</li>
+ * <li>empty token, trailing or interior -- {@code "5-"} becomes {@code
"5/"}, which
+ * {@code StoragePath} normalizes back to {@code "5"}, and {@code "a--b"}
becomes
+ * {@code "a//b"}, which {@code URI.normalize()} collapses to {@code
"a/b"} -- either way the
+ * writer records a partition string that no longer names its directory, so
+ * {@code FSUtils#getFileName} slices the file name at the wrong offset
and the metadata-table
+ * FILES partition ends up with a truncated entry</li>
+ * <li>dot segment -- {@code "2026-.-05"} becomes {@code "2026/./05"},
which URI-normalizes to
+ * {@code "2026/05"}, and {@code "..-a"} becomes {@code "../a"}, which
resolves OUTSIDE the
+ * table base path entirely ({@code
PartitionPathEncodeUtils#escapePathName} leaves dots and
+ * dashes alone, so url-encoding does not neutralize this)</li>
+ * </ul>
+ *
+ * <p>None of these values is a date to begin with, so nothing is lost by
not slashing them.
+ */
+ private static String slashSeparateDateValue(String partitionPath) {
+ return hasPathBreakingDash(partitionPath) ? partitionPath :
partitionPath.replace('-', '/');
+ }
+
+ /**
+ * Whether substituting the dashes in {@code partitionPath} would yield a
path-breaking segment:
+ * any dash-delimited token that is empty (a leading, trailing or doubled
dash), {@code "."} or
+ * {@code ".."}. See {@link #slashSeparateDateValue} for why each case is
excluded.
+ */
+ public static boolean hasPathBreakingDash(String partitionPath) {
+ int tokenStart = 0;
+ int length = partitionPath.length();
+ for (int i = 0; i <= length; i++) {
+ if (i == length || partitionPath.charAt(i) == '-') {
+ int tokenLength = i - tokenStart;
+ if (tokenLength == 0
+ || (tokenLength == 1 && partitionPath.charAt(tokenStart) == '.')
+ || (tokenLength == 2 && partitionPath.charAt(tokenStart) == '.' &&
partitionPath.charAt(tokenStart + 1) == '.')) {
+ return true;
+ }
+ tokenStart = i + 1;
+ }
+ }
+ return false;
+ }
+
/**
* Create a date time parser class for TimestampBasedKeyGenerator, passing
in any configs needed.
*/
diff --git
a/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/PartitionPathFormatterBase.java
b/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/PartitionPathFormatterBase.java
index d08545cf8fd5..0332b24af0bd 100644
---
a/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/PartitionPathFormatterBase.java
+++
b/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/PartitionPathFormatterBase.java
@@ -62,11 +62,12 @@ public abstract class PartitionPathFormatterBase<S> {
// Avoid creating [[StringBuilder]] in case there's just one
partition-path part,
// and Hive-style of partitioning is not required
if (!useHiveStylePartitioning && partitionPathParts.length == 1) {
- if (slashSeparatedDatePartitioning) {
- return ((S) ((String) toString(partitionPathParts[0])).replace('-',
'/'));
- } else {
- return tryEncode(handleEmpty(toString(partitionPathParts[0])));
- }
+ S partitionPathPart =
tryEncode(handleEmpty(toString(partitionPathParts[0])));
+ // NOTE: See [[replaceDashesWithSlashes]] on how this lines up with the
Avro write path, and
+ // [[hasPathBreakingDash]] on which dashes suppress the
substitution
+ return slashSeparatedDatePartitioning &&
!hasPathBreakingDash(partitionPathPart)
+ ? replaceDashesWithSlashes(partitionPathPart)
+ : partitionPathPart;
}
StringBuilder<S> sb = stringBuilderFactory.get();
@@ -77,9 +78,16 @@ public abstract class PartitionPathFormatterBase<S> {
sb.appendJava(partitionPathFields.get(i))
.appendJava("=")
.append(partitionPathPartStr);
- } else if (slashSeparatedDatePartitioning) {
- String res = ((String) partitionPathPartStr).replace('-', '/');
- sb.append(((S) res));
+ } else if (slashSeparatedDatePartitioning &&
!hasPathBreakingDash(partitionPathPartStr)) {
+ // NOTE: Every part is substituted here, preserving the behaviour this
branch had before the
+ // [[ClassCastException]] fix. Spark writes reject slash
partitioning with more than
+ // one partition field
([[HoodieWriterUtils#validateTableConfig]]), so this branch
+ // mainly serves reads of tables predating that rejection:
[[CustomKeyGenerator]] built
+ // one single-field sub-keygen per field, so such tables slashed
each field
+ // individually, and
[[SparkHoodieTableFileIndex#composeRelativePartitionPath]] has to
+ // land on the same directory when it composes the prefix in one
[[combine]] call over
+ // all N columns. See HUDI issue #19666
+ sb.append(replaceDashesWithSlashes(partitionPathPartStr));
} else {
sb.append(partitionPathPartStr);
}
@@ -102,6 +110,54 @@ public abstract class PartitionPathFormatterBase<S> {
protected abstract S handleEmpty(S partitionPathPart);
+ /**
+ * Turns a {@code yyyy-MM-dd} formatted date value into the {@code
yyyy/MM/dd} directory structure
+ * requested by {@code
hoodie.datasource.write.slash.separated.date.partitioning}.
+ *
+ * <p>NOTE: This has to be implemented by every sub-class, since the
substitution has to be
+ * performed on the concrete string representation {@code S} the formatter
operates on.
+ *
+ * <p>NOTE: For {@code SimpleKeyGenerator}/{@code ComplexKeyGenerator} the
single-part branch of
+ * {@link #combine} routes only a table partitioned by a single (date)
column here, mirroring
+ * {@code KeyGenUtils#getPartitionPath} (single field) and {@code
KeyGenUtils#getRecordPartitionPath}
+ * (which guards on a single field as well) driving the Avro write-path:
both write-paths have to
+ * derive the very same partition path for a record.
+ *
+ * <p>NOTE: The multi-part branch substitutes every part, which reads of
legacy tables depend on:
+ * {@code CustomKeyGenerator} built one single-field sub-key-generator per
partition field, so
+ * such tables slashed each field individually, and
+ * {@code SparkHoodieTableFileIndex#composeRelativePartitionPath} -- which
calls {@link #combine}
+ * once over all N columns to compose a listing prefix -- has to name the
very same directory, or
+ * the prefix misses and the query silently returns no rows. Spark writes
cannot reach this
+ * branch with slash partitioning enabled: multi-field slash tables produce
a layout the extra
+ * fragments leave {@code HoodieSparkUtils#doParsePartitionColumnValues}
unable to line up with
+ * the partition columns, so {@code HoodieWriterUtils#validateTableConfig}
rejects them at write
+ * time (a HoodieStreamer first write to a not-yet-existing table bypasses
that validation).
+ * See HUDI issue #19666.
+ */
+ protected abstract S replaceDashesWithSlashes(S partitionPathPart);
+
+ /**
+ * Whether substituting the dashes in {@code partitionPathPart} would yield
a path-breaking
+ * segment -- any dash-delimited token that is empty (a leading, trailing or
doubled dash),
+ * {@code "."} or {@code ".."} -- in which case {@link
#replaceDashesWithSlashes(Object)} must not
+ * be applied to it. Kept in step with {@code
KeyGenUtils#hasPathBreakingDash}, which guards the
+ * Avro write path and documents each case.
+ *
+ * <p>NOTE: This has to be implemented by every sub-class, since the check
has to be performed on
+ * the concrete string representation {@code S} the formatter operates on.
+ *
+ * <p>NOTE: None of these shapes survives the round trip back from storage.
An empty token turns
+ * the path absolute ({@code "-5"} -> {@code "/5"}, resolved inconsistently
by the two
+ * {@code FSUtils#constructAbsolutePath} overloads) or leaves the recorded
partition string
+ * longer than the directory it normalizes to ({@code "5-"} -> {@code "5/"},
+ * {@code "a--b"} -> {@code "a//b"}), so {@code FSUtils#getFileName} slices
the file name at the
+ * wrong offset. A dot segment is resolved away by {@code URI.normalize()}
-- {@code "..-a"}
+ * becomes {@code "../a"} and escapes the table base path entirely. None of
these values is a
+ * date to begin with, so nothing is lost by not slashing them.
+ */
+ protected abstract boolean hasPathBreakingDash(S partitionPathPart);
+
/**
* This is a generic interface closing the gap and unifying the {@link
java.lang.StringBuilder} with
* {@link org.apache.hudi.unsafe.UTF8StringBuilder} implementations,
allowing us to avoid code-duplication by performing
diff --git
a/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/StringPartitionPathFormatter.java
b/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/StringPartitionPathFormatter.java
index 5709193b5fdf..4739b6ca59f7 100644
---
a/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/StringPartitionPathFormatter.java
+++
b/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/StringPartitionPathFormatter.java
@@ -56,6 +56,16 @@ public class StringPartitionPathFormatter extends
PartitionPathFormatterBase<Str
}
}
+ @Override
+ protected String replaceDashesWithSlashes(String partitionPathPart) {
+ return partitionPathPart.replace('-', '/');
+ }
+
+ @Override
+ protected boolean hasPathBreakingDash(String partitionPathPart) {
+ return KeyGenUtils.hasPathBreakingDash(partitionPathPart);
+ }
+
public static class JavaStringBuilder implements
PartitionPathFormatterBase.StringBuilder<String> {
private final java.lang.StringBuilder sb = new java.lang.StringBuilder();
diff --git
a/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/UTF8StringPartitionPathFormatter.java
b/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/UTF8StringPartitionPathFormatter.java
index f9da94ce8bc0..30fac138bfcc 100644
---
a/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/UTF8StringPartitionPathFormatter.java
+++
b/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/UTF8StringPartitionPathFormatter.java
@@ -33,6 +33,9 @@ public class UTF8StringPartitionPathFormatter extends
PartitionPathFormatterBase
protected static final UTF8String HUDI_DEFAULT_PARTITION_PATH_UTF8 =
UTF8String.fromString(HUDI_DEFAULT_PARTITION_PATH);
+ private static final UTF8String DASH_UTF8 = UTF8String.fromString("-");
+ private static final UTF8String SLASH_UTF8 = UTF8String.fromString("/");
+
public UTF8StringPartitionPathFormatter(Supplier<StringBuilder<UTF8String>>
stringBuilderFactory,
boolean useHiveStylePartitioning,
boolean useEncoding,
@@ -59,6 +62,31 @@ public class UTF8StringPartitionPathFormatter extends
PartitionPathFormatterBase
return partitionPathPart;
}
+ @Override
+ protected UTF8String replaceDashesWithSlashes(UTF8String partitionPathPart) {
+ return partitionPathPart.replace(DASH_UTF8, SLASH_UTF8);
+ }
+
+ @Override
+ protected boolean hasPathBreakingDash(UTF8String partitionPathPart) {
+ // Byte-wise mirror of KeyGenUtils#hasPathBreakingDash: '-' and '.' are
ASCII, so they can
+ // never collide with a UTF-8 continuation byte and a plain byte scan is
exact
+ byte[] bytes = partitionPathPart.getBytes();
+ int tokenStart = 0;
+ for (int i = 0; i <= bytes.length; i++) {
+ if (i == bytes.length || bytes[i] == '-') {
+ int tokenLength = i - tokenStart;
+ if (tokenLength == 0
+ || (tokenLength == 1 && bytes[tokenStart] == '.')
+ || (tokenLength == 2 && bytes[tokenStart] == '.' &&
bytes[tokenStart + 1] == '.')) {
+ return true;
+ }
+ tokenStart = i + 1;
+ }
+ }
+ return false;
+ }
+
public static class UTF8StringBuilder implements StringBuilder<UTF8String> {
private final org.apache.hudi.unsafe.UTF8StringBuilder sb = new
org.apache.hudi.unsafe.UTF8StringBuilder();
diff --git
a/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/keygen/TestComplexKeyGenerator.java
b/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/keygen/TestComplexKeyGenerator.java
index 3bb69e0424fb..877125e5d45e 100644
---
a/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/keygen/TestComplexKeyGenerator.java
+++
b/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/keygen/TestComplexKeyGenerator.java
@@ -244,6 +244,24 @@ public class TestComplexKeyGenerator extends
KeyGeneratorTestUtilities {
assertEquals("2026/01/05", key.getPartitionPath());
}
+ @Test
+ void testSlashSeparatedDatePartitioningLeavesLeadingDashesAlone() {
+ TypedProperties properties = new TypedProperties();
+ properties.put(KeyGeneratorOptions.RECORDKEY_FIELD_NAME.key(), "_row_key");
+ properties.put(KeyGeneratorOptions.PARTITIONPATH_FIELD_NAME.key(),
"timestamp");
+
properties.put(KeyGeneratorOptions.SLASH_SEPARATED_DATE_PARTITIONING.key(),
"true");
+ properties.put(KeyGeneratorOptions.HIVE_STYLE_PARTITIONING_ENABLE.key(),
"false");
+
+ ComplexKeyGenerator keyGenerator = new ComplexKeyGenerator(properties);
+
+ // This is the only production caller of the multi-field
KeyGenUtils#getRecordPartitionPath
+ // overload, so the guard on that line is unexercised without this case
+ GenericRecord avroRecord = KeyGeneratorTestUtilities.getRecord();
+ avroRecord.put("timestamp", "-5");
+
+ assertEquals("-5", keyGenerator.getKey(avroRecord).getPartitionPath());
+ }
+
@Test
void testSlashSeparatedDatePartitioningWithAlreadyFormattedInput() {
TypedProperties properties = new TypedProperties();
diff --git
a/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/keygen/TestCustomKeyGenerator.java
b/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/keygen/TestCustomKeyGenerator.java
index 6e526ccb6c73..b2bbf93eafca 100644
---
a/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/keygen/TestCustomKeyGenerator.java
+++
b/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/keygen/TestCustomKeyGenerator.java
@@ -419,7 +419,9 @@ class TestCustomKeyGenerator extends
KeyGeneratorTestUtilities {
void testSlashSeparatedDatePartitioning() {
TypedProperties properties = new TypedProperties();
properties.put(KeyGeneratorOptions.RECORDKEY_FIELD_NAME.key(), "_row_key");
- properties.put(KeyGeneratorOptions.PARTITIONPATH_FIELD_NAME.key(),
"timestamp:simple");
+ // NOTE: "ts_ms" is the string-typed field of the example schema,
"timestamp" is a long, so only
+ // the former survives the conversion into a [[Row]]/[[InternalRow]]
+ properties.put(KeyGeneratorOptions.PARTITIONPATH_FIELD_NAME.key(),
"ts_ms:simple");
properties.put(KeyGeneratorOptions.SLASH_SEPARATED_DATE_PARTITIONING.key(),
"true");
properties.put(KeyGeneratorOptions.HIVE_STYLE_PARTITIONING_ENABLE.key(),
"false");
properties.put(HoodieWriteConfig.KEYGENERATOR_CLASS_NAME.key(),
CustomKeyGenerator.class.getName());
@@ -429,12 +431,21 @@ class TestCustomKeyGenerator extends
KeyGeneratorTestUtilities {
// Create a record with date in yyyy-MM-dd format
GenericRecord avroRecord = KeyGeneratorTestUtilities.getRecord();
- avroRecord.put("timestamp", "2026-01-05");
+ avroRecord.put("ts_ms", "2026-01-05");
// The partition path should be transformed to yyyy/MM/dd format
HoodieKey key = keyGenerator.getKey(avroRecord);
assertEquals("key1", key.getRecordKey());
assertEquals("2026/01/05", key.getPartitionPath());
+
+ // NOTE: [[CustomKeyGenerator]] builds one single-field sub-key-generator
per partition field, so
+ // the row-writer paths have to derive the very same partition path
as the Avro one above
+ Row row = KeyGeneratorTestUtilities.getRow(avroRecord);
+ assertEquals("2026/01/05", keyGenerator.getPartitionPath(row));
+
+ InternalRow internalRow = KeyGeneratorTestUtilities.getInternalRow(row);
+ assertEquals(UTF8String.fromString("2026/01/05"),
+ keyGenerator.getPartitionPath(internalRow, row.schema()));
}
@Test
diff --git
a/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/keygen/TestPartitionPathFormatter.java
b/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/keygen/TestPartitionPathFormatter.java
new file mode 100644
index 000000000000..47911b9eacd5
--- /dev/null
+++
b/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/keygen/TestPartitionPathFormatter.java
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.keygen;
+
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+
+import static org.apache.hudi.keygen.KeyGenUtils.HUDI_DEFAULT_PARTITION_PATH;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+/**
+ * Tests the partition-path formatters backing the key generators, making sure
that both the
+ * {@link String} (Avro/{@link org.apache.spark.sql.Row} write path) and the
{@link UTF8String}
+ * (row-writer/{@link org.apache.spark.sql.catalyst.InternalRow} write path)
flavors produce
+ * identical partition paths.
+ */
+class TestPartitionPathFormatter {
+
+ private static final List<String> SINGLE_FIELD =
Collections.singletonList("date_col");
+ private static final List<String> TWO_FIELDS = Arrays.asList("date_col",
"city");
+
+ private String combine(boolean useRowWriterPath,
+ boolean hiveStylePartitioning,
+ boolean encode,
+ boolean slashSeparatedDatePartitioning,
+ List<String> fields,
+ Object... parts) {
+ if (useRowWriterPath) {
+ return new UTF8StringPartitionPathFormatter(
+ UTF8StringPartitionPathFormatter.UTF8StringBuilder::new,
hiveStylePartitioning, encode,
+ slashSeparatedDatePartitioning).combine(fields, parts).toString();
+ }
+ return new StringPartitionPathFormatter(
+ StringPartitionPathFormatter.JavaStringBuilder::new,
hiveStylePartitioning, encode,
+ slashSeparatedDatePartitioning).combine(fields, parts);
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void testSlashSeparatedDatePartitioningAppliesToEveryField(boolean
useRowWriterPath) {
+ // NOTE: Every part is substituted, which is what legacy
[[CustomKeyGenerator]] tables hold on
+ // disk (one single-field sub-keygen per field) and what
+ // [[SparkHoodieTableFileIndex#composeRelativePartitionPath]] has to
reproduce when it
+ // composes a listing prefix over all N columns in one call. New
writes cannot reach this
+ // combination -- [[HoodieWriterUtils#validateTableConfig]] rejects
slash partitioning
+ // with more than one partition field. See HUDI issue #19666
+ assertEquals("2026/01/05/san/francisco",
+ combine(useRowWriterPath, false, false, true, TWO_FIELDS,
"2026-01-05", "san-francisco"));
+ // The guard applies per part, not just to the first one
+ assertEquals("2026/01/05/-5",
+ combine(useRowWriterPath, false, false, true, TWO_FIELDS,
"2026-01-05", "-5"));
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void testSlashSeparatedDatePartitioningHandlesNullAndEmptyValues(boolean
useRowWriterPath) {
+ assertEquals(HUDI_DEFAULT_PARTITION_PATH,
+ combine(useRowWriterPath, false, false, true, SINGLE_FIELD, new
Object[] {null}));
+ assertEquals(HUDI_DEFAULT_PARTITION_PATH,
+ combine(useRowWriterPath, false, false, true, SINGLE_FIELD, ""));
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void testSlashSeparatedDatePartitioningLeavesPathBreakingDashesAlone(boolean
useRowWriterPath) {
+ // NOTE: Substituting in any of these would yield a partition path that
does not survive the
+ // round trip back from storage. A leading "/" is resolved
differently by
+ // [[FSUtils#constructAbsolutePath(String, String)]] and the
[[StoragePath]] overload used
+ // by [[AbstractTableFileSystemView]] -- the former chops it, the
latter URI-resolves the
+ // table base path away. A trailing "/" is normalized off by
[[StoragePath#normalize]] and
+ // a doubled "//" is collapsed by [[java.net.URI#normalize]],
leaving the writer recording
+ // a partition string longer than the directory it actually resolves
to
+ assertEquals("-5", combine(useRowWriterPath, false, false, true,
SINGLE_FIELD, "-5"));
+ assertEquals("-", combine(useRowWriterPath, false, false, true,
SINGLE_FIELD, "-"));
+ assertEquals("--5", combine(useRowWriterPath, false, false, true,
SINGLE_FIELD, "--5"));
+ assertEquals("5-", combine(useRowWriterPath, false, false, true,
SINGLE_FIELD, "5-"));
+ assertEquals("a--b", combine(useRowWriterPath, false, false, true,
SINGLE_FIELD, "a--b"));
+ // A dash-delimited "." or ".." would become a URI dot segment:
"2026/./05" normalizes to
+ // "2026/05", and "../a" resolves outside the table base path entirely
+ assertEquals("2026-.-05", combine(useRowWriterPath, false, false, true,
SINGLE_FIELD, "2026-.-05"));
+ assertEquals("a-..-b", combine(useRowWriterPath, false, false, true,
SINGLE_FIELD, "a-..-b"));
+ assertEquals("..-a", combine(useRowWriterPath, false, false, true,
SINGLE_FIELD, "..-a"));
+ // A single interior dash is still a separator, and dots inside a token
are untouched
+ assertEquals("2026/01/05", combine(useRowWriterPath, false, false, true,
SINGLE_FIELD, "2026-01-05"));
+ assertEquals("v1.2/v3.4", combine(useRowWriterPath, false, false, true,
SINGLE_FIELD, "v1.2-v3.4"));
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void testSlashSeparatedDatePartitioningEncodesValues(boolean
useRowWriterPath) {
+ // '?' has to be escaped, while the date separators are turned into
directory separators
+ assertEquals("2026/01/05", combine(useRowWriterPath, false, true, true,
SINGLE_FIELD, "2026-01-05"));
+ assertEquals("a%3Fb", combine(useRowWriterPath, false, true, true,
SINGLE_FIELD, "a?b"));
+ // Encoding runs before the substitution (parity with KeyGenUtils), so an
already slash-separated
+ // value is escaped rather than turned into directories
+ assertEquals("2026%2F01%2F05", combine(useRowWriterPath, false, true,
true, SINGLE_FIELD, "2026/01/05"));
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void testHiveStylePartitioningTakesPrecedence(boolean useRowWriterPath) {
+ // NOTE: Hive-style partitioning and slash-separated date partitioning are
documented as mutually
+ // exclusive
([[KeyGeneratorOptions#SLASH_SEPARATED_DATE_PARTITIONING]]), but only
+ // [[HoodieCatalogTable#extraTableConfig]] enforces it, and it
inspects the SQL options
+ // alone -- df.write and HoodieStreamer still accept the
combination. The formatter
+ // deliberately leaves the value alone here rather than mirroring
the Avro path, which
+ // produces a layout
[[HoodieSparkUtils#doParsePartitionColumnValues]] cannot read back.
+ // This asserts the pre-existing behavior stays put, it is not a
statement about what the
+ // combination *should* produce
+ assertEquals("date_col=2026-01-05",
+ combine(useRowWriterPath, true, false, true, SINGLE_FIELD,
"2026-01-05"));
+ }
+}
diff --git
a/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/keygen/TestSimpleKeyGenerator.java
b/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/keygen/TestSimpleKeyGenerator.java
index 43d20fee2919..b8739066cb3a 100644
---
a/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/keygen/TestSimpleKeyGenerator.java
+++
b/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/keygen/TestSimpleKeyGenerator.java
@@ -29,12 +29,14 @@ import org.apache.hudi.keygen.constant.KeyGeneratorOptions;
import org.apache.avro.generic.GenericData;
import org.apache.avro.generic.GenericRecord;
import org.apache.spark.sql.Row;
+import org.apache.spark.sql.catalyst.CatalystTypeConverters;
import org.apache.spark.sql.catalyst.InternalRow;
import org.apache.spark.unsafe.types.UTF8String;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.MethodSource;
+import org.junit.jupiter.params.provider.ValueSource;
import java.util.stream.Stream;
@@ -213,4 +215,79 @@ class TestSimpleKeyGenerator extends
KeyGeneratorTestUtilities {
Assertions.assertEquals("key1", key.getRecordKey());
Assertions.assertEquals("2026/01/01", key.getPartitionPath());
}
+
+ @Test
+ void testSlashSeparatedDatePartitioningLeavesLeadingDashesAlone() {
+ SimpleKeyGenerator keyGenerator = new
SimpleKeyGenerator(getPropsWithSlashSeparatedDatePartitioning());
+
+ // NOTE: Substituting here would yield a partition path starting with "/",
which
+ // [[FSUtils#constructAbsolutePath(String, String)]] and the
[[StoragePath]] overload used
+ // by [[AbstractTableFileSystemView]] resolve differently -- the
former chops the leading
+ // "/", the latter URI-resolves the table base path away -- so the
writer and the
+ // file-system view would disagree on where the partition lives
+ GenericRecord avroRecord = new
GenericData.Record(HoodieSchema.parse(KeyGeneratorTestUtilities.EXAMPLE_SCHEMA).getAvroSchema());
+ avroRecord.put("timestamp", "-5");
+ avroRecord.put("_row_key", "key1");
+ avroRecord.put("ts_ms", "-5");
+ avroRecord.put("pii_col", "val1");
+
+ HoodieKey key = keyGenerator.getKey(avroRecord);
+ Assertions.assertEquals("key1", key.getRecordKey());
+ Assertions.assertEquals("-5", key.getPartitionPath());
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void testSlashSeparatedDatePartitioningOnRowWritingPaths(boolean urlEncode) {
+ TypedProperties properties = getPropsWithSlashSeparatedDatePartitioning();
+ // NOTE: "ts_ms" is the string-typed field of the example schema,
"timestamp" is a long
+ properties.put(KeyGeneratorOptions.PARTITIONPATH_FIELD_NAME.key(),
"ts_ms");
+ properties.put(KeyGeneratorOptions.URL_ENCODE_PARTITIONING.key(),
String.valueOf(urlEncode));
+ SimpleKeyGenerator keyGenerator = new SimpleKeyGenerator(properties);
+
+ GenericRecord avroRecord = getRecord();
+ Assertions.assertEquals("2020/03/21",
keyGenerator.getPartitionPath(avroRecord));
+
+ Row row = KeyGeneratorTestUtilities.getRow(avroRecord);
+ Assertions.assertEquals("2020/03/21", keyGenerator.getPartitionPath(row));
+
+ InternalRow internalRow = KeyGeneratorTestUtilities.getInternalRow(row);
+ Assertions.assertEquals(UTF8String.fromString("2020/03/21"),
+ keyGenerator.getPartitionPath(internalRow, row.schema()));
+
+ // Encoding runs before the substitution on all three write paths, so an
escapable character
+ // is escaped while the dash still becomes a directory separator. This
pins the ordering at
+ // the KeyGenUtils/Avro level too, which the formatter-level encode test
cannot reach
+ avroRecord.put("ts_ms", "a?b-c");
+ String expected = urlEncode ? "a%3Fb/c" : "a?b/c";
+ Assertions.assertEquals(expected,
keyGenerator.getPartitionPath(avroRecord));
+
+ Row encodedRow = KeyGeneratorTestUtilities.getRow(avroRecord);
+ Assertions.assertEquals(expected,
keyGenerator.getPartitionPath(encodedRow));
+ Assertions.assertEquals(UTF8String.fromString(expected),
+
keyGenerator.getPartitionPath(KeyGeneratorTestUtilities.getInternalRow(encodedRow),
encodedRow.schema()));
+ }
+
+ @Test
+ void testSlashSeparatedDatePartitioningWithNullValue() {
+ TypedProperties properties = getPropsWithSlashSeparatedDatePartitioning();
+ properties.put(KeyGeneratorOptions.PARTITIONPATH_FIELD_NAME.key(),
"nested_col.prop1");
+ SimpleKeyGenerator keyGenerator = new SimpleKeyGenerator(properties);
+
+ GenericRecord avroRecord = getRecord(getNestedColRecord(null, 10L));
+ // The Avro arm covers the HUDI-1888 class (NPE on a null nested partition
value) under slash
+ Assertions.assertEquals(HUDI_DEFAULT_PARTITION_PATH,
keyGenerator.getPartitionPath(avroRecord));
+
+ Row row = KeyGeneratorTestUtilities.getRow(avroRecord);
+ Assertions.assertEquals(HUDI_DEFAULT_PARTITION_PATH,
keyGenerator.getPartitionPath(row));
+
+ // NOTE: [[KeyGeneratorTestUtilities#getInternalRow]] builds a flat
[[GenericInternalRow]], leaving
+ // a nested value as a [[Row]], so the conversion has to go through
Spark here. "nested_col.prop1"
+ // is the only nullable field of the example schema, and a null on a
non-nullable one is
+ // rejected by [[org.apache.spark.sql.HoodieUnsafeRowUtils]] before
the formatter is reached
+ InternalRow internalRow =
+ (InternalRow)
CatalystTypeConverters.createToCatalystConverter(row.schema()).apply(row);
+ Assertions.assertEquals(UTF8String.fromString(HUDI_DEFAULT_PARTITION_PATH),
+ keyGenerator.getPartitionPath(internalRow, row.schema()));
+ }
}
diff --git
a/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieWriterUtils.scala
b/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieWriterUtils.scala
index b831ec165ff9..9fc7b83cc9ec 100644
---
a/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieWriterUtils.scala
+++
b/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieWriterUtils.scala
@@ -262,6 +262,34 @@ object HoodieWriterUtils {
*/
def validateTableConfig(spark: SparkSession, params: Map[String, String],
tableConfig: HoodieTableConfig, isOverWriteMode:
Boolean): Unit = {
+ // Fail fast on writes that would produce a slash-separated layout with
more than one
+ // partition field. The extra path fragments cannot be lined up with the
partition columns on
+ // read (HUDI issue #19666), so without this check the write commits
cleanly and every
+ // subsequent read of the table fails. Checked regardless of save mode,
since an Overwrite
+ // produces the same layout.
+ // NOTE: equalsIgnoreCase rather than toBoolean, since every other reader
of this flag goes
+ // through Boolean.parseBoolean semantics and treats a value like
"1" as false, while
+ // toBoolean would throw on it
+ val slashSeparatedDatePartitioning =
+
params.get(HoodieTableConfig.SLASH_SEPARATED_DATE_PARTITIONING.key).exists(_.equalsIgnoreCase("true"))
||
+ (null != tableConfig && tableConfig.getSlashSeparatedDatePartitioning)
+ if (slashSeparatedDatePartitioning) {
+ // The table config is the source of truth for an existing table; for a
new one the
+ // datasource value may still carry the CustomKeyGenerator "field:type"
format, which the
+ // comma count is insensitive to
+ val partitionFields = Option(tableConfig)
+ .flatMap(tc =>
Option(HoodieTableConfig.getPartitionFieldProp(tc).orElse(null)))
+ .filter(_.nonEmpty)
+ .getOrElse(params.getOrElse(PARTITIONPATH_FIELD.key(), ""))
+ val partitionFieldCount =
partitionFields.split(",").count(_.trim.nonEmpty)
+ if (partitionFieldCount > 1) {
+ throw new
HoodieException(s"${HoodieTableConfig.SLASH_SEPARATED_DATE_PARTITIONING.key}
requires"
+ + s" a single partition field, but found $partitionFieldCount:
$partitionFields."
+ + " The slash-separated layout of a multi-field partition path
cannot be read back."
+ + " Recreate the table with a single date partition field or without
this config --"
+ + " an existing table's config cannot be changed in place.")
+ }
+ }
// If Overwrite is set as save mode, we don't need to do table config
validation.
if (!isOverWriteMode) {
val resolver = spark.sessionState.conf.resolver
diff --git
a/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/catalyst/catalog/HoodieCatalogTable.scala
b/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/catalyst/catalog/HoodieCatalogTable.scala
index 5f772cb5a5b0..dab105adcba0 100644
---
a/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/catalyst/catalog/HoodieCatalogTable.scala
+++
b/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/catalyst/catalog/HoodieCatalogTable.scala
@@ -298,6 +298,19 @@ class HoodieCatalogTable(val spark: SparkSession, var
table: CatalogTable) exten
&&
sqlOptions.contains(KeyGeneratorOptions.SLASH_SEPARATED_DATE_PARTITIONING.key)),
s"Table configs cannot contain both
${HIVE_STYLE_PARTITIONING_ENABLE.key} "
+ s"and ${KeyGeneratorOptions.SLASH_SEPARATED_DATE_PARTITIONING.key}")
+ // Reject slash-separated date partitioning with more than one partition
field at table
+ // creation: the resulting layout cannot be read back (HUDI issue #19666).
+ // HoodieWriterUtils#validateTableConfig enforces the same invariant at
write time and already
+ // rejects a pre-existing table on the tableExists branch
(parseSchemaAndConfigs calls it
+ // before reaching here), so the !tableExists gate only avoids a second,
differently-worded
+ // error on that path.
+ if (!tableExists
+ &&
sqlOptions.get(KeyGeneratorOptions.SLASH_SEPARATED_DATE_PARTITIONING.key).exists(_.equalsIgnoreCase("true")))
{
+ ValidationUtils.checkArgument(table.partitionColumnNames.size <= 1,
+ s"${KeyGeneratorOptions.SLASH_SEPARATED_DATE_PARTITIONING.key}
requires a single partition"
+ + s" field, but found ${table.partitionColumnNames.size}:"
+ + s" ${table.partitionColumnNames.mkString(",")}")
+ }
val extraConfig = mutable.Map.empty[String, String]
if (tableExists) {
val allPartitionPaths = getPartitionPaths
diff --git
a/hudi-spark-datasource/hudi-spark-common/src/test/java/org/apache/hudi/TestHoodieWriterUtils.java
b/hudi-spark-datasource/hudi-spark-common/src/test/java/org/apache/hudi/TestHoodieWriterUtils.java
index 2f745beb2935..54b8ddba1c13 100644
---
a/hudi-spark-datasource/hudi-spark-common/src/test/java/org/apache/hudi/TestHoodieWriterUtils.java
+++
b/hudi-spark-datasource/hudi-spark-common/src/test/java/org/apache/hudi/TestHoodieWriterUtils.java
@@ -60,6 +60,38 @@ class TestHoodieWriterUtils extends HoodieClientTestBase {
assertEquals("randomKey",
HoodieWriterUtils.getKeyInTableConfig("randomKey", null));
}
+ @Test
+ void validateTableConfigRejectsMultiFieldSlashPartitioningOnLegacyTable()
throws IOException {
+ // A legacy table already holding slash-separated date partitioning with
two partition fields:
+ // such a table can no longer be created through SQL or df.write, so the
rejection has to fire
+ // off the table config alone, with the write providing no slash or
partition configs of its own
+ Properties props = new Properties();
+ props.setProperty(HoodieTableConfig.PARTITION_FIELDS.key(),
"datestr,city");
+
props.setProperty(HoodieTableConfig.SLASH_SEPARATED_DATE_PARTITIONING.key(),
"true");
+ HoodieTableMetaClient tableMetaClient =
getMetaClientBuilder(HoodieTableType.COPY_ON_WRITE, props, "")
+ .initTable(storageConf,
tempDir.resolve("legacyMultiFieldSlashTable").toString());
+ HoodieTableConfig tableConfig = tableMetaClient.getTableConfig();
+
+ HoodieException ex = assertThrows(HoodieException.class,
+ () -> HoodieWriterUtils.validateTableConfig(
+ sparkSession,
JavaScalaConverters.convertJavaPropertiesToScalaMap(new TypedProperties()),
tableConfig));
+ assertTrue(ex.getMessage().contains("requires a single partition field"),
ex.getMessage());
+ }
+
+ @Test
+ void validateTableConfigRejectsMultiFieldSlashPartitioningOnOverwrite() {
+ // SaveMode.Overwrite nulls the table config, so the rejection has to fire
off the params
+ // alone, ahead of the isOverWriteMode gate that skips the rest of the
validation
+ TypedProperties writeProps = new TypedProperties();
+ writeProps.put(HoodieTableConfig.SLASH_SEPARATED_DATE_PARTITIONING.key(),
"true");
+ writeProps.put("hoodie.datasource.write.partitionpath.field",
"datestr,city");
+
+ HoodieException ex = assertThrows(HoodieException.class,
+ () -> HoodieWriterUtils.validateTableConfig(
+ sparkSession,
JavaScalaConverters.convertJavaPropertiesToScalaMap(writeProps), null, true));
+ assertTrue(ex.getMessage().contains("requires a single partition field"),
ex.getMessage());
+ }
+
/**
* The meta-fields-mode guard compares normalized modes, not raw legacy
property presence. A table
* written before {@code hoodie.meta.fields.mode} existed is normalized to
ALL or NONE when its
diff --git
a/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/common/TestSlashSeparatedPartitionValue.scala
b/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/common/TestSlashSeparatedPartitionValue.scala
index 34e6dffd1a13..fd0402117db7 100644
---
a/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/common/TestSlashSeparatedPartitionValue.scala
+++
b/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/common/TestSlashSeparatedPartitionValue.scala
@@ -22,34 +22,82 @@ import org.apache.hudi.common.config.HoodieMetadataConfig
import org.apache.hudi.common.table.HoodieTableMetaClient
import org.apache.hudi.hadoop.fs.HadoopFSUtils
import org.apache.hudi.metadata.HoodieBackedTableMetadata
-import org.apache.hudi.storage.{HoodieStorage, StoragePath}
+import org.apache.hudi.storage.StoragePath
import org.junit.jupiter.api.Assertions.assertTrue
class TestSlashSeparatedPartitionValue extends HoodieSparkSqlTestBase {
+ private def createSlashPartitionedTable(targetTable: String,
+ tablePath: String,
+ partitionColumnType: String =
"STRING",
+ tableType: String = "COW",
+ slashSeparatedPartitioning: Boolean
= true): Unit = {
+ spark.sql(
+ s"""
+ |create table $targetTable (
+ | `id` string,
+ | `name` string,
+ | `ts` bigint,
+ | `datestr` $partitionColumnType
+ |) using hudi
+ | tblproperties (
+ | 'primaryKey' = 'id',
+ | 'type' = '$tableType',
+ | 'preCombineField'='ts',
+ |
'hoodie.datasource.write.slash.separated.date.partitioning'='$slashSeparatedPartitioning'
+ | )
+ | partitioned by (`datestr`)
+ | location '$tablePath'
+ """.stripMargin)
+ }
+
+ private def buildMetaClient(tablePath: String): HoodieTableMetaClient = {
+ HoodieTableMetaClient.builder()
+
.setConf(HadoopFSUtils.getStorageConfWithCopy(spark.sparkContext.hadoopConfiguration))
+ .setBasePath(tablePath)
+ .build()
+ }
+
+ private def assertPartitionDirsExist(metaClient: HoodieTableMetaClient,
+ tablePath: String,
+ partitions: String*): Unit = {
+ partitions.foreach { partition =>
+ assertTrue(metaClient.getStorage.exists(new StoragePath(tablePath,
partition)),
+ s"Partition path $partition should exist")
+ }
+ }
+
+ /**
+ * Asserts that the metadata table names the very same directories the
writer created. The
+ * `_hoodie_partition_path` column and an `exists` check on storage both
still pass when the
+ * writer-recorded partition string and the metadata-table entry disagree,
so this is the
+ * assertion that actually pins the two together.
+ */
+ private def assertMetadataTablePartitions(metaClient: HoodieTableMetaClient,
+ tablePath: String,
+ partitions: String*): Unit = {
+ val engine = new HoodieSparkEngineContext(spark.sparkContext)
+ val metadataConfig = HoodieMetadataConfig.newBuilder().build()
+ val metadataTable =
+ new HoodieBackedTableMetadata(engine, metaClient.getStorage,
metadataConfig, tablePath)
+ try {
+ val partitionPaths = metadataTable.getAllPartitionPaths
+ partitions.foreach { partition =>
+ assertTrue(partitionPaths.contains(partition),
+ s"Metadata table should list partition $partition")
+ }
+ } finally {
+ metadataTable.close()
+ }
+ }
+
test("Test slash separated date partitions") {
withTempDir { tmp =>
val targetTable = generateTableName
val tablePath = s"${tmp.getCanonicalPath}/$targetTable"
- spark.sql(
- s"""
- |create table $targetTable (
- | `id` string,
- | `name` string,
- | `ts` bigint,
- | `datestr` STRING
- |) using hudi
- | tblproperties (
- | 'primaryKey' = 'id',
- | 'type' = 'COW',
- | 'preCombineField'='ts',
- |
'hoodie.datasource.write.slash.separated.date.partitioning'='true'
- | )
- | partitioned by (`datestr`)
- | location '$tablePath'
- """.stripMargin)
+ createSlashPartitionedTable(targetTable, tablePath)
spark.sql(
s"""
@@ -58,62 +106,183 @@ class TestSlashSeparatedPartitionValue extends
HoodieSparkSqlTestBase {
| (2, 'a2', 2000, "2026-01-06")
""".stripMargin)
- // check result after insert and merge data into target table
checkAnswer(s"select id, name, ts, _hoodie_partition_path, datestr from
$targetTable limit 10")(
Seq("1", "a1", 1000, "2026/01/05", "2026-01-05"),
Seq("2", "a2", 2000, "2026/01/06", "2026-01-06")
)
- // Verify table config has slash separated date partitioning enabled
- val metaClient = HoodieTableMetaClient.builder()
-
.setConf(HadoopFSUtils.getStorageConfWithCopy(spark.sparkContext.hadoopConfiguration))
- .setBasePath(tablePath)
- .build()
- val tableConfig = metaClient.getTableConfig
- assertTrue(tableConfig.getSlashSeparatedDatePartitioning,
+ val metaClient = buildMetaClient(tablePath)
+ assertTrue(metaClient.getTableConfig.getSlashSeparatedDatePartitioning,
"Table config should have slash separated date partitioning enabled")
- // Verify that partition paths are created with slash separated date
format (yyyy/MM/dd)
- assertTrue(metaClient.getStorage.exists(new StoragePath(tablePath,
"2026/01/05")),
- s"Partition path 2026/01/05 should exist")
- assertTrue(metaClient.getStorage.exists(new StoragePath(tablePath,
"2026/01/06")),
- s"Partition path 2026/01/06 should exist")
+ assertPartitionDirsExist(metaClient, tablePath, "2026/01/05",
"2026/01/06")
+ assertMetadataTablePartitions(metaClient, tablePath, "2026/01/05",
"2026/01/06")
+ }
+ }
- val engine = new HoodieSparkEngineContext(spark.sparkContext)
- val storage = metaClient.getStorage()
- val metadataConfig = HoodieMetadataConfig.newBuilder().build()
- val metadataTable = new HoodieBackedTableMetadata(engine, storage,
metadataConfig, tablePath)
- val partitionPaths = metadataTable.getAllPartitionPaths
- assertTrue(partitionPaths.contains("2026/01/05"))
- assertTrue(partitionPaths.contains("2026/01/06"))
- metadataTable.close()
+ test("Test slash separated date partitions written through the row writer") {
+ withSQLConf("hoodie.spark.sql.insert.into.operation" -> "bulk_insert") {
+ withTempDir { tmp =>
+ val targetTable = generateTableName
+ val tablePath = s"${tmp.getCanonicalPath}/$targetTable"
+
+ createSlashPartitionedTable(targetTable, tablePath)
+
+ // NOTE: The row writer derives the partition path off of an
[[InternalRow]], which used to
+ // blow up with a [[ClassCastException]]; a null partition value
used to NPE
+ spark.sql(
+ s"""
+ | insert into $targetTable values
+ | (1, 'a1', 1000, "2026-01-05"),
+ | (2, 'a2', 2000, "2026-01-06"),
+ | (3, 'a3', 3000, null)
+ """.stripMargin)
+
+ checkAnswer(s"select id, name, ts, _hoodie_partition_path, datestr
from $targetTable order by id")(
+ Seq("1", "a1", 1000, "2026/01/05", "2026-01-05"),
+ Seq("2", "a2", 2000, "2026/01/06", "2026-01-06"),
+ Seq("3", "a3", 3000, "__HIVE_DEFAULT_PARTITION__", null)
+ )
+
+ val metaClient = buildMetaClient(tablePath)
+ assertPartitionDirsExist(metaClient, tablePath,
+ "2026/01/05", "2026/01/06", "__HIVE_DEFAULT_PARTITION__")
+ assertMetadataTablePartitions(metaClient, tablePath,
+ "2026/01/05", "2026/01/06", "__HIVE_DEFAULT_PARTITION__")
+ }
}
}
- test("Test slash separated date partitions with already formatted input") {
- Seq(true, false).foreach { slashSeparatedPartitioning =>
+ test("Test slash separated date partitions on a DATE typed partition
column") {
+ // NOTE: Only bulk_insert exercises the row-writer rendering this fixes --
the insert path on a
+ // DATE column is already covered by [[TestTypedPartitionValues]]
+ withSQLConf("hoodie.spark.sql.insert.into.operation" -> "bulk_insert") {
withTempDir { tmp =>
val targetTable = generateTableName
val tablePath = s"${tmp.getCanonicalPath}/$targetTable"
+ createSlashPartitionedTable(targetTable, tablePath,
partitionColumnType = "DATE")
+
+ // NOTE: A DATE partition value is rendered through
[[BuiltinKeyGenerator#convertToLogicalDataType]]
+ // on the row-writer paths and
[[HoodieAvroUtils#convertValueForAvroLogicalTypes]] on the
+ // Avro one, so all three have to land in the very same directory
spark.sql(
s"""
- |create table $targetTable (
- | `id` string,
- | `name` string,
- | `ts` bigint,
- | `datestr` STRING
- |) using hudi
- | tblproperties (
- | 'primaryKey' = 'id',
- | 'type' = 'COW',
- | 'preCombineField'='ts',
- |
'hoodie.datasource.write.slash.separated.date.partitioning'='$slashSeparatedPartitioning'
- | )
- | partitioned by (`datestr`)
- | location '$tablePath'
+ | insert into $targetTable values
+ | (1, 'a1', 1000, date'2026-01-05'),
+ | (2, 'a2', 2000, date'2026-01-06')
""".stripMargin)
+ checkAnswer(s"select id, name, ts, _hoodie_partition_path, datestr
from $targetTable order by id")(
+ Seq("1", "a1", 1000, "2026/01/05",
java.sql.Date.valueOf("2026-01-05")),
+ Seq("2", "a2", 2000, "2026/01/06",
java.sql.Date.valueOf("2026-01-06"))
+ )
+
+ val metaClient = buildMetaClient(tablePath)
+ assertPartitionDirsExist(metaClient, tablePath, "2026/01/05",
"2026/01/06")
+ assertMetadataTablePartitions(metaClient, tablePath, "2026/01/05",
"2026/01/06")
+ }
+ }
+ }
+
+ test("Test upsert into a slash separated date partition") {
+ Seq("COW", "MOR").foreach { tableType =>
+ withTempDir { tmp =>
+ val targetTable = generateTableName
+ val tablePath = s"${tmp.getCanonicalPath}/$targetTable"
+
+ createSlashPartitionedTable(targetTable, tablePath, tableType =
tableType)
+
+ spark.sql(
+ s"""
+ | insert into $targetTable values
+ | (1, 'a1', 1000, "2026-01-05")
+ """.stripMargin)
+
+ // NOTE: The index and the file-system view have to agree on the slash
partition path for
+ // the second write to be recognised as an update. When they
disagree the record is
+ // inserted a second time instead, which surfaces as a duplicate
rather than an error
+ spark.sql(
+ s"""
+ | merge into $targetTable as t
+ | using (select '1' as id, 'a1_updated' as name, 2000L as ts,
cast("2026-01-05" as string) as datestr) as s
+ | on t.id = s.id
+ | when matched then update set *
+ | when not matched then insert *
+ """.stripMargin)
+
+ checkAnswer(s"select id, name, ts, _hoodie_partition_path, datestr
from $targetTable")(
+ Seq("1", "a1_updated", 2000, "2026/01/05", "2026-01-05")
+ )
+
+ val metaClient = buildMetaClient(tablePath)
+ assertTrue(!metaClient.getStorage.exists(new StoragePath(tablePath,
"2026-01-05")),
+ s"No second directory should be created for table type $tableType")
+ }
+ }
+ }
+
+ test("Test slash separated date partitioning rejects multiple partition
fields at create") {
+ withTempDir { tmp =>
+ val targetTable = generateTableName
+ val tablePath = s"${tmp.getCanonicalPath}/$targetTable"
+
+ // NOTE: A multi-field slash table writes extra path fragments that
+ // [[HoodieSparkUtils#doParsePartitionColumnValues]] cannot line
up with the partition
+ // columns, so every read of the table fails under the default
lazy listing. Rejecting
+ // the combination up front keeps that layout from ever being
written -- HUDI issue #19666
+ checkExceptionContain(
+ s"""
+ |create table $targetTable (
+ | `id` string,
+ | `name` string,
+ | `ts` bigint,
+ | `datestr` STRING,
+ | `city` STRING
+ |) using hudi
+ | tblproperties (
+ | 'primaryKey' = 'id',
+ | 'type' = 'COW',
+ | 'preCombineField'='ts',
+ |
'hoodie.datasource.write.keygenerator.class'='org.apache.hudi.keygen.CustomKeyGenerator',
+ |
'hoodie.datasource.write.partitionpath.field'='datestr:simple,city:simple',
+ |
'hoodie.datasource.write.slash.separated.date.partitioning'='true'
+ | )
+ | partitioned by (`datestr`, `city`)
+ | location '$tablePath'
+ """.stripMargin)("but found 2: datestr,city")
+ }
+ }
+
+ test("Test slash separated date partitioning rejects multiple partition
fields at write") {
+ withTempDir { tmp =>
+ val tablePath = s"${tmp.getCanonicalPath}/${generateTableName}"
+
+ // df.write bypasses the catalog-level check, so the rejection has to
come from
+ // [[HoodieWriterUtils#validateTableConfig]]
+ val df = spark.sql(
+ "select '1' as id, 'a1' as name, 1000L as ts, '2026-01-05' as datestr,
'NYC' as city")
+ checkExceptionContain(() =>
+ df.write.format("hudi")
+ .option("hoodie.table.name", "rejected_slash_table")
+ .option("hoodie.datasource.write.recordkey.field", "id")
+ .option("hoodie.datasource.write.partitionpath.field",
"datestr,city")
+ .option("hoodie.datasource.write.slash.separated.date.partitioning",
"true")
+ .mode("append")
+ .save(tablePath)
+ )("cannot be read back")
+ }
+ }
+
+ test("Test slash separated date partitions with already formatted input") {
+ Seq(true, false).foreach { slashSeparatedPartitioning =>
+ withTempDir { tmp =>
+ val targetTable = generateTableName
+ val tablePath = s"${tmp.getCanonicalPath}/$targetTable"
+
+ createSlashPartitionedTable(targetTable, tablePath,
+ slashSeparatedPartitioning = slashSeparatedPartitioning)
+
spark.sql(
s"""
| insert into $targetTable values
@@ -132,29 +301,12 @@ class TestSlashSeparatedPartitionValue extends
HoodieSparkSqlTestBase {
Seq("2", "a2", 2000, "2026/01/02", secondPartitionValue)
)
- // Verify table config
- val metaClient = HoodieTableMetaClient.builder()
-
.setConf(HadoopFSUtils.getStorageConfWithCopy(spark.sparkContext.hadoopConfiguration))
- .setBasePath(tablePath)
- .build()
- val tableConfig = metaClient.getTableConfig
- assertTrue(tableConfig.getSlashSeparatedDatePartitioning ==
slashSeparatedPartitioning,
+ val metaClient = buildMetaClient(tablePath)
+ assertTrue(metaClient.getTableConfig.getSlashSeparatedDatePartitioning
== slashSeparatedPartitioning,
s"Table config should have slash separated date partitioning set to
$slashSeparatedPartitioning")
- // Verify that partition paths are created with slash separated date
format (yyyy/MM/dd)
- assertTrue(metaClient.getStorage.exists(new StoragePath(tablePath,
"2026/01/01")),
- s"Partition path 2026/01/01 should exist")
- assertTrue(metaClient.getStorage.exists(new StoragePath(tablePath,
"2026/01/02")),
- s"Partition path 2026/01/02 should exist")
-
- val engine = new HoodieSparkEngineContext(spark.sparkContext)
- val storage = metaClient.getStorage()
- val metadataConfig = HoodieMetadataConfig.newBuilder().build()
- val metadataTable = new HoodieBackedTableMetadata(engine, storage,
metadataConfig, tablePath)
- val partitionPaths = metadataTable.getAllPartitionPaths
- assertTrue(partitionPaths.contains("2026/01/01"))
- assertTrue(partitionPaths.contains("2026/01/02"))
- metadataTable.close()
+ assertPartitionDirsExist(metaClient, tablePath, "2026/01/01",
"2026/01/02")
+ assertMetadataTablePartitions(metaClient, tablePath, "2026/01/01",
"2026/01/02")
}
}
}
