voonhous commented on code in PR #19582:
URL: https://github.com/apache/hudi/pull/19582#discussion_r3765893605
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/schema/TestVariantDataType.scala:
##########
@@ -448,6 +450,77 @@ class TestVariantDataType extends HoodieSparkSqlTestBase {
})
}
+ test("Test COW small-file merge preserves shredded VARIANT values") {
Review Comment:
Added, split the same way as the clustering pair: `Test COW small-file merge
preserves unshredded VARIANT values` - shredding disabled, no force-shredding
config, same bin-pack and same one-file-group pin, and the pre-merge files
asserted to NOT carry `typed_value` so the twin cannot silently become a copy
of the shredded one.
It is a no-op guard rather than a red-before-fix test, which is the point
you raised: an unshredded column is a plain two-field `{metadata, value}`
record, nothing matches the shredded shape, so `alignShreddedVariants` must
return the file schema untouched and the round trip must look exactly as it did
before. Also pinned that directly at unit level in
`testIsStrictProjectionWithAlignedShreddedVariants` - a footer-derived plain
`{metadata, value}` column against a variant requested column comes back as the
same schema instance (`assertSame`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
