voonhous commented on code in PR #19458:
URL: https://github.com/apache/hudi/pull/19458#discussion_r3701852437
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/common/TestNestedSchemaPruningOptimization.scala:
##########
@@ -58,26 +64,34 @@ class TestNestedSchemaPruningOptimization extends
HoodieSparkSqlTestBase {
}
}
- test("Test nested schema pruning with DefaultHoodieRecordPayload") {
+ test("Test nested schema pruning with a projection-incompatible custom
payload") {
withTempDir { tmp =>
val tableName = generateTableName
val tablePath = s"${tmp.getCanonicalPath}/$tableName"
- // NOTE: On the file-group-reader based read path the payload class does
not affect nested
- // schema pruning, so the read schema is pruned the same way as
with the default payload
+ // NOTE: A payload class outside the well-known set puts the table in
CUSTOM merge mode, whose
+ // merger is not projection compatible, so the file group reader
merges on the full
+ // table schema internally
(FileGroupReaderSchemaHandler#generateRequiredSchema) and
+ // projects the merged rows back down to the pruned read schema
afterwards
createTableWithNestedStructSchema("mor", tableName, tablePath,
- Map(HoodieWriteConfig.WRITE_PAYLOAD_CLASS_NAME.key ->
"org.apache.hudi.common.model.DefaultHoodieRecordPayload"))
+ Map(HoodieWriteConfig.WRITE_PAYLOAD_CLASS_NAME.key ->
classOf[CustomPayloadForTesting].getName),
+ populateMetaFields = true)
+
+ // The update writes a log file, so the pruned reads below actually
merge through that gate
+ spark.sql(s"UPDATE $tableName SET ts = 123457 WHERE id = 1")
val selectDF = spark.sql(s"SELECT id, item.name FROM $tableName")
+ // Spark still prunes the scan schema; the full-schema requirement is
internal to the reader
val expectedSchema = StructType(Seq(
StructField("id", IntegerType, nullable = true),
StructField("item", StructType(Seq(StructField("name", StringType,
nullable = false))), nullable = true)
))
-
assertPrunedReadSchema(selectDF, tableName, expectedSchema)
checkAnswer(s"SELECT id, item.name FROM $tableName")(Seq(1, "a1"))
+ // The merged row keeps nested leaves that the pruned read schema dropped
+ checkAnswer(s"SELECT id, item.price, ts FROM $tableName")(Seq(1, 10,
123457))
Review Comment:
Fair on both counts -- dropped the E2E fixture idea.
Added `testMorNestedMandatoryFieldMergesIntoNarrowedRecord`
(`TestFileGroupReaderSchemaHandler`): a projection-compatible CUSTOM merger
declaring `fare.currency` mandatory against a request narrowed to
`fare.amount`, asserting the required schema re-widens to the full `fare
{amount, currency}`. That drives `generateRequiredSchema` through the
`appendFieldsToSchemaDedupNested` / `mergeSchemas` collision path, which the
existing top-level mandatory fields never reach. (The merge mechanics were
already pinned at the utility level in `TestHoodieSchemaUtils`, but nothing
exercised the handler-level composition.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
