voonhous commented on code in PR #19458:
URL: https://github.com/apache/hudi/pull/19458#discussion_r3701558392
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/SchemaHandlerTestBase.java:
##########
@@ -130,6 +130,27 @@ public void testMor(RecordMergeMode mergeMode,
}
assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
assertFalse(readerContext.getNeedsBootstrapMerge());
+
+ //read subset of columns with a nested-narrowed field, the shape Spark's
nested schema pruning
+ //requests: "fare" keeps only its "amount" leaf
+ requestedSchema =
narrowFareToAmountOnly(generateProjectionSchema("begin_lat", "fare", "rider"));
+ schemaHandler = createSchemaHandler(readerContext, dataSchema,
requestedSchema, supportsParquetRowIndex);
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema =
narrowFareToAmountOnly(generateProjectionSchema(hasBuiltInDelete, "begin_lat",
"fare", "rider", "_hoodie_record_key", "timestamp"));
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode ==
COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema =
narrowFareToAmountOnly(generateProjectionSchema(hasBuiltInDelete, "begin_lat",
"fare", "rider", "_hoodie_record_key"));
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema =
narrowFareToAmountOnly(generateProjectionSchema("begin_lat", "fare", "rider",
"begin_lon", "_hoodie_record_key", "timestamp"));
+ } else {
+ //a projection-incompatible custom merger may need any column to merge,
so the handler must
+ //re-expand the nested-narrowed request back to the full data schema
(see HUDI-5443)
+ expectedRequiredSchema = dataSchema;
Review Comment:
Agreed -- confirmed the branch does `return this.tableSchema` before ever
reading `requestedSchema`, so this arm asserted exactly what the
flat-projection block above already pins. Dropped it in the latest commit: the
nested-narrowed block now runs only for the arms where the narrowing has to
survive `generateRequiredSchema`.
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/common/TestNestedSchemaPruningOptimization.scala:
##########
@@ -58,26 +64,34 @@ class TestNestedSchemaPruningOptimization extends
HoodieSparkSqlTestBase {
}
}
- test("Test nested schema pruning with DefaultHoodieRecordPayload") {
+ test("Test nested schema pruning with a projection-incompatible custom
payload") {
Review Comment:
Done. The description now splits **Production changes** / **Test changes**,
covers the `HoodieTableState.recordPayloadClassName` removal and both test
edits, and Risk Level covers the test tightening. The stale "kept unchanged"
claim is gone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
