Davis-Zhang-Onehouse opened a new pull request, #20701:
URL: https://github.com/apache/hudi/pull/20701

   ### Describe the issue this Pull Request addresses
   
   `HoodieSchemaUtils.generateProjectionSchema(HoodieSchema, List<String>)` 
builds the projected record with a bare `HoodieSchema.createRecord(name, 
namespace, doc, fields)`, so any schema-level (record-level) properties on the 
source schema are lost in the projection.
   
   The other helpers in `HoodieSchemaUtils` that rebuild a record schema keep 
those properties: `removeFields` copies `getObjectProps()`, `pruneDataSchema` 
calls `copyProperties`, and `addMetadataFields`, `appendFieldsToSchema` and 
`mergeSchemas` go through `createNewSchemaFromFieldsWithReference`. 
`generateProjectionSchema` is the one helper that does not. Its callers include 
the bootstrap record-key schema, the Hive file group reader and Avro Parquet 
reader schemas, and the MERGE INTO partial-update schema.
   
   ### Summary and Changelog
   
   A projection now keeps the source schema's record-level properties, as the 
other record-rebuilding helpers in `HoodieSchemaUtils` already do.
   
   - `HoodieSchemaUtils.generateProjectionSchema` builds its result with 
`createNewSchemaFromFieldsWithReference(originalSchema, projectedFields)`, 
which keeps the name, namespace, doc and schema-level properties of the source 
schema. Field matching and field copying are unchanged.
   - Adds 
`TestHoodieSchemaUtils#testGenerateProjectionSchemaPreservesSchemaLevelProps`. 
The test gives the source schema a namespace, a doc and a record-level 
property. It then asserts that the projection keeps all three, and that the 
property also survives a `toString`/`parse` round trip. Before this change it 
fails with `expected: <value> but was: <null>`, and after it the test passes.
   - Not changed here: the Hive realtime reader has its own 
`HoodieRealtimeRecordReaderUtils.generateProjectionSchema`, and 
`HoodieSchemaUtils.projectSchema` (nested-path projection) also rebuilds 
records without properties. Both can follow separately.
   
   ### Impact
   
   No public API change. A projected schema now carries the record-level 
properties of the schema it was projected from. This change leaves field 
handling as it was. For MERGE INTO partial updates on MOR tables, the projected 
schema is serialized as the partial-update schema written with the update. Any 
record-level properties of the table schema are therefore now kept there as 
well.
   
   ### Risk Level
   
   low. The change affects only record-level properties of the projected 
schema, and only for source schemas that carry them. The `hudi-common` 
schema-handler tests and the `hudi-hadoop-mr` reader tests that build 
projection schemas still pass.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

  • [PR] fix(common): keep schem... Davis-Zhang-Onehouse (via GitHub) via commits

Reply via email to