yihua opened a new pull request, #18667: URL: https://github.com/apache/hudi/pull/18667
### Describe the issue this Pull Request addresses Builds on top of #18666 (unstructured-types schema as a forward-looking opt-in). That PR added the constants but no test actually carried the new types through the Spark write/read pipeline. This is the first slice of phase 2: wire up a real unstructured-aware generator and flip a small, targeted set of Spark tests so they exercise `variant_data`, `embedding` (vector) and `blob_data` end-to-end. ### Summary and Changelog `HoodieTestDataGenerator`: - Add `generatePayloadForUnstructuredSchema` / `generateRecordForUnstructuredSchema` that build records against `AVRO_SCHEMA_WITH_UNSTRUCTURED`. The legacy field helpers are reused; their `getEffectiveAvroSchema`-derived lookups are tolerant to a structurally compatible outer schema. - Split `generateRandomValueAsPerSchema`: `TRIP_EXAMPLE_SCHEMA_NO_UNSTRUCTURED` → legacy generator, `TRIP_EXAMPLE_SCHEMA` → unstructured generator. Symmetric fork added for the delete branch. - Convenience methods that hard-coded `TRIP_EXAMPLE_SCHEMA` (`generateInserts` / `generateInsertsContainsAllPartitions` / `generateInsertsForPartition` / the `generateUpdates|Deletes` and `getExistingKeys` variants) now hard-code `TRIP_EXAMPLE_SCHEMA_NO_UNSTRUCTURED`, so callers using the no-arg API stay on the legacy path in lock-step with the no-arg constructor. Test flips: - `TestSavepoint` and `TestMultiFS` now write with `TRIP_EXAMPLE_SCHEMA` via `withSchema(...)` and `dataGen.generateInsertsAsPerSchema(..., TRIP_EXAMPLE_SCHEMA)`. Each generated record carries populated variant/vector/blob fields through the COW write/read path. All cases pass on Spark 3.5 locally (8 + 1). Bug surfaced (deferred): - `TestHoodieDefaultMergeHandle`: when records carry unstructured types, `FileGroupReaderBasedMergeHandle.doMerge` → `SerializableIndexedRecord.getData` raises `HoodieIOException: Failed to parse record into provided schema` (caused by Avro `EOFException` inside the binary decoder). The flip is reverted in this PR; this needs a fix before the broader rollout. ### Impact Test-only. Two test classes start exercising the unstructured-types schema; everything else keeps current behavior. ### Risk Level low ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
