voonhous commented on issue #14263: URL: https://github.com/apache/hudi/issues/14263#issuecomment-5538342176
### Internal Avro round-trips still standing Under the conversion-boundary rule in the description (memory to disk, disk to memory, engine boundary), a `toAvroSchema()` -> Avro implementation -> `fromAvroSchema()` hop inside a hudi-common helper is not a boundary. Four of those are left. They are correct and covered; what is missing is a native implementation. Line numbers are as of #19810 (`0fb3a5ff6cfc`). | Helper | Location | Round trip today | Blocked on | | --- | --- | --- | --- | | `HoodieSchemaUtils#projectSchema` | `common/schema/HoodieSchemaUtils.java:788` | `HoodieSchema.fromAvroSchema(HoodieAvroUtils.projectSchema(fileSchema.toAvroSchema(), fields))` | a nested-projection benchmark, so a native rewrite (roughly 95 lines) can be shown not to regress the nested path | | `HoodieSchemaUtils#createNewSchemaField`, 5-arg with `HoodieFieldOrder` | `common/schema/HoodieSchemaUtils.java:427` | validated alias of `HoodieSchemaField.of` that routes through `HoodieAvroUtils#createNewSchemaField` | `HoodieSchemaField.of` taking a `HoodieFieldOrder`, then moving the call sites off the alias | | `HoodieAvroUtils#recordNeedsRewriteForExtendedAvroTypePromotion` | `common/avro/HoodieAvroUtils.java:1476` | signature and walk are `(Schema, Schema)`; callers unwrap a `HoodieSchema` to reach it | a public logical-type accessor on `HoodieSchema`, which does not exist yet | | `HoodieSchemaUtils#asNullable` | `common/schema/HoodieSchemaUtils.java:273` | no longer Avro as of #19810, but still `HoodieSchema -> InternalSchema (ColumnUpdateChange) -> HoodieSchema` | a field-wise `HoodieSchema.createNullable` wrap, which is a behavior change: it would keep the non-null defaults, ENUM and union order the `InternalSchema` round trip drops (pinned by `testAsNullablePinsTheInternalSchemaRoundTripLosses`) | The `HoodieSchemaUtils` class javadoc and the 5-arg `createNewSchemaField` javadoc point here for the first two. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
