voonhous commented on PR #12795: URL: https://github.com/apache/hudi/pull/12795#issuecomment-6553795326
@cshuo @danny0405 picking this up again now that `HoodieSchema` has landed and Trino is coming in. RFC-99 is closing with a reduced scope (#20488). What it leaves open overlaps a lot with this RFC: the remaining logical types, the conversion code between `HoodieSchema` and each engine / format / catalog, and removing Avro from `HoodieSchema`. As discussed, we'll co-author a restructure of this RFC to hold them instead of opening another schema RFC. Proposed shape: - **Schema**: build on `HoodieSchema` instead of a new `Schema` class. Add one mapper per boundary: `TypeSystemMapper` for engines and catalogs, `FormatSchemaMapper` for Parquet / Lance / ORC / Avro. Each mapper declares per type whether it maps natively, lossily or not at all, with shared tests against RFC-99's mapping table. This is the per-type cross-engine contract from my earlier comment here. - **Reader / writer**: formats read and write a single Hudi row type (`HoodieRow`, typed getters), and each engine wraps its own rows in both directions, the same model Paimon uses (`InternalRow` with `FlinkRowWrapper` / `FlinkRowData`). `HoodieRow` is essentially the `StructLike` from the expression section, grown into the read / write row. That keeps it to one class per engine plus one per format; the pseudo-code's per-engine `XFormatStructWriter` / `XFormatStructReader` subclasses would multiply once applied to every format. The `StructWriter` / `ValueWriter` split stays inside each format. The physical schema comes from `FormatSchemaMapper`; engine vectorized readers stay as optional fast paths. - **HoodieRecord / KeyGenerator**: unchanged. - **Expressions**: translate engine filters into Hudi `Expression`, render out to file readers and catalogs, one planner for metadata-table indexes. Shared vs engine-native stats evaluation becomes a pluggable implementation, which keeps Vinoth's earlier point open rather than deciding it. Field ids and schema evolution stay in RFC-104. Rollout would start with the mappers, which need no new row type, then move one format at a time onto `HoodieRow`, starting with Avro log blocks (removes the per-record `GenericRecord` conversion without changing bytes on disk). I'll put the changes up as a PR against this RFC. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
