cshuo opened a new pull request, #12795:
URL: https://github.com/apache/hudi/pull/12795
### Change Logs
Hudi currently is tightly coupled with Avro, particularly in terms of basic
data types, schema, and the internal record representation used in read/write
paths. This coupling leads to numerous issues, for example, record-level
unnecessary Ser/De costs are introduced because of engine specific row and Avro
record converting, the data type can not be extended to support other
complex/advanced type, such as Variant and the basic read/ write functionality
codes cannot be effectively reused among different engines. As for Expression,
currently, different engines have their own implementation to achieve pushdown
optimization, which is not friendly for extending as more indices are
introduced.
This RFC aims to propose an improvement to the current
Schema/Type/Expression abstractions, to achieve the following goals:
Use a native schema as the authoritative schema, and make the type system
extensible to support or customize other types, e.g, Variant.
Abstract the common implementation of writer/readers and move them to
hudi-common module, and engines just need implement getter/setters for specific
rows(Flink RowData and Spark InternalRow).
Add a concentrated and sharable expression abstraction for all kinds of
expression pushdown for all engines and integrate it deeply with the MDT
indices.
### Impact
* Introduce some public API, such as new Schema abstraction.
* Users should see enhanced performance for reading/writing, since redundant
Avro Ser/De is eliminated.
### Risk level (write none, low medium or high below)
low
### Documentation Update
_Describe any necessary documentation update if there is any new feature,
config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the
default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website.
Please create a Jira ticket, attach the
ticket number here and follow the
[instruction](https://hudi.apache.org/contribute/developer-setup#website) to
make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
