manojpec opened a new pull request #4447:
URL: https://github.com/apache/hudi/pull/4447
## What is the purpose of the pull request
- The backing log format for the metadata table is HFile, a KeyValue type.
Since the key field in the metadata record payload is a duplicate of the
Key in the Cell, the redundant key field in the record can be emptied
to save on the cost.
- HoodieHFileWriter and HoodieHFileDataBlock will now serialize records
with the key field emptied by default. HFile writer level relies on the
callers to tell about the key field in the schema.
- HoodieHFileReader when reading the serialized records back from disk,
it materializes the missing keyFields if any. It relies on the callers
to tell about the key field in the record schema.
- WriteHandles and all its derived classes rely on the table properties
for the key field when constructing the file and log readers. This way
base file creation, append and merging all work seamlessly irrespective
of data or metadata table.
## Verify this pull request
- Tests have been added to verify the default virtual keys and key
deduplication support for the metadata table records.
- Manually verified the serialized records on the disk are trimmed
off the key field
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under
an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
