satishkotha opened a new pull request #3475:
URL: https://github.com/apache/hudi/pull/3475
## What is the purpose of the pull request
Add range index support with metadata table
## Brief change log
* This is a WIP PR primarily used for data collection.
* Lot of fundamental metadata table bottlenecks have to be resolved. For
example, bootstrapping rangeIndex for 30 partitions of large table (4000
columns) runs into memory issues and driver result too large errors. We also
need to implement async metadata table design in V2
* Tried to implement primitive bucketing by column name to improve index
locality
* Only implemented commit integration. Clean and other action integration
needs to be added.
## Verify this pull request
* Verified end-to-end by bootstrapping range index on few partitions of
large table (4000 columns)
*
* Unit Tests TBD
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under
an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
