satishkotha opened a new pull request #2263:
URL: https://github.com/apache/hudi/pull/2263
## What is the purpose of the pull request
Implement simple clustering strategies to create and run ClusteringPlan
## Brief change log
- Add simple strategy to to schedule clustering and create ClusteringPlan.
Initial strategy can limit partitions and bound the amount of data considered
for clustering
- Add simple strategy to run the clustering plan and create 'replacecommit'.
Initial strategy is implemented by using BulkInsert partitioner.
- This can work for both COW and MOR tables. Clustering also does
'compaction' if there are log files. We ensure compaction and clustering do not
work on same file groups.
- Added basic support for sorting by custom column(s). I may refactor this
little bit.
- Few other things to pay attention:
1. After clustering all new records have a new commit time. I'm trying to
see if it's possible to preserve commit_time from original file to support
incremental queries.
2. Schema is not serializable. So I pass around schema string in some cases.
Let me know if there is a better way to restructure code.
3. Initial strategy is implemented based on requirements for Uber. But the
interfaces are flexible to allow other implementation. But please double check
if I made bad assumptions in any of the strategies.
## Verify this pull request
- Manually tested 'inline clustering' end-to-end in local docker setup.
- I will be working on adding lot more unit tests (Publishing this PR to get
high level feedback).
- Plan to add integ-tests using test-suite as part of separate PR.
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under
an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
