yihua opened a new pull request #3991:
URL: https://github.com/apache/hudi/pull/3991
## What is the purpose of the pull request
This PR adds an option to run async compaction and clustering jobs without
providing the instant time, using the earliest instant of the corresponding
requested action by default, to improve usability.
## Brief change log
- Adds an API in `HoodieTimeline` to find the first instant of a specific
action and state.
- For `HoodieClusteringJob`, when `execute` mode is used, if the instant
time is not provided from `--instant-time`, the job automatically finds the
earliest scheduled clustering instant for execution.
- For `HoodieCompactor`, when executing compaction, if the instant time is
not provided from `--instant-time`, the job automatically finds the earliest
scheduled compaction instant for execution.
## Verify this pull request
This change added tests in
`TestHoodieDeltaStreamer#testHoodieAsyncClusteringJob`.
The change has also been verified by running `HoodieCompactor` and
`HoodieClusteringJob` with kafka-connect:
- Compaction scheduled, run `HoodieCompactor`, earliest requested compaction
instant is executed.
- Compaction done, run `HoodieCompactor`, throws an exception.
- Clustering scheduled, run `HoodieClusteringJob`, earliest requested
clustering instant is executed.
- Clustering done, run `HoodieClusteringJob`, throws an exception.
Async compaction command:
```
spark-submit \
--class org.apache.hudi.utilities.HoodieCompactor \
hudi/packaging/hudi-utilities-bundle/target/hudi-utilities-bundle_2.11-0.10.0-SNAPSHOT.jar
\
--base-path /tmp/hoodie/hudi-test-topic \
--table-name hudi-test-topic \
--schema-file /repo/hudi/docker/demo/config/schema.avsc \
--parallelism 2 \
--spark-memory 1g
```
Async clustering command:
```
spark-submit \
--class org.apache.hudi.utilities.HoodieClusteringJob \
hudi/packaging/hudi-utilities-bundle/target/hudi-utilities-bundle_2.11-0.10.0-SNAPSHOT.jar
\
--props clusteringjob.properties \
--mode execute \
--base-path /tmp/hoodie/hudi-test-topic \
--table-name sample_table \
--spark-memory 1g
```
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under
an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
