nsivabalan opened a new pull request #4485:
URL: https://github.com/apache/hudi/pull/4485
## What is the purpose of the pull request
Some operations in hudi may not add CHECKPOINT_KEY and CHECKPOINT_RESET_KEY
like incline clustering, cleaning etc. So, deltastreamer should ignore those
and go back to find the valid commit metadata from which hudi can fetch the
checkpoint info.
Actual issue reported: Users starts deltastreamer in continuous mode w/
inline clustering and provides a checkpoint to start with. After few commits
shutsdown the job. And then restarts. Lets say incidentally last commit was
replace commit (clustering). So, when deltastreamer starts, it checks for
CHECKPOINT_KEY and last commit metadata which will not be present in replace
commit and hence starts resuming from user provided checkpoint which should not
happen.
With the fix in this patch, we go back to find the right commit metadata
which has the CHECKPOINT_KEY or CHECKPOINT_RESET key and uses that to determine
whether to go with checkpoint from this commit or to go with user provided one.
## Brief change log
- Fixed the way Deltastreamer fetches checkpoint info from commits. Goes
back to find the right commit metadata.
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test
coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please
describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under
an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
