prashantwason opened a new pull request #2819:
URL: https://github.com/apache/hudi/pull/2819
Added unit test to ensure new instant time can be generated in multiple
threads correctly.
## What is the purpose of the pull request
When generating a new instant time in HoodieActiveTimeline, a static
instance of SimpleDateFormat is used. This class is not thread safe.
We have a production usecase where multiple HUDI datasets are processed in
parallel in different threads of a ThreadPool. Each of these threads creates
its own SparkRDDBackedWriteClient and calls startCommit() which generates a new
commit time. Because SimpleDateFormat is not thread safe, we get corrupted
instant times in several threads.
The solution is to use a thread-specific instance of the SimpleDateFormat
for generating new instant times.
## Brief change log
Moved static COMMIT_FORMATTER to thread local variable
## Verify this pull request
This change added tests and can be verified as follows:
- *Added test TestHoodieActiveTimeline#testCreateNewInstantTime*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under
an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
