sivabalan narayanan created HUDI-2559:
-----------------------------------------
Summary: Ensure unique timestamps are generated for commit times
with concurrent writers
Key: HUDI-2559
URL: https://issues.apache.org/jira/browse/HUDI-2559
Project: Apache Hudi
Issue Type: Improvement
Reporter: sivabalan narayanan
Ensure unique timestamps are generated for commit times with concurrent writers.
this is the piece of code in HoodieActiveTimeline which creates a new commit
time.
{code:java}
public static String createNewInstantTime(long milliseconds) {
return lastInstantTime.updateAndGet((oldVal) -> {
String newCommitTime;
do {
newCommitTime = HoodieActiveTimeline.COMMIT_FORMATTER.format(new
Date(System.currentTimeMillis() + milliseconds));
} while (HoodieTimeline.compareTimestamps(newCommitTime,
LESSER_THAN_OR_EQUALS, oldVal));
return newCommitTime;
});
}
{code}
There are chances that a deltastreamer and a concurrent spark ds writer gets
same timestamp and one of them fails.
Related issues and github jiras:
[https://github.com/apache/hudi/issues/3782]
https://issues.apache.org/jira/browse/HUDI-2549
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
