Joy-2000 commented on issue #19587: URL: https://github.com/apache/hudi/issues/19587#issuecomment-5378742845
> Any lock-free allocation/retry should check/reserve the request time across the relevant comparable-action family, not just retry when the exact target path already exists. This is especially important because createNewInstantTime(false) only has the process-local monotonic guard while skipping the distributed time-generator lock. @danny0405 Sorry for the late reply, and thanks for the detailed discussion here. Re: 2. Request-time generation in startCommit I agree with your point that request-time generation shouldn't simply be moved out of the lock. One additional reason from our side: duplicate request instants can cause silent MDT data loss. If two concurrent writers generate the same request time (e.g. a deltacommit and a clean landing on the same instant), they map to the same instant on the Metadata Table, and when the later commit is applied the MDT reconciliation path treats the pre-existing instant as a leftover and rolls it back — so the data the first instant wrote to the MDT gets dropped. This means "atomic file creation + loser retries" isn't sufficient on its own: even if only one writer wins on the data timeline, the collision has already corrupted MDT state, so any collision-detection/retry mechanism would have to cover the MDT side, not just the data timeline. We encounter another problem in our test. In the Flink async-instant flow, the write task sends an RPC to the coordinator to obtain the request time for the next instant. Under lock contention this RPC can exceed its timeout (default 60s), which fails the task and triggers a full job restart. For a streaming job this is a significant stability hit — contention on the instant-request path doesn't just cost throughput, it can take the job down. So we'd still like to find a way to reduce this contention, just without weakening the guarantees. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
