cshuo commented on PR #12857: URL: https://github.com/apache/hudi/pull/12857#issuecomment-2699773384
I've made some research on s3 and filesystem based lock proivder. I think currently it's not feasible to implement fs lock provider on S3 based on the following infos: * the basic mechanism of fs lock provider: file existence checking and mutually exclusive file creating. * S3 does not support object locking for concurrent writers. If two PUT requests are simultaneously made to the same key, the request with the latest timestamp wins, i.e., last-writer-wins semantic. * S3A fs doesn't support atomic create-no-overwrite operation, which means two processes can send `fs#create(path, overwrite=false)` requests concurrently without any exception. reference: * https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md#how-s3a-writes-data-to-s3 * https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html#ConsistencyModel cc @danny0405 @CTTY, please correct me if I missed something. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
