chungen0126 commented on PR #10202: URL: https://github.com/apache/ozone/pull/10202#issuecomment-4413001123
Thanks @peterxcli for working on this. However, I have some concerns regarding the feasibility of placing this atomic validation within the pre-commit hook. Here are my thoughts: I believe the original issue stems from scenarios where a timeout occurs due to network instability or other unknown factors(on client side or datanode side), leading to a wrong commit. Consider this flow: If an exception is thrown during the IO copy process because of timeout, it will jump to the catch block. At this point, the content length validation has not yet been added into the pre-commit hook, meaning the incorrect commit would still proceed. ou can try testing one of the specific scenarios provided by @xichen01 for this [issue](https://issues.apache.org/jira/browse/HDDS-9526). > Another similar problem: > Ozone can generate an incomplete key. > > Reproduce Step: > Ozone S3 > Upload the key and interrupt (ctrl + c) the upload before it completes. > > An incomplete key will commit > > [root@VM-8-3-centos ~]$ aws configure set default.s3.multipart_threshold 1GB > [root@VM-8-3-centos ~]$ aws s3 --endpoint-url http://localhost:9878 cp ~/500M.img s3://bucket1/500M.img > > ^Ccancelled: ctrl-c received > [root@VM-8-3-centos /tmp]$ aws s3 ls s3://bucket1/ --endpoint-url http://localhost:9878 > 2023-10-24 15:48:13 121208832 500M.img > AWS S3 > Upload the key and interrupt (ctrl + c) the upload before it completes. > > the incomplete key will not be committed > > [root@VM-8-3-centos ~]$ aws configure set default.s3.multipart_threshold 1GB > [root@VM-8-3-centos ~]$ aws s3 cp ~/500M.img s3://bucket1/500M.img > ^Ccancelled: ctrl-c received > [root@VM-8-3-centos ~]$ aws s3 ls s3://bucket1/500M.img > // nothing output -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
