Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/11959 )
Change subject: IMPALA-7804: Mitigate s3 consistency issues for test_scanners ...................................................................... Patch Set 2: Code-Review+1 (2 comments) Had some questions, but lgtm http://gerrit.cloudera.org:8080/#/c/11959/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11959/2//COMMIT_MSG@27 PS2, Line 27: This may improve the : likelihood that the filesystem is consistent by : the time we read it. The docs at https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html says that S3 provides read-after-write consistency for PUTS of new objects, why does it fall to eventual consistency in our case? I suspect it is the listing of directory contents which is eventual consistent only. Is there a way to ensure stronger consistency to make the tests completely stable? Wouldn't S3Guard http://blog.cloudera.com/blog/2017/08/introducing-s3guard-s3-consistency-for-apache-hadoop/ solve this problem? http://gerrit.cloudera.org:8080/#/c/11959/2/tests/common/file_utils.py File tests/common/file_utils.py: http://gerrit.cloudera.org:8080/#/c/11959/2/tests/common/file_utils.py@67 PS2, Line 67: command nit: introducing 'command' variable seems unnecessary, and we don't do that at other places in the file either -- To view, visit http://gerrit.cloudera.org:8080/11959 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e Gerrit-Change-Number: 11959 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]> Gerrit-Comment-Date: Tue, 20 Nov 2018 11:42:46 +0000 Gerrit-HasComments: Yes
