Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/11959 )
Change subject: IMPALA-7804: Mitigate s3 consistency issues for test_scanners ...................................................................... Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/11959/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11959/2//COMMIT_MSG@27 PS2, Line 27: This may improve the : likelihood that the filesystem is consistent by : the time we read it. > The docs at https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.ht Based on the s3 tracing, we always do an existence check before we do a write. This puts us in the eventually consistent case. I think one thing we can do if we know the destination won't exist is to get rid of the existence check. That should change us over to read-after-write consistency. S3 tests are definitely going to need more than this change. I'm interested in trying out S3Guard. I haven't tested it, so I'm not sure whether it fixes this issue. S3Guard looks like it helps for listing directories, but we're actually trying to read the file. We'll see if it helps. There might be further changes to read our writes and wait for consistency. http://gerrit.cloudera.org:8080/#/c/11959/2/tests/common/file_utils.py File tests/common/file_utils.py: http://gerrit.cloudera.org:8080/#/c/11959/2/tests/common/file_utils.py@67 PS2, Line 67: command > nit: introducing 'command' variable seems unnecessary, and we don't do that You're right, there's no need. I think this came about from an earlier version where I was trying to work around a long line (which is now no longer long). Fixed. -- To view, visit http://gerrit.cloudera.org:8080/11959 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e Gerrit-Change-Number: 11959 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]> Gerrit-Comment-Date: Tue, 20 Nov 2018 21:01:43 +0000 Gerrit-HasComments: Yes
