IMPALA-3957: Test failure in S3 build: TestLoadData.test_load The test_load() test failed with an error which said that the number of of files in the destination was wrong. This could probably be because the filesystem_client.copy() (for S3) in the setup_method() silently failed without copying one of the files as a one off error.
I'm not sure why S3 failed to do the copy, but this patch adds an assert after the copy to make sure that if it the s3_client.copy() fails to do the copy, it will assert instead of continuing with the rest of the tests. Change-Id: I966a469e94099d3d971e470ae6e992386070c5e9 Reviewed-on: http://gerrit.cloudera.org:8080/3881 Reviewed-by: Sailesh Mukil <[email protected]> Tested-by: Internal Jenkins Project: http://git-wip-us.apache.org/repos/asf/incubator-impala/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-impala/commit/ac4f22b1 Tree: http://git-wip-us.apache.org/repos/asf/incubator-impala/tree/ac4f22b1 Diff: http://git-wip-us.apache.org/repos/asf/incubator-impala/diff/ac4f22b1 Branch: refs/heads/master Commit: ac4f22b1b0a68600884358d373fffbcd817f22b8 Parents: e6702c6 Author: Sailesh Mukil <[email protected]> Authored: Tue Aug 9 22:10:54 2016 -0700 Committer: Internal Jenkins <[email protected]> Committed: Thu Aug 11 03:06:09 2016 +0000 ---------------------------------------------------------------------- tests/util/s3_util.py | 2 ++ 1 file changed, 2 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/ac4f22b1/tests/util/s3_util.py ---------------------------------------------------------------------- diff --git a/tests/util/s3_util.py b/tests/util/s3_util.py index 549a5c9..ec5cfdf 100644 --- a/tests/util/s3_util.py +++ b/tests/util/s3_util.py @@ -45,6 +45,8 @@ class S3Client(BaseFilesystem): def copy(self, src, dst): self.s3client.copy_object(Bucket=self.bucketname, CopySource={'Bucket':self.bucketname, 'Key':src}, Key=dst) + assert self.exists(dst), \ + 'S3 copy failed: Destination file {dst} does not exist'.format(dst=dst) # Since S3 is a key-value store, it does not have a command like 'ls' for a directory # structured filesystem. It lists everything under a path recursively.
