Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/11959
to look at the new patch set (#2).
Change subject: IMPALA-7804: Mitigate s3 consistency issues for test_scanners
......................................................................
IMPALA-7804: Mitigate s3 consistency issues for test_scanners
test_scanners.py has seen several flaky failures on
s3 due to eventual consistency. The symptom is Impala
being unable to read a file that it just loaded to s3.
A large number of tables used in test_scanners.py
use the file_utils helper functions for creating
the tables. These follow the pattern:
1. Copy files to temporary directory in HDFS/S3/etc
2. Create table
3. Run LOAD DATA to move the files to the table
In step #3, LOAD DATA gets the metadata for the
table before it runs the move statement on the
files. Subsequent queries on the table will not
need to reload metadata and can access the file
quickly after the move.
This changes the ordering to put the files in place
before loading metadata. This may improve the
likelihood that the filesystem is consistent by
the time we read it. Specifically, we now do:
1. Put the files in directory that the table
will use when it is created.
2. Create table
Neither of these steps load metadata, so the next
query that runs will load metadata.
Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e
---
M tests/common/file_utils.py
1 file changed, 25 insertions(+), 13 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/11959/2
--
To view, visit http://gerrit.cloudera.org:8080/11959
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e
Gerrit-Change-Number: 11959
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <joemcdonn...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
