Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13020 )
Change subject: IMPALA-8344: Add support for running the minicluster with S3Guard ...................................................................... Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/13020/5/tests/query_test/test_scanners_fuzz.py File tests/query_test/test_scanners_fuzz.py: http://gerrit.cloudera.org:8080/#/c/13020/5/tests/query_test/test_scanners_fuzz.py@179 PS5, Line 179: check_call(['hdfs', 'dfs', '-copyFromLocal', '-d'] + to_copy + [fuzz_table_location]) > I saw a failure here where s3guard showed the temporary file used for copyi Makes sense. I'm wondering if we could add the -d option just for S3 (and do it for all calls to copyFromLocal). For HDFS, I think it makes sense to run without -d, as it makes the upload atomic. For S3, I think 99% of the time, you should run with -d. I don't think there is much benefit to writing data to a temp file on S3, and then re-copying it to the final file, plus I think S3 uploads are already atomic. Not a blocker, so we could do this in a follow up JIRA, just a thought. Yeah, I think your explanation is correct - e.g. you could write a file from one impalad, and S3Guard will store an entry for that file in DynamoDB, then another impalad could try to read that file, and S3Guard will indicate the file exists, but since S3 is eventually consistent the file might not appear yet. -- To view, visit http://gerrit.cloudera.org:8080/13020 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3c748529a494bb6e70fec96dc031523ff79bf61d Gerrit-Change-Number: 13020 Gerrit-PatchSet: 5 Gerrit-Owner: Joe McDonnell <[email protected]> Gerrit-Reviewer: David Knupp <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Laszlo Gaal <[email protected]> Gerrit-Reviewer: Michael Ho <[email protected]> Gerrit-Reviewer: Philip Zeyliger <[email protected]> Gerrit-Reviewer: Sahil Takiar <[email protected]> Gerrit-Comment-Date: Tue, 21 May 2019 21:27:31 +0000 Gerrit-HasComments: Yes
