Sailesh Mukil has posted comments on this change. Change subject: IMPALA-3577, IMPALA-3486: Partitions on multiple filesystems breaks with S3_SKIP_INSERT_STAGING ......................................................................
Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/3146/2/tests/query_test/test_multiple_filesystems.py File tests/query_test/test_multiple_filesystems.py: Line 14: FILESYSTEM_PREFIX > I don't understand this. If it's an HDFS run, how will FILESYSTEM_PREFIX b Since we cannot run this test with S3 as the default filesystem (because HDFS doesn't start with S3 as the defaultFS), we need to run this test with HDFS as the defaultFS and S3 as the secondary FS. The only way to make S3 as the secondary filesystem for now would be to set $FILESYSTEM_PREFIX to point to an S3 bucket. We have to configure our jobs to do so. Recall that we currently don't use FILESYSTEM_PREFIX at all unless it's a localFS run: https://github.com/cloudera/Impala/blob/cdh5-trunk/bin/impala-config.sh#L177 Line 15: # bucket. Figure out a way to run all possible combinations of supported filesystems. > this comment is confusing because the skip marker also means we skip it in Yes, we do skip it on S3 runs. We run it only on HDFS runs with S3 configured as the secondary FS (by setting FILESYSTEM_PREFIX to point to an S3 bucket). Done. -- To view, visit http://gerrit.cloudera.org:8080/3146 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Ib13b610eb9efb68c83894786cea862d7eae43aa7 Gerrit-PatchSet: 2 Gerrit-Project: Impala Gerrit-Branch: cdh5-2.6.0_5.8.0 Gerrit-Owner: Sailesh Mukil <[email protected]> Gerrit-Reviewer: Dan Hecht <[email protected]> Gerrit-Reviewer: Sailesh Mukil <[email protected]> Gerrit-HasComments: Yes
