Sailesh Mukil has uploaded a new patch set (#5). Change subject: IMPALA-3577, IMPALA-3486: Partitions on multiple filesystems breaks with S3_SKIP_INSERT_STAGING ......................................................................
IMPALA-3577, IMPALA-3486: Partitions on multiple filesystems breaks with S3_SKIP_INSERT_STAGING The HdfsTableSink usualy creates a HDFS connection to the filesystem that the base table resides in. However, if we create a partition in a FS different than that of the base table and set S3_SKIP_INSERT_STAGING to "true", the table sink will try to write to a different filesystem with the wrong filesystem connector. This patch allows the table sink itself to work with different filesystems by getting rid of a single FS connector and getting a connector per partition. This also reenables the multiple_filesystems test and modifies it to use the unique_database fixture so that parallel runs on the same bucket do not clash and end up in failures. All jobs with HDFS as the default filesystem need to set the appropriate environment for S3, i.e. the following: - export AWS_SECERT_ACCESS_KEY - export AWS_ACCESS_KEY_ID - export S3_BUCKET The FILESYSTEM_PREFIX will be set by the test to allow S3 to be used as the secondary FS. Change-Id: Ib13b610eb9efb68c83894786cea862d7eae43aa7 --- M be/src/exec/hdfs-table-sink.cc M be/src/exec/hdfs-table-sink.h M testdata/workloads/functional-query/queries/QueryTest/multiple-filesystems.test M tests/common/impala_test_suite.py M tests/common/skip.py M tests/query_test/test_multiple_filesystems.py M tests/util/filesystem_utils.py 7 files changed, 51 insertions(+), 35 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/46/3146/5 -- To view, visit http://gerrit.cloudera.org:8080/3146 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib13b610eb9efb68c83894786cea862d7eae43aa7 Gerrit-PatchSet: 5 Gerrit-Project: Impala Gerrit-Branch: cdh5-2.6.0_5.8.0 Gerrit-Owner: Sailesh Mukil <[email protected]> Gerrit-Reviewer: Dan Hecht <[email protected]> Gerrit-Reviewer: Sailesh Mukil <[email protected]>
