IMPALA-5072: Fix test_recover_partitions on S3 On S3, directory creation is a nop, so we have to actually create values to materialize the partitions.
Was able to merge this against my fork and successfully ran the metadata/test_recover_partitions test on S3. Also ran locally to verify there is no slowdown for HDFS. Change-Id: I30d86e49b000db925a778ff53479b61a075dc88c Reviewed-on: http://gerrit.cloudera.org:8080/6408 Reviewed-by: Michael Brown <[email protected]> Reviewed-by: Dimitris Tsirogiannis <[email protected]> Tested-by: Impala Public Jenkins Project: http://git-wip-us.apache.org/repos/asf/incubator-impala/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-impala/commit/bccbf9eb Tree: http://git-wip-us.apache.org/repos/asf/incubator-impala/tree/bccbf9eb Diff: http://git-wip-us.apache.org/repos/asf/incubator-impala/diff/bccbf9eb Branch: refs/heads/master Commit: bccbf9ebdaed9a03d9a5ecf2a39aad044f178ee1 Parents: 5ecb975 Author: Zach Amsden <[email protected]> Authored: Wed Mar 15 21:36:53 2017 +0000 Committer: Impala Public Jenkins <[email protected]> Committed: Tue Mar 21 02:29:53 2017 +0000 ---------------------------------------------------------------------- tests/metadata/test_recover_partitions.py | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/bccbf9eb/tests/metadata/test_recover_partitions.py ---------------------------------------------------------------------- diff --git a/tests/metadata/test_recover_partitions.py b/tests/metadata/test_recover_partitions.py index 58a7219..2490ce2 100644 --- a/tests/metadata/test_recover_partitions.py +++ b/tests/metadata/test_recover_partitions.py @@ -21,7 +21,7 @@ from tests.common.impala_test_suite import ImpalaTestSuite from tests.common.skip import SkipIfLocal from tests.common.test_dimensions import ALL_NODES_ONLY from tests.common.test_dimensions import create_exec_option_dimension -from tests.util.filesystem_utils import WAREHOUSE +from tests.util.filesystem_utils import WAREHOUSE, IS_S3 from tests.common.test_dimensions import create_uncompressed_text_dimension @@ -183,7 +183,13 @@ class TestRecoverPartitions(ImpalaTestSuite): for i in xrange(1, 700): PART_DIR = "s=part%d/" % i FILE_PATH = "test" + INSERTED_VALUE = "666" self.filesystem_client.make_dir(TBL_LOCATION + PART_DIR) + if IS_S3: + # S3 is a key/value store and directory creation is a NOP; actually + # create the file. + self.filesystem_client.create_file(TBL_LOCATION + PART_DIR + FILE_PATH, + INSERTED_VALUE) result = self.execute_query_expect_success(self.client, "SHOW PARTITIONS %s" % FQ_TBL_NAME)
