Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15475
Change subject: IMPALA-9492: Fix test_unescaped_string_partition failing on S3 ...................................................................... IMPALA-9492: Fix test_unescaped_string_partition failing on S3 test_unescaped_string_partition in metadata/test_recover_partitions.py use hdfs clients to create four partition directories with special characters, i.e. single quote, double quotes and back slash. It aims to test on whether ALTER TABLE RECOVER PARTITIONS can recognize those directories correctly. However, when running against s3, only two directories are created as expected, which causes the failure. The reason is that when running against s3, we use hadoop cli for operations. A shell command will be launched for each operation. Passing arguments through shell results in duplicate unescaping. So the 4 dirs, [p=', p=", p=\', p=\"] finally became [p=', p=", p=', p="], resulting in two distinct directories. When the test running against hdfs, we use webhdfs_client so don't have this issue. Actually, we shouldn't use special characters in partition paths. Hive converts them to their ascii hex values when creating partition directories. E.g. partition paths of [p=', p=", p=\', p=\"] are [p=%27, p=%22, p=%5C%27, p=%5C%22]. We should follow this rule when creating directories in test. Also we won't have the above shell issue on s3 anymore. Tests: - Added two more special partitions in test_unescaped_string_partition. - Ran test_unescaped_string_partition in S3. Change-Id: I63d149c9bdec52c2e1c0b25c8c3f0448cf7bdadb --- M tests/metadata/test_recover_partitions.py 1 file changed, 23 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/15475/1 -- To view, visit http://gerrit.cloudera.org:8080/15475 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I63d149c9bdec52c2e1c0b25c8c3f0448cf7bdadb Gerrit-Change-Number: 15475 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang <[email protected]>
