[
https://issues.apache.org/jira/browse/IMPALA-12080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17715324#comment-17715324
]
ASF subversion and git services commented on IMPALA-12080:
----------------------------------------------------------
Commit a08a6b84b0ad758efac14bb1768118fc77921f89 in impala's branch
refs/heads/master from Michael Smith
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=a08a6b84b ]
IMPALA-12080: Create test partitions locally
Creates test partitions locally for test_recover_many_partitions to
avoid invoking the Java hdfs CLI repeatedly for non-HDFS filesystems.
This drastically reduces runtime for S3 and Ozone test runs.
Testing:
- tested with Ozone and S3
Change-Id: Iab8c2f49c1940a3a63a279a911e0897b5ed8808e
Reviewed-on: http://gerrit.cloudera.org:8080/19782
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Test test_recover_many_partitions is very slow on S3, Ozone
> -----------------------------------------------------------
>
> Key: IMPALA-12080
> URL: https://issues.apache.org/jira/browse/IMPALA-12080
> Project: IMPALA
> Issue Type: Task
> Components: Infrastructure
> Reporter: Michael Smith
> Assignee: Michael Smith
> Priority: Major
>
> The test
> metadata/test_recover_partitions.py::TestRecoverPartitions::test_recover_many_partitions
> takes <2 minutes for HDFS runs, ~1 hour for Ozone and S3 tests. This appears
> to be because test setup invokes a filesystem client 1400 times, to create
> directories and files for 700 partitions. With HDFS that's fast because it
> uses [pywebhdfs|https://pypi.org/project/pywebhdfs/] to do it in-process, but
> for all other filesystems it defaults to the hdfs Java CLI.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
