[
https://issues.apache.org/jira/browse/IMPALA-12080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Smith resolved IMPALA-12080.
------------------------------------
Fix Version/s: Impala 4.3.0
Resolution: Fixed
> Test test_recover_many_partitions is very slow on S3, Ozone
> -----------------------------------------------------------
>
> Key: IMPALA-12080
> URL: https://issues.apache.org/jira/browse/IMPALA-12080
> Project: IMPALA
> Issue Type: Task
> Components: Infrastructure
> Reporter: Michael Smith
> Assignee: Michael Smith
> Priority: Major
> Fix For: Impala 4.3.0
>
>
> The test
> metadata/test_recover_partitions.py::TestRecoverPartitions::test_recover_many_partitions
> takes <2 minutes for HDFS runs, ~1 hour for Ozone and S3 tests. This appears
> to be because test setup invokes a filesystem client 1400 times, to create
> directories and files for 700 partitions. With HDFS that's fast because it
> uses [pywebhdfs|https://pypi.org/project/pywebhdfs/] to do it in-process, but
> for all other filesystems it defaults to the hdfs Java CLI.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
