Michael Smith created IMPALA-12080:
--------------------------------------
Summary: Test test_recover_many_partitions is very slow on S3,
Ozone
Key: IMPALA-12080
URL: https://issues.apache.org/jira/browse/IMPALA-12080
Project: IMPALA
Issue Type: Task
Components: Infrastructure
Reporter: Michael Smith
The test
metadata/test_recover_partitions.py::TestRecoverPartitions::test_recover_many_partitions
takes <2 minutes for HDFS runs, ~1 hour for Ozone and S3 tests. This appears
to be because test setup invokes a filesystem client 1400 times, to create
directories and files for 700 partitions. With HDFS that's fast because it uses
[pywebhdfs|https://pypi.org/project/pywebhdfs/] to do it in-process, but for
all other filesystems it defaults to the hdfs Java CLI.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
