Quanlong Huang created IMPALA-12929:
---------------------------------------
Summary: Skip loading HDFS permissions in local-catalog mode
Key: IMPALA-12929
URL: https://issues.apache.org/jira/browse/IMPALA-12929
Project: IMPALA
Issue Type: Bug
Components: Catalog
Reporter: Quanlong Huang
Assignee: Quanlong Huang
As IMPALA-7539 is unresolved, HDFS file/dir permissions are not used at all in
local catalog mode. However, in catalogd, we still load them (in single thread
for a table!) which could dominant the table loading time when there are lots
of partitions. Here is an example timeline for a REFRESH statement on an
unloaded table:
{noformat}
Catalog Server Operation: 2s300ms
- Got catalog version read lock: 26.407us (26.407us)
- Start loading table: 314.663us (288.256us)
- Got Metastore client: 629.599us (314.936us)
- Fetched table from Metastore: 7.248ms (6.618ms)
- Loaded table schema: 27.947ms (20.699ms)
- Preloaded permissions cache for 1824 partitions: 1s514ms (1s486ms)
- Got access level: 1s514ms (588.314us)
- Created partition builders: 2s103ms (588.270ms)
- Start loading file metadata: 2s103ms (49.760us)
- Loaded file metadata for 1824 partitions: 2s282ms (179.839ms)
- Async loaded table: 2s289ms (6.931ms)
- Loaded table from scratch: 2s289ms (72.038us)
- Got table read lock: 2s289ms (2.289us)
- Finished resetMetadata request: 2s300ms (10.188ms){noformat}
The majority of the time is spent in "Preloaded permissions cache for 1824
partitions".
Currently, catalogd can skip loading HDFS permissions in local catalog mode
until IMPALA-7539 is unresolved.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
