Quanlong Huang created IMPALA-12301:
---------------------------------------
Summary: Inconsistent disk id mapping between FE and BE
Key: IMPALA-12301
URL: https://issues.apache.org/jira/browse/IMPALA-12301
Project: IMPALA
Issue Type: Bug
Components: Frontend
Reporter: Quanlong Huang
Attachments: DiskIoMgr-metrics.jpg
In BE, we map the device to disk ids based on their order in /proc/partitions:
https://github.com/apache/impala/blob/be11d3eb44bbca0dcbde2fa75cc32dffb7cf015c/be/src/util/disk-info.cc#L79-L82
However, in FE, we map the storageUuid of HDFS to disk ids based on the order
they are used:
https://github.com/apache/impala/blob/be11d3eb44bbca0dcbde2fa75cc32dffb7cf015c/fe/src/main/java/org/apache/impala/catalog/DiskIdMapper.java#L55-L56
It's indeed a random order which cannot guarentee to match the disk ids of BE.
The disk ids are used to identify the disk queues in BE. Each queue corresponds
to a disk or a remote storage. Their thread pool have different parallism based
on the disk types, e.g. 1 thread for each rotational disk, 8 threads for each
SSD. Disk ids mismatch might degrade performance if requests on a SSD are
assigned to the thread pool of a rotational disk.
The issue is reported by a community user who saw weird metrics on disk-io-mgr
(see [^DiskIoMgr-metrics.jpg]). In the screenshot, nvme0n1 has no reads but it
stores hdfs data, while sda doesn't store any hdfs data but have a large read
count.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
