Yuchen Fan created IMPALA-11659:
-----------------------------------
Summary: Server network util of Statestored is much higher than
impala-3.4
Key: IMPALA-11659
URL: https://issues.apache.org/jira/browse/IMPALA-11659
Project: IMPALA
Issue Type: Bug
Components: Perf Investigation
Affects Versions: Impala 4.1.0
Reporter: Yuchen Fan
Attachments: image-2022-10-13-19-49-28-456.png,
image-2022-10-13-19-57-31-248.png, image-2022-10-13-20-08-24-931.png
We found that server network traffic of Statestored is rised immediatrly after
upgraded to Impala-4.1. Using 'iftop' shows Statestored has aboud 6MB/s
outbound network communication with every Impalad. With more than 250 nodes of
Impalad, Statestored server has 10~15Gb/s(1~2GB/s) outbound(60x higher than it
before upgrade) and about 90Mb/s(>10MB/s) inbound(10x higher than it before
upgrade) network communication, which will occupy about 75% server network
util. TCP packet snapshot shows detail of packet is 'Pool' and 'Stat'
information in impala-request-queue topic. We found that value of key 'STAT:'
contains per host stat of all Impalads. Related function is from
https://issues.apache.org/jira/browse/IMPALA-8762. So initial update size of
impala-request-queue is more than 4MB when there are more than 250 Impalads. If
cluster has more Impalad, Statestored server may hang because of network
blocking.
I think coordinators updating topic of per host statistics is just fine. All
Impalads(including coordinators and executors) registering subscriber to topic
of impala-request-queue makes Statestored need to broadcast all per node
statistics to all Impalads.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
