[
https://issues.apache.org/jira/browse/IMPALA-11659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17630996#comment-17630996
]
zhi tang commented on IMPALA-11659:
-----------------------------------
Thanks Quanlong! This processing method does not seem to solve this case well:
all nodes of Impala cluster are coordinator and executor, although this is not
a best practice.
> Server network util of Statestored is much higher than impala-3.4
> -----------------------------------------------------------------
>
> Key: IMPALA-11659
> URL: https://issues.apache.org/jira/browse/IMPALA-11659
> Project: IMPALA
> Issue Type: Bug
> Components: Perf Investigation
> Affects Versions: Impala 4.1.0
> Reporter: Yuchen Fan
> Priority: Major
> Attachments: image-2022-10-13-19-49-28-456.png,
> image-2022-10-13-19-57-31-248.png, image-2022-10-13-20-08-24-931.png
>
>
> We found that server network traffic of Statestored is rised immediatrly
> after upgraded to Impala-4.1. Using 'iftop' shows Statestored has aboud 6MB/s
> outbound network communication with every Impalad. With more than 250 nodes
> of Impalad, Statestored server has 10~15Gb/s(1~2GB/s) outbound(60x higher
> than it before upgrade) and about 90Mb/s(>10MB/s) inbound(10x higher than it
> before upgrade) network communication, which will occupy about 75% server
> network util. TCP packet snapshot shows detail of packet is 'Pool' and 'Stat'
> information in impala-request-queue topic. We found that value of key 'STAT:'
> contains per host stat of all Impalads. Related function is from
> https://issues.apache.org/jira/browse/IMPALA-8762. So initial update size of
> impala-request-queue is more than 4MB when there are more than 250 Impalads.
> If cluster has more Impalad, Statestored server may hang because of network
> blocking.
> I think coordinators updating topic of per host statistics is just fine. All
> Impalads(including coordinators and executors) registering subscriber to
> topic of impala-request-queue makes Statestored need to broadcast all per
> node statistics to all Impalads.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
