[
https://issues.apache.org/jira/browse/IMPALA-11659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17634707#comment-17634707
]
ASF subversion and git services commented on IMPALA-11659:
----------------------------------------------------------
Commit 5f7210a8df60a7d1a58742b00b3a37c7a27a91de in impala's branch
refs/heads/master from ttttttz
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=5f7210a8d ]
IMPALA-11659: Avoid statestored sending the host stats to executors
This patch is used to solve the problem that the outbound traffic of
the statestore is too large due to the statestore broadcast hosts stats
to all impalads in a large cluster.
Change-Id: I293c63e71c587b0b217f7b13c1931034ce390f62
Reviewed-on: http://gerrit.cloudera.org:8080/19227
Reviewed-by: Wenzhe Zhou <[email protected]>
Reviewed-by: Quanlong Huang <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Server network util of Statestored is much higher than impala-3.4
> -----------------------------------------------------------------
>
> Key: IMPALA-11659
> URL: https://issues.apache.org/jira/browse/IMPALA-11659
> Project: IMPALA
> Issue Type: Bug
> Components: Perf Investigation
> Affects Versions: Impala 4.1.0
> Reporter: Yuchen Fan
> Assignee: zhi tang
> Priority: Major
> Attachments: image-2022-10-13-19-49-28-456.png,
> image-2022-10-13-19-57-31-248.png, image-2022-10-13-20-08-24-931.png
>
>
> We found that server network traffic of Statestored is rised immediatrly
> after upgraded to Impala-4.1. Using 'iftop' shows Statestored has aboud 6MB/s
> outbound network communication with every Impalad. With more than 250 nodes
> of Impalad, Statestored server has 10~15Gb/s(1~2GB/s) outbound(60x higher
> than it before upgrade) and about 90Mb/s(>10MB/s) inbound(10x higher than it
> before upgrade) network communication, which will occupy about 75% server
> network util. TCP packet snapshot shows detail of packet is 'Pool' and 'Stat'
> information in impala-request-queue topic. We found that value of key 'STAT:'
> contains per host stat of all Impalads. Related function is from
> https://issues.apache.org/jira/browse/IMPALA-8762. So initial update size of
> impala-request-queue is more than 4MB when there are more than 250 Impalads.
> If cluster has more Impalad, Statestored server may hang because of network
> blocking.
> I think coordinators updating topic of per host statistics is just fine. All
> Impalads(including coordinators and executors) registering subscriber to
> topic of impala-request-queue makes Statestored need to broadcast all per
> node statistics to all Impalads.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
