[
https://issues.apache.org/jira/browse/IMPALA-11659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17630801#comment-17630801
]
Quanlong Huang commented on IMPALA-11659:
-----------------------------------------
Maybe we can simply fix this by not sending updates to executors. They just
send stats to statestore, and statestore sends updates to just the coordinators.
This is similar to the pattern of the catalog topic, i.e. catalogd ->
statestore -> coordinators. Statestore just propagates the catalog updates sent
from catalogd to all coordinators. It won't send the updates back to the
catalogd. This is achieved by adding a special filter_prefix when registering
to statestore:
{code:cpp}
// The catalogd never needs to read any entries from the topic. It only
publishes
// entries. So, we set a prefix to some random character that we know won't
be a
// prefix of any key. This saves a bit of network communication from the
statestore
// back to the catalog.
string filter_prefix = "!";
Status status = statestore_subscriber_->AddTopic(IMPALA_CATALOG_TOPIC,
/* is_transient=*/ false, /* populate_min_subscriber_topic_version=*/
false,
filter_prefix, cb);
{code}
[https://github.com/apache/impala/blob/4bc86ac638fb85deda317b9b1dfd1f574fa0553b/be/src/catalog/catalog-server.cc#L320-L327]
> Server network util of Statestored is much higher than impala-3.4
> -----------------------------------------------------------------
>
> Key: IMPALA-11659
> URL: https://issues.apache.org/jira/browse/IMPALA-11659
> Project: IMPALA
> Issue Type: Bug
> Components: Perf Investigation
> Affects Versions: Impala 4.1.0
> Reporter: Yuchen Fan
> Priority: Major
> Attachments: image-2022-10-13-19-49-28-456.png,
> image-2022-10-13-19-57-31-248.png, image-2022-10-13-20-08-24-931.png
>
>
> We found that server network traffic of Statestored is rised immediatrly
> after upgraded to Impala-4.1. Using 'iftop' shows Statestored has aboud 6MB/s
> outbound network communication with every Impalad. With more than 250 nodes
> of Impalad, Statestored server has 10~15Gb/s(1~2GB/s) outbound(60x higher
> than it before upgrade) and about 90Mb/s(>10MB/s) inbound(10x higher than it
> before upgrade) network communication, which will occupy about 75% server
> network util. TCP packet snapshot shows detail of packet is 'Pool' and 'Stat'
> information in impala-request-queue topic. We found that value of key 'STAT:'
> contains per host stat of all Impalads. Related function is from
> https://issues.apache.org/jira/browse/IMPALA-8762. So initial update size of
> impala-request-queue is more than 4MB when there are more than 250 Impalads.
> If cluster has more Impalad, Statestored server may hang because of network
> blocking.
> I think coordinators updating topic of per host statistics is just fine. All
> Impalads(including coordinators and executors) registering subscriber to
> topic of impala-request-queue makes Statestored need to broadcast all per
> node statistics to all Impalads.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
