[
https://issues.apache.org/jira/browse/IMPALA-9221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17542596#comment-17542596
]
Michael Smith edited comment on IMPALA-9221 at 5/26/22 5:29 PM:
----------------------------------------------------------------
Captured some perf numbers in
[https://docs.google.com/spreadsheets/d/1rrgGjGvX9KkEfWDcZyTrowRKs6hZDXY60sRVrx0LWIU/edit?usp=sharing]
using num_replicas=25 (hard-coded at
https://github.com/apache/impala/blob/4.1.0-rc2/be/src/scheduling/executor-group.cc#L27)
and comparing read performance with 10, 50, and 200 hosts.
!image-2022-05-26-10-23-57-678.png|width=640,height=400!
Map vs flat_map shows ~5% slower for flat_map at 10 hosts, improving to ~15%
faster at 200 hosts.
Test script
{code}
for host in 10 50 200; do
for reads in 1000000 10000000 100000000; do
$IMPALA_HOME/be/build/release/experiments/hash-ring-util -num_hosts=$host
-num_replicas=25 -num_reads=$reads > hash-ring-$host-$reads.out
done
done
{code}
was (Author: JIRAUSER288956):
Captured some perf numbers in
[https://docs.google.com/spreadsheets/d/1rrgGjGvX9KkEfWDcZyTrowRKs6hZDXY60sRVrx0LWIU/edit?usp=sharing]
using num_replicas=25 (hard-coded at
https://github.com/apache/impala/blob/4.1.0-rc2/be/src/scheduling/executor-group.cc#L27)
and comparing read performance with 10, 50, and 200 hosts.
!image-2022-05-26-10-23-57-678.png|width=640,height=400!
Map vs flat_map shows ~5% slower for flat_map at 10 hosts, improving to ~15%
faster at 200 hosts.
Test script
{code}
for host in 10 50 200; do
for reads in 1000000 10000000 100000000; do
# Hard-codes 25 replicas because that's what's hard-coded in the executor.
$IMPALA_HOME/be/build/release/experiments/hash-ring-util -num_hosts=$host
-num_replicas=25 -num_reads=$reads > hash-ring-$host-$reads.out
done
done
{code}
> Optimize HashRing's map implementation
> --------------------------------------
>
> Key: IMPALA-9221
> URL: https://issues.apache.org/jira/browse/IMPALA-9221
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Affects Versions: Impala 3.4.0
> Reporter: Joe McDonnell
> Assignee: Michael Smith
> Priority: Major
> Attachments: image-2022-05-26-10-23-57-678.png
>
>
> The hash ring used for consistent scheduling currently uses a std::map for
> the hash-to-IpAddr lookup. HashRing is heavy on reads, with writes only
> happening when executors come and go. There are some cases where we copy the
> HashRing.
> The standard map uses a large number of small allocations. This hurts cache
> performance, adds overhead, and also increases the cost of copying the
> structure. Something like boost's flat_map or Abseil's btree_map is likely to
> be more efficient.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
