[
https://issues.apache.org/jira/browse/IMPALA-10519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17287174#comment-17287174
]
Thomas Tauber-Marshall commented on IMPALA-10519:
-------------------------------------------------
That's a really good question, and definitely an important area of follow up
work. Presumably the answer is going to depend a lot on how many scans Kudu can
constructively work on at the same time, which in turn I would assume depends
on the characteristics of the hardware and Kudu's configuration, eg. how many
physical disk reads are supported simultaneously.
For context, this was filed in response to an issue seen in a real cluster
where Kudu scans were slowing down significantly on individual nodes in an
otherwise healthy cluster. We observed in pstacks taken of Impala that there
were a large number (often 40-50) of stack traces in
kudu::client::KuduScanner::Data::SendScanRpc, suggesting that there were a lot
of Kudu scan rpcs waiting on reactor threads (though to be clear we haven't yet
been able to deploy this fix there and confirm it actually works as expected).
> Allow setting of num_reactors for KuduClient
> --------------------------------------------
>
> Key: IMPALA-10519
> URL: https://issues.apache.org/jira/browse/IMPALA-10519
> Project: IMPALA
> Issue Type: Improvement
> Affects Versions: Impala 4.0
> Reporter: Thomas Tauber-Marshall
> Assignee: Thomas Tauber-Marshall
> Priority: Major
>
> The KuduClient allows setting a number of reactor threads, which are used for
> sending rpcs to Kudu such as for scans. Impala does not currently set this
> value, so it defaults to 4 threads, which may be too few for many use cases,
> since there is a single KuduClient per impalad. We should determine a better
> default (eg. based on the number of cpu cores) and also expose a flag to make
> it configurable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
