[
https://issues.apache.org/jira/browse/IMPALA-9903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17169434#comment-17169434
]
ASF subversion and git services commented on IMPALA-9903:
---------------------------------------------------------
Commit da2999afd9ddc45d35141649d17db507e03ee9bf in impala's branch
refs/heads/master from Grant Henke
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=da2999a ]
IMPALA-9903: Reduce Kudu openTable calls per query
This patch reduces the number of Kudu openTable calls for the
lifetime of a query by storing the KuduTable object in the
Analyzer GlobalState and using it in the KuduScanNode.
It does not cache the KuduTable object longer than a single
query, does not impact DDL statements, and does not
introduce the need to invalidate metadata when interacting with
Kudu tables.
Additionally, this patch adjusts the backend scanner to use the
KuduTable instance from the KuduScanner instead of using
openTable to get a new instance.
Reducing the number of openTable calls is important because each
call results in a GetTableSchema RPC to the remote leader Kudu
master. With very high rates of queries against Kudu tables this
can overload the master leading to degraded query performance.
In manual testing this patched reduced the Kudu GetTableSchema
RPC calls to the master from 5 per query to 1 per query.
Change-Id: Iec12a5be9b30e19a123142af5453a91bd4300b63
Reviewed-on: http://gerrit.cloudera.org:8080/16120
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Queries on a Kudu table call openTable multiple times
> -----------------------------------------------------
>
> Key: IMPALA-9903
> URL: https://issues.apache.org/jira/browse/IMPALA-9903
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
> Affects Versions: Impala 3.4.0
> Reporter: Grant Henke
> Priority: Major
> Labels: kudu, performance
>
> When testing the results of KUDU-1802 I saw that each query ran would result
> in 5 GetTableSchema requests to the Kudu master server. The request comes
> from each KuduClient.openTable call in the Impala frontend.
> https://github.com/apache/impala/search?l=Java&q=openTable
> Ideally Impala would only call KuduClient.openTable once and then use the
> returned KuduTable object for the length of the query. That would result in
> 5x fewer remote RPC calls to the Kudu master server per query.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
