[
https://issues.apache.org/jira/browse/PHOENIX-4903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616584#comment-16616584
]
ASF GitHub Bot commented on PHOENIX-4903:
-----------------------------------------
GitHub user ortutay opened a pull request:
https://github.com/apache/phoenix/pull/350
PHOENIX-4903 Use same hash cache RPC message across all calls
See ticket https://issues.apache.org/jira/browse/PHOENIX-4903 , this change
makes it so a single RPC message is used to distribute the hash cache across
region servers. The previous code would have O(N) memory usage. This has O(1)
memory usage.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ortutay/phoenix
PHOENIX-4903-hash-cache-rpc-message-mem
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/phoenix/pull/350.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #350
----
commit 0d11f7a388ce0e856329ef7e93a719433229e5df
Author: Marcell Ortutay <marcell.ortutay@...>
Date: 2018-09-16T04:13:54Z
PHOENIX-4903 Use same hash cache RPC message across all calls
----
> Hash cache RPC uses O(N) memory on master
> -----------------------------------------
>
> Key: PHOENIX-4903
> URL: https://issues.apache.org/jira/browse/PHOENIX-4903
> Project: Phoenix
> Issue Type: Improvement
> Reporter: Marcell Ortutay
> Priority: Major
>
> To distribute the hash cache to region servers, the master node makes an
> `AddServerCacheRequest` RPC to each region servers. If there are N region
> servers, it makes N of these RPC's. For each of the region servers, it
> generates a serialized RPC message and sends it out. This happens
> concurrently, and the result is that it uses O(N) memory on the master.
> As an example, if the `AddServerCacheRequest` RPC message is 100MB, and you
> have a cluster of 100 nodes, it would use 10GB memory on the master,
> potentially resulting in an "OutOfMemory" exception.
> It would be better if the master could use O(1) memory for the RPC.
> I observed this behavior in Phoenix 4.14.1
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
