[
https://issues.apache.org/jira/browse/IOTDB-1005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17244936#comment-17244936
]
Houliang Qi commented on IOTDB-1005:
------------------------------------
As we can see from the log, the auto-resize of the max frame size bring another
problem, which will cause OOM, as we can see from the profile, the byte[]
array consume about 18GB, which is generated by thrift thread.
this analyze alose brings another one problem, its reasonable that the max
thread client num is 10000？
I think we should do the following two things:
1. There must be a segmentation mechanism to divide large requests into small
requests, as far I know, apache HBase default max frame is 2MB,
2. The maximum number of threads needs to be reduced(now the default value is
1000), but some tests may be needed to set the appropriate size.
[^log.out]!image-2020-12-07-10-52-44-343.png!
> [Distributed] Make the frame size of thrift scalable
> ----------------------------------------------------
>
> Key: IOTDB-1005
> URL: https://issues.apache.org/jira/browse/IOTDB-1005
> Project: Apache IoTDB
> Issue Type: Improvement
> Components: Core/Cluster
> Reporter: Houliang Qi
> Assignee: Houliang Qi
> Priority: Major
> Labels: pull-request-available
> Attachments: image-2020-12-07-10-52-44-343.png, log.out
>
>
> As described in IOTDB-998[1], when the snapshot size larger than the max
> frame size, it would fail when do snapshot, another way is to make the frame
> size of thrift is scalable. it would be easier than IOTDB-998
> [1]https://issues.apache.org/jira/browse/IOTDB-998
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
