[
https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13235852#comment-13235852
]
Jean-Daniel Cryans commented on HBASE-5190:
-------------------------------------------
The current patch works, I've tested it extensively through massive imports.
Current concerns:
- I haven't done a performance comparison, like is it going to slow down
traffic because of additional checks? Most of my testing was done so that I'm
hitting the limit all the time, so that does definitely slow down my throughput
but it's expected :)
- The exception "Call queue already full" doesn't make it to the client, what
happens is that it's being printed server-side and the client gets an EOF.
That's bad.
- What default should we use? In my testing I saw that 100MB might be too
small, but ideally that needs to scale with the amount of memory.
I don't mind finishing this for 0.94 if there's demand/motivation for it.
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>
> Key: HBASE-5190
> URL: https://issues.apache.org/jira/browse/HBASE-5190
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.94.0
>
> Attachments: HBASE-5190.patch
>
>
> Currently we limit the number of calls in the IPC queue only on their count.
> It used to be really high and was dropped down recently to num_handlers * 10
> (so 100 by default) because it was easy to OOME yourself when huge calls were
> being queued. It's still possible to hit this problem if you use really big
> values and/or a lot of handlers, so the idea is that we should take into
> account the payload size. I can see 3 solutions:
> - Do the accounting outside of the queue itself for all calls coming in and
> out and when a call doesn't fit, throw a retryable exception.
> - Same accounting but instead block the call when it comes in until space is
> made available.
> - Add a new parameter for the maximum size (in bytes) of a Call and then set
> the size the IPC queue (in terms of the number of items) so that it could
> only contain as many items as some predefined maximum size (in bytes) for the
> whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
