[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13240892#comment-13240892 ] Hudson commented on HBASE-5190: --- Integrated in HBase-TRUNK #2697 (See [https://builds.apache.org/job/HBase-TRUNK/2697/]) HBASE-5190 Limit the IPC queue size based on calls' payload size (Ted's addendum) (Revision 1305468) Result = FAILURE jdcryans : Files : * /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java Limit the IPC queue size based on calls' payload size - Key: HBASE-5190 URL: https://issues.apache.org/jira/browse/HBASE-5190 Project: HBase Issue Type: Improvement Affects Versions: 0.90.5 Reporter: Jean-Daniel Cryans Assignee: Jean-Daniel Cryans Fix For: 0.94.0, 0.96.0 Attachments: 5190.addendum, HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions: - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception. - Same accounting but instead block the call when it comes in until space is made available. - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239211#comment-13239211 ] Hudson commented on HBASE-5190: --- Integrated in HBase-TRUNK-security #151 (See [https://builds.apache.org/job/HBase-TRUNK-security/151/]) HBASE-5190 Limit the IPC queue size based on calls' payload size (Ted's addendum) (Revision 1305468) Result = FAILURE jdcryans : Files : * /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java Limit the IPC queue size based on calls' payload size - Key: HBASE-5190 URL: https://issues.apache.org/jira/browse/HBASE-5190 Project: HBase Issue Type: Improvement Affects Versions: 0.90.5 Reporter: Jean-Daniel Cryans Assignee: Jean-Daniel Cryans Fix For: 0.94.0, 0.96.0 Attachments: 5190.addendum, HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions: - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception. - Same accounting but instead block the call when it comes in until space is made available. - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238697#comment-13238697 ] Hudson commented on HBASE-5190: --- Integrated in HBase-0.94 #56 (See [https://builds.apache.org/job/HBase-0.94/56/]) HBASE-5190 Limit the IPC queue size based on calls' payload size (Ted's addendum) (Revision 1305469) Result = FAILURE jdcryans : Files : * /hbase/branches/0.94/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java Limit the IPC queue size based on calls' payload size - Key: HBASE-5190 URL: https://issues.apache.org/jira/browse/HBASE-5190 Project: HBase Issue Type: Improvement Affects Versions: 0.90.5 Reporter: Jean-Daniel Cryans Assignee: Jean-Daniel Cryans Fix For: 0.94.0, 0.96.0 Attachments: 5190.addendum, HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions: - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception. - Same accounting but instead block the call when it comes in until space is made available. - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238999#comment-13238999 ] Hudson commented on HBASE-5190: --- Integrated in HBase-0.94-security #3 (See [https://builds.apache.org/job/HBase-0.94-security/3/]) HBASE-5190 Limit the IPC queue size based on calls' payload size (Ted's addendum) (Revision 1305469) Result = ABORTED jdcryans : Files : * /hbase/branches/0.94/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java Limit the IPC queue size based on calls' payload size - Key: HBASE-5190 URL: https://issues.apache.org/jira/browse/HBASE-5190 Project: HBase Issue Type: Improvement Affects Versions: 0.90.5 Reporter: Jean-Daniel Cryans Assignee: Jean-Daniel Cryans Fix For: 0.94.0, 0.96.0 Attachments: 5190.addendum, HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions: - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception. - Same accounting but instead block the call when it comes in until space is made available. - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237977#comment-13237977 ] Lars Hofhansl commented on HBASE-5190: -- +1 on addendum Limit the IPC queue size based on calls' payload size - Key: HBASE-5190 URL: https://issues.apache.org/jira/browse/HBASE-5190 Project: HBase Issue Type: Improvement Affects Versions: 0.90.5 Reporter: Jean-Daniel Cryans Assignee: Jean-Daniel Cryans Fix For: 0.94.0, 0.96.0 Attachments: 5190.addendum, HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions: - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception. - Same accounting but instead block the call when it comes in until space is made available. - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238014#comment-13238014 ] stack commented on HBASE-5190: -- Addendum looks good to me. +1 Limit the IPC queue size based on calls' payload size - Key: HBASE-5190 URL: https://issues.apache.org/jira/browse/HBASE-5190 Project: HBase Issue Type: Improvement Affects Versions: 0.90.5 Reporter: Jean-Daniel Cryans Assignee: Jean-Daniel Cryans Fix For: 0.94.0, 0.96.0 Attachments: 5190.addendum, HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions: - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception. - Same accounting but instead block the call when it comes in until space is made available. - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237529#comment-13237529 ] Hudson commented on HBASE-5190: --- Integrated in HBase-TRUNK #2694 (See [https://builds.apache.org/job/HBase-TRUNK/2694/]) HBASE-5190 Limit the IPC queue size based on calls' payload size (Revision 1304634) Result = SUCCESS jdcryans : Files : * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java Limit the IPC queue size based on calls' payload size - Key: HBASE-5190 URL: https://issues.apache.org/jira/browse/HBASE-5190 Project: HBase Issue Type: Improvement Affects Versions: 0.90.5 Reporter: Jean-Daniel Cryans Assignee: Jean-Daniel Cryans Fix For: 0.94.0, 0.96.0 Attachments: HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions: - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception. - Same accounting but instead block the call when it comes in until space is made available. - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236790#comment-13236790 ] Jean-Daniel Cryans commented on HBASE-5190: --- It's not exactly 10, it's num_handlers * 10 so by default 100. Is that better? Limit the IPC queue size based on calls' payload size - Key: HBASE-5190 URL: https://issues.apache.org/jira/browse/HBASE-5190 Project: HBase Issue Type: Improvement Affects Versions: 0.90.5 Reporter: Jean-Daniel Cryans Assignee: Jean-Daniel Cryans Fix For: 0.94.0 Attachments: HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions: - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception. - Same accounting but instead block the call when it comes in until space is made available. - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236810#comment-13236810 ] stack commented on HBASE-5190: -- @JD Can do in another issue. Limit the IPC queue size based on calls' payload size - Key: HBASE-5190 URL: https://issues.apache.org/jira/browse/HBASE-5190 Project: HBase Issue Type: Improvement Affects Versions: 0.90.5 Reporter: Jean-Daniel Cryans Assignee: Jean-Daniel Cryans Fix For: 0.94.0 Attachments: HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions: - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception. - Same accounting but instead block the call when it comes in until space is made available. - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236855#comment-13236855 ] Lars Hofhansl commented on HBASE-5190: -- @J-D: Oh yeah forgot that. Fine then. :) Still +1 Limit the IPC queue size based on calls' payload size - Key: HBASE-5190 URL: https://issues.apache.org/jira/browse/HBASE-5190 Project: HBase Issue Type: Improvement Affects Versions: 0.90.5 Reporter: Jean-Daniel Cryans Assignee: Jean-Daniel Cryans Fix For: 0.94.0 Attachments: HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions: - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception. - Same accounting but instead block the call when it comes in until space is made available. - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237124#comment-13237124 ] Lars Hofhansl commented on HBASE-5190: -- Wanna commit, J-D? Limit the IPC queue size based on calls' payload size - Key: HBASE-5190 URL: https://issues.apache.org/jira/browse/HBASE-5190 Project: HBase Issue Type: Improvement Affects Versions: 0.90.5 Reporter: Jean-Daniel Cryans Assignee: Jean-Daniel Cryans Fix For: 0.94.0 Attachments: HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions: - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception. - Same accounting but instead block the call when it comes in until space is made available. - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237146#comment-13237146 ] Jean-Daniel Cryans commented on HBASE-5190: --- Yep, right after I finishing running medium tests on trunk. Limit the IPC queue size based on calls' payload size - Key: HBASE-5190 URL: https://issues.apache.org/jira/browse/HBASE-5190 Project: HBase Issue Type: Improvement Affects Versions: 0.90.5 Reporter: Jean-Daniel Cryans Assignee: Jean-Daniel Cryans Fix For: 0.94.0 Attachments: HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions: - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception. - Same accounting but instead block the call when it comes in until space is made available. - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237307#comment-13237307 ] Hudson commented on HBASE-5190: --- Integrated in HBase-0.94 #49 (See [https://builds.apache.org/job/HBase-0.94/49/]) HBASE-5190 Limit the IPC queue size based on calls' payload size (Revision 1304635) Result = SUCCESS jdcryans : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java Limit the IPC queue size based on calls' payload size - Key: HBASE-5190 URL: https://issues.apache.org/jira/browse/HBASE-5190 Project: HBase Issue Type: Improvement Affects Versions: 0.90.5 Reporter: Jean-Daniel Cryans Assignee: Jean-Daniel Cryans Fix For: 0.94.0, 0.96.0 Attachments: HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions: - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception. - Same accounting but instead block the call when it comes in until space is made available. - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235852#comment-13235852 ] Jean-Daniel Cryans commented on HBASE-5190: --- The current patch works, I've tested it extensively through massive imports. Current concerns: - I haven't done a performance comparison, like is it going to slow down traffic because of additional checks? Most of my testing was done so that I'm hitting the limit all the time, so that does definitely slow down my throughput but it's expected :) - The exception Call queue already full doesn't make it to the client, what happens is that it's being printed server-side and the client gets an EOF. That's bad. - What default should we use? In my testing I saw that 100MB might be too small, but ideally that needs to scale with the amount of memory. I don't mind finishing this for 0.94 if there's demand/motivation for it. Limit the IPC queue size based on calls' payload size - Key: HBASE-5190 URL: https://issues.apache.org/jira/browse/HBASE-5190 Project: HBase Issue Type: Improvement Affects Versions: 0.90.5 Reporter: Jean-Daniel Cryans Assignee: Jean-Daniel Cryans Fix For: 0.94.0 Attachments: HBASE-5190.patch Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions: - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception. - Same accounting but instead block the call when it comes in until space is made available. - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235895#comment-13235895 ] Lars Hofhansl commented on HBASE-5190: -- I doubt you'll see a slowdown from this. Then again AtomicLong needs a memory barrier as far as I know, so could potentially add a noticeable slowdown. Hmm... 100MB is per RegionServer, right? Does seem a bit small. Maybe 1G? We really need a streaming API for large rows, but that's a different story. I take it that maybe this is something to consider for 0.96. Agreed? Limit the IPC queue size based on calls' payload size - Key: HBASE-5190 URL: https://issues.apache.org/jira/browse/HBASE-5190 Project: HBase Issue Type: Improvement Affects Versions: 0.90.5 Reporter: Jean-Daniel Cryans Assignee: Jean-Daniel Cryans Fix For: 0.94.0 Attachments: HBASE-5190.patch Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions: - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception. - Same accounting but instead block the call when it comes in until space is made available. - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235909#comment-13235909 ] Jean-Daniel Cryans commented on HBASE-5190: --- bq. 100MB is per RegionServer, right? Does seem a bit small. Maybe 1G? Might be a good default, those with the default heap will definitely not get any help here though. bq. I take it that maybe this is something to consider for 0.96. Agreed? The more I think about it, the more I want this in 0.94 because it can really give us a better understanding of those issues we see on the mailing list. Limit the IPC queue size based on calls' payload size - Key: HBASE-5190 URL: https://issues.apache.org/jira/browse/HBASE-5190 Project: HBase Issue Type: Improvement Affects Versions: 0.90.5 Reporter: Jean-Daniel Cryans Assignee: Jean-Daniel Cryans Fix For: 0.94.0 Attachments: HBASE-5190.patch Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions: - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception. - Same accounting but instead block the call when it comes in until space is made available. - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235970#comment-13235970 ] Lars Hofhansl commented on HBASE-5190: -- Fair enough. I think we don't need to worry about a slow down here. So that leaves passing the exception on to the client. How is the exception handled currently when we get over the queue length, if the client in that case also gets an EOF then we can live with that. Limit the IPC queue size based on calls' payload size - Key: HBASE-5190 URL: https://issues.apache.org/jira/browse/HBASE-5190 Project: HBase Issue Type: Improvement Affects Versions: 0.90.5 Reporter: Jean-Daniel Cryans Assignee: Jean-Daniel Cryans Fix For: 0.94.0 Attachments: HBASE-5190.patch Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions: - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception. - Same accounting but instead block the call when it comes in until space is made available. - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236037#comment-13236037 ] Jean-Daniel Cryans commented on HBASE-5190: --- Instead of throwing the IOE it's better if I put it in the try block just below in the code as it already handles the setting up of a response. I just need to change the messaging there and refactor some bits. Limit the IPC queue size based on calls' payload size - Key: HBASE-5190 URL: https://issues.apache.org/jira/browse/HBASE-5190 Project: HBase Issue Type: Improvement Affects Versions: 0.90.5 Reporter: Jean-Daniel Cryans Assignee: Jean-Daniel Cryans Fix For: 0.94.0 Attachments: HBASE-5190.patch Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions: - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception. - Same accounting but instead block the call when it comes in until space is made available. - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size
[
https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236110#comment-13236110
]
Zhihong Yu commented on HBASE-5190:
---
Should we use high_scale_lib for the callQueueSize ?
{code}
if (LOG.isDebugEnabled())
-LOG.debug( got call # + id + , + buf.length + bytes);
+LOG.debug( got call # + id + , + callSize + bytes);
{code}
Please wrap the log with braces.
{code}
+IOException.class.getName(),
+Call queue is full, retry later);
{code}
Would giving some details about queue capacity help client make better decision
?
{code}
+// TODO add a deprecated comment for ipc.server.max.queue.size
{code}
Should the above comment be added to 0.90.7 ?
Limit the IPC queue size based on calls' payload size
-
Key: HBASE-5190
URL: https://issues.apache.org/jira/browse/HBASE-5190
Project: HBase
Issue Type: Improvement
Affects Versions: 0.90.5
Reporter: Jean-Daniel Cryans
Assignee: Jean-Daniel Cryans
Fix For: 0.94.0
Attachments: HBASE-5190-v2.patch, HBASE-5190.patch
Currently we limit the number of calls in the IPC queue only on their count.
It used to be really high and was dropped down recently to num_handlers * 10
(so 100 by default) because it was easy to OOME yourself when huge calls were
being queued. It's still possible to hit this problem if you use really big
values and/or a lot of handlers, so the idea is that we should take into
account the payload size. I can see 3 solutions:
- Do the accounting outside of the queue itself for all calls coming in and
out and when a call doesn't fit, throw a retryable exception.
- Same accounting but instead block the call when it comes in until space is
made available.
- Add a new parameter for the maximum size (in bytes) of a Call and then set
the size the IPC queue (in terms of the number of items) so that it could
only contain as many items as some predefined maximum size (in bytes) for the
whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236115#comment-13236115 ] stack commented on HBASE-5190: -- +1 after chatting w/ J-D about what happens when we fill the queue (we send client IOE, retries, fills network w/ more back and forths but it should act like a pushback on the client...) Limit the IPC queue size based on calls' payload size - Key: HBASE-5190 URL: https://issues.apache.org/jira/browse/HBASE-5190 Project: HBase Issue Type: Improvement Affects Versions: 0.90.5 Reporter: Jean-Daniel Cryans Assignee: Jean-Daniel Cryans Fix For: 0.94.0 Attachments: HBASE-5190-v2.patch, HBASE-5190.patch Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions: - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception. - Same accounting but instead block the call when it comes in until space is made available. - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236205#comment-13236205 ] Lars Hofhansl commented on HBASE-5190: -- +1 on v3 Limit the IPC queue size based on calls' payload size - Key: HBASE-5190 URL: https://issues.apache.org/jira/browse/HBASE-5190 Project: HBase Issue Type: Improvement Affects Versions: 0.90.5 Reporter: Jean-Daniel Cryans Assignee: Jean-Daniel Cryans Fix For: 0.94.0 Attachments: HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions: - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception. - Same accounting but instead block the call when it comes in until space is made available. - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236227#comment-13236227 ] Lars Hofhansl commented on HBASE-5190: -- Actually, can we new increase the default queue length? For small requests 10 is awfully small. Limit the IPC queue size based on calls' payload size - Key: HBASE-5190 URL: https://issues.apache.org/jira/browse/HBASE-5190 Project: HBase Issue Type: Improvement Affects Versions: 0.90.5 Reporter: Jean-Daniel Cryans Assignee: Jean-Daniel Cryans Fix For: 0.94.0 Attachments: HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions: - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception. - Same accounting but instead block the call when it comes in until space is made available. - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236354#comment-13236354 ] stack commented on HBASE-5190: -- bq. Actually, can we new increase the default queue length? This is a good suggestion Limit the IPC queue size based on calls' payload size - Key: HBASE-5190 URL: https://issues.apache.org/jira/browse/HBASE-5190 Project: HBase Issue Type: Improvement Affects Versions: 0.90.5 Reporter: Jean-Daniel Cryans Assignee: Jean-Daniel Cryans Fix For: 0.94.0 Attachments: HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions: - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception. - Same accounting but instead block the call when it comes in until space is made available. - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13234536#comment-13234536 ] Lars Hofhansl commented on HBASE-5190: -- Wanna finish this for 0.94, J-D? Or should bump to 0.96? Limit the IPC queue size based on calls' payload size - Key: HBASE-5190 URL: https://issues.apache.org/jira/browse/HBASE-5190 Project: HBase Issue Type: Improvement Affects Versions: 0.90.5 Reporter: Jean-Daniel Cryans Assignee: Jean-Daniel Cryans Fix For: 0.94.0 Attachments: HBASE-5190.patch Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions: - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception. - Same accounting but instead block the call when it comes in until space is made available. - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13185898#comment-13185898 ] stack commented on HBASE-5190: -- Talking w/ J-D on backchannel, #2 is bad (blocking is what we currently do too so its bad too). If we block, its likely client can timeout. If it times out, it will redo the request adding a new payload to the serverside ipc queue. Now queue has same load twice (bad if ICV). Server doesn't know client timed out till it goes to reply and the client is no longer there so it can double-count or double enter value. Limit the IPC queue size based on calls' payload size - Key: HBASE-5190 URL: https://issues.apache.org/jira/browse/HBASE-5190 Project: HBase Issue Type: Improvement Affects Versions: 0.90.5 Reporter: Jean-Daniel Cryans Fix For: 0.94.0 Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions: - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception. - Same accounting but instead block the call when it comes in until space is made available. - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13185909#comment-13185909 ] Jean-Daniel Cryans commented on HBASE-5190: --- About #3. I like the elegance of that solution since we don't have to keep track of the calls in flight but I see 2 big issues: - if you set a max call size you need to keep both clients and servers in sink and also decide who's going to do the check. - if you plan for big calls by default, you may end up with a tiny size for the queue. For example, let's say you cap calls at 10% of the heap and set their max individual size at 10MB, it means that you can only allow 10 items in the queue (and you don't account listeners). Limit the IPC queue size based on calls' payload size - Key: HBASE-5190 URL: https://issues.apache.org/jira/browse/HBASE-5190 Project: HBase Issue Type: Improvement Affects Versions: 0.90.5 Reporter: Jean-Daniel Cryans Fix For: 0.94.0 Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions: - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception. - Same accounting but instead block the call when it comes in until space is made available. - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13185943#comment-13185943 ] Jean-Daniel Cryans commented on HBASE-5190: --- About #1. The issue with this is that by throwing the call back to the client you generate a lot more IO and CPU (serializing, deserializing) and there's the possibility the of starving those clients that have bigger calls. Limit the IPC queue size based on calls' payload size - Key: HBASE-5190 URL: https://issues.apache.org/jira/browse/HBASE-5190 Project: HBase Issue Type: Improvement Affects Versions: 0.90.5 Reporter: Jean-Daniel Cryans Fix For: 0.94.0 Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions: - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception. - Same accounting but instead block the call when it comes in until space is made available. - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13186043#comment-13186043 ] Zhihong Yu commented on HBASE-5190: --- Can we utilize Cliff Click Counter to reduce contention ? Limit the IPC queue size based on calls' payload size - Key: HBASE-5190 URL: https://issues.apache.org/jira/browse/HBASE-5190 Project: HBase Issue Type: Improvement Affects Versions: 0.90.5 Reporter: Jean-Daniel Cryans Assignee: Jean-Daniel Cryans Fix For: 0.94.0 Attachments: HBASE-5190.patch Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions: - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception. - Same accounting but instead block the call when it comes in until space is made available. - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13186047#comment-13186047 ] Todd Lipcon commented on HBASE-5190: I haven't looked at the patch yet (sorry) but it seems from the discussion that this is trying to do the limiting at queue level. If instead, we did the limiting where we read from the network socket, then we'd have an automatic backpressure mechanism - clients would block on send, and the memory usage would be pushed to the client side TCP buffers rather than the HBase heap. The downside is, of course, that META requests and such can get blocked behind others. But that can be solved other ways as well (eg separate TCP ports for meta regions or somesuch) Limit the IPC queue size based on calls' payload size - Key: HBASE-5190 URL: https://issues.apache.org/jira/browse/HBASE-5190 Project: HBase Issue Type: Improvement Affects Versions: 0.90.5 Reporter: Jean-Daniel Cryans Assignee: Jean-Daniel Cryans Fix For: 0.94.0 Attachments: HBASE-5190.patch Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions: - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception. - Same accounting but instead block the call when it comes in until space is made available. - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size
[
https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13186052#comment-13186052
]
Zhihong Yu commented on HBASE-5190:
---
{code}
- this.priorityCallQueue = new LinkedBlockingQueueCall(maxQueueSize); //
TODO hack on size
+ this.priorityCallQueue = new LinkedBlockingQueueCall(maxQueueLength);
// TODO hack on size
{code}
The TODO above can be dropped.
Todd's comment makes sense. But since it requires more work, we can checkin
this patch for now.
Limit the IPC queue size based on calls' payload size
-
Key: HBASE-5190
URL: https://issues.apache.org/jira/browse/HBASE-5190
Project: HBase
Issue Type: Improvement
Affects Versions: 0.90.5
Reporter: Jean-Daniel Cryans
Assignee: Jean-Daniel Cryans
Fix For: 0.94.0
Attachments: HBASE-5190.patch
Currently we limit the number of calls in the IPC queue only on their count.
It used to be really high and was dropped down recently to num_handlers * 10
(so 100 by default) because it was easy to OOME yourself when huge calls were
being queued. It's still possible to hit this problem if you use really big
values and/or a lot of handlers, so the idea is that we should take into
account the payload size. I can see 3 solutions:
- Do the accounting outside of the queue itself for all calls coming in and
out and when a call doesn't fit, throw a retryable exception.
- Same accounting but instead block the call when it comes in until space is
made available.
- Add a new parameter for the maximum size (in bytes) of a Call and then set
the size the IPC queue (in terms of the number of items) so that it could
only contain as many items as some predefined maximum size (in bytes) for the
whole queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size
[ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13186064#comment-13186064 ] Jean-Daniel Cryans commented on HBASE-5190: --- bq. The TODO above can be dropped My understanding of that todo is that it shouldn't be setting as many PRI as there's normal handlers, so it still stands. bq. Todd's comment makes sense I agree. bq. But since it requires more work, we can checkin this patch for now. I disagree, I'll investigate his idea and I'm in no hurry to check this in since it's targeted for 0.94 Limit the IPC queue size based on calls' payload size - Key: HBASE-5190 URL: https://issues.apache.org/jira/browse/HBASE-5190 Project: HBase Issue Type: Improvement Affects Versions: 0.90.5 Reporter: Jean-Daniel Cryans Assignee: Jean-Daniel Cryans Fix For: 0.94.0 Attachments: HBASE-5190.patch Currently we limit the number of calls in the IPC queue only on their count. It used to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because it was easy to OOME yourself when huge calls were being queued. It's still possible to hit this problem if you use really big values and/or a lot of handlers, so the idea is that we should take into account the payload size. I can see 3 solutions: - Do the accounting outside of the queue itself for all calls coming in and out and when a call doesn't fit, throw a retryable exception. - Same accounting but instead block the call when it comes in until space is made available. - Add a new parameter for the maximum size (in bytes) of a Call and then set the size the IPC queue (in terms of the number of items) so that it could only contain as many items as some predefined maximum size (in bytes) for the whole queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
