[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13767357#comment-13767357 ] stack commented on HBASE-8884: -- Continuation over in HBASE-9461. Changes CallRunner so not Runnable. Other 'cleaning' so easier to make sense of what is going on (does other stuff like make the CallRunner do all call size accounting, makes it so Scheduler does less reaching into RpcServer instance, takes rpc server Interface instead, etc.). Pluggable RpcScheduler -- Key: HBASE-8884 URL: https://issues.apache.org/jira/browse/HBASE-8884 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi Assignee: Chao Shi Fix For: 0.98.0 Attachments: hbase-8884.patch, hbase-8884-v2.patch, hbase-8884-v3.patch, hbase-8884-v4.patch, hbase-8884-v5.patch, hbase-8884-v6.patch, hbase-8884-v7.patch, hbase-8884-v8.patch Today, the RPC scheduling mechanism is pretty simple: it execute requests in isolated thread-pools based on their priority. In the current implementation, all normal get/put requests are using the same pool. We'd like to add some per-user or per-region level isolation, so that a misbehaved user/region will not saturate the thread-pool and cause DoS to others easily. The idea is similar to FairScheduler in MR. The current scheduling code is not standalone and is mixed with others (Connection#processRequest). The issue is the first step to extract it to an interface, so that people are free to write and test their own implementations. This patch doesn't make it completely pluggable yet, as some parameters are pass from constructor. This is because HMaster and HRegionServer both use RpcServer and they have different thread-pool size config. Let me know if you have a solution to this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13762774#comment-13762774 ] Chao Shi commented on HBASE-8884: - stack, could you please explain a little bit more on pooling of buffers across requests. I don't quite understand. In fact, the very first rationale for us to introduce pluggable RpcScheduler, is that we want to isolate read and write ops. So we can simply write a RpcScheduler with two thread-pools. My case is pretty easy, and I'm interested to listen about your case. Pluggable RpcScheduler -- Key: HBASE-8884 URL: https://issues.apache.org/jira/browse/HBASE-8884 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi Assignee: Chao Shi Fix For: 0.98.0 Attachments: hbase-8884.patch, hbase-8884-v2.patch, hbase-8884-v3.patch, hbase-8884-v4.patch, hbase-8884-v5.patch, hbase-8884-v6.patch, hbase-8884-v7.patch, hbase-8884-v8.patch Today, the RPC scheduling mechanism is pretty simple: it execute requests in isolated thread-pools based on their priority. In the current implementation, all normal get/put requests are using the same pool. We'd like to add some per-user or per-region level isolation, so that a misbehaved user/region will not saturate the thread-pool and cause DoS to others easily. The idea is similar to FairScheduler in MR. The current scheduling code is not standalone and is mixed with others (Connection#processRequest). The issue is the first step to extract it to an interface, so that people are free to write and test their own implementations. This patch doesn't make it completely pluggable yet, as some parameters are pass from constructor. This is because HMaster and HRegionServer both use RpcServer and they have different thread-pool size config. Let me know if you have a solution to this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13762979#comment-13762979 ] stack commented on HBASE-8884: -- [~stepinto] I was reading rpc code so I could better review an incoming rpc patch and because I have notions I will likely never get to (see below). While reading, I was trying to write up documentation of how it all worked. This is where I ran into how opaque and convoluted its operation what w/ unused thread locals used passing messages and then the stuff added by this patch -- complications we can hopefully clean up in subsequent refactorings as you suggest. Do you have any pushback on my review comments? 'Pooling of buffers across requests' is a notion that rather than do data = ByteBuffer.allocate(dataLength); inside in a rpc Reader thread every time we get a new request, since we have read the total rpc size and we know its size, that instead we could go to a pool of buffers and ask it for a buffer that is of appropriate size. We'd check it out for the length of the request. We'd need to check it back in when done (likely good spot is at the tail of the Handler when it adds the response to the Responder queue). This could save us a bunch of allocations (and GC load, etc.). I think we could get away w/ this given how KeyValues are copied into the MSLAB when we add them to the MemStore (we'd have to figure what to do about those that are not copied; i.e. KeyValues that are large). If the above worked, we could then entertain making the pool be a pool of direct byte buffers (Downsides on DBBs are they take a while to allocate and their cleanup is unpreditable -- having them in a pool that we set up on server start w/ skirt some of these downsides; the copy from the socket channel to the DBB would be offheap making for more savings. If the memstore implementation was itself offheap but now I am into fantasy so will stop. Thanks Pluggable RpcScheduler -- Key: HBASE-8884 URL: https://issues.apache.org/jira/browse/HBASE-8884 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi Assignee: Chao Shi Fix For: 0.98.0 Attachments: hbase-8884.patch, hbase-8884-v2.patch, hbase-8884-v3.patch, hbase-8884-v4.patch, hbase-8884-v5.patch, hbase-8884-v6.patch, hbase-8884-v7.patch, hbase-8884-v8.patch Today, the RPC scheduling mechanism is pretty simple: it execute requests in isolated thread-pools based on their priority. In the current implementation, all normal get/put requests are using the same pool. We'd like to add some per-user or per-region level isolation, so that a misbehaved user/region will not saturate the thread-pool and cause DoS to others easily. The idea is similar to FairScheduler in MR. The current scheduling code is not standalone and is mixed with others (Connection#processRequest). The issue is the first step to extract it to an interface, so that people are free to write and test their own implementations. This patch doesn't make it completely pluggable yet, as some parameters are pass from constructor. This is because HMaster and HRegionServer both use RpcServer and they have different thread-pool size config. Let me know if you have a solution to this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13763140#comment-13763140 ] Chao Shi commented on HBASE-8884: - I haven't thought on the interface clearly. You idea sounds similar to using per-request memory pool in the old C days. I will try to do some refactor as you suggested (callable + remove thread local). Pluggable RpcScheduler -- Key: HBASE-8884 URL: https://issues.apache.org/jira/browse/HBASE-8884 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi Assignee: Chao Shi Fix For: 0.98.0 Attachments: hbase-8884.patch, hbase-8884-v2.patch, hbase-8884-v3.patch, hbase-8884-v4.patch, hbase-8884-v5.patch, hbase-8884-v6.patch, hbase-8884-v7.patch, hbase-8884-v8.patch Today, the RPC scheduling mechanism is pretty simple: it execute requests in isolated thread-pools based on their priority. In the current implementation, all normal get/put requests are using the same pool. We'd like to add some per-user or per-region level isolation, so that a misbehaved user/region will not saturate the thread-pool and cause DoS to others easily. The idea is similar to FairScheduler in MR. The current scheduling code is not standalone and is mixed with others (Connection#processRequest). The issue is the first step to extract it to an interface, so that people are free to write and test their own implementations. This patch doesn't make it completely pluggable yet, as some parameters are pass from constructor. This is because HMaster and HRegionServer both use RpcServer and they have different thread-pool size config. Let me know if you have a solution to this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13762706#comment-13762706 ] Chao Shi commented on HBASE-8884: - Hi stack, Thanks for your review comments. bq. Shouldn't MONITORED_RPC be in Handler rather than kept as a thread local in RpcServer? Handler could give it to the CallRunner rather than have it jump hoops to get at its stashed instance? The reason is to isolate logic of a RpcScheduler (implementation-specific) and RpcServer provides (common logic shared by all RpcScheduler implementations). Putting it into Handler is OK, but because that it has its own thread, it may not be convenient to incorporate with, for example ThreadPoolExecutor. I'm open for any better suggestions if you have. I think we can continue to do some refactor work to get it clean and easy for understeanding. Pluggable RpcScheduler -- Key: HBASE-8884 URL: https://issues.apache.org/jira/browse/HBASE-8884 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi Assignee: Chao Shi Fix For: 0.98.0 Attachments: hbase-8884.patch, hbase-8884-v2.patch, hbase-8884-v3.patch, hbase-8884-v4.patch, hbase-8884-v5.patch, hbase-8884-v6.patch, hbase-8884-v7.patch, hbase-8884-v8.patch Today, the RPC scheduling mechanism is pretty simple: it execute requests in isolated thread-pools based on their priority. In the current implementation, all normal get/put requests are using the same pool. We'd like to add some per-user or per-region level isolation, so that a misbehaved user/region will not saturate the thread-pool and cause DoS to others easily. The idea is similar to FairScheduler in MR. The current scheduling code is not standalone and is mixed with others (Connection#processRequest). The issue is the first step to extract it to an interface, so that people are free to write and test their own implementations. This patch doesn't make it completely pluggable yet, as some parameters are pass from constructor. This is because HMaster and HRegionServer both use RpcServer and they have different thread-pool size config. Let me know if you have a solution to this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13762717#comment-13762717 ] stack commented on HBASE-8884: -- As is, it is hard to follow what is going on. + Do we have to have a thread local up in the parent server class as the means of keeping context across the calls; can this not be in handler? A context it gives each Call? + Does CallRunner have to be a Runnable? You see how elsewhere in hbase we the notion of Callable? It seems to have much what you introduce here with the CallRunner? Would be good to have it all align. Agree more refactor would help. Thanks Chao Shi. I want to be able to add pooling of buffers across requests. At the moment it is difficult figuring how/where to insert. Thanks. Pluggable RpcScheduler -- Key: HBASE-8884 URL: https://issues.apache.org/jira/browse/HBASE-8884 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi Assignee: Chao Shi Fix For: 0.98.0 Attachments: hbase-8884.patch, hbase-8884-v2.patch, hbase-8884-v3.patch, hbase-8884-v4.patch, hbase-8884-v5.patch, hbase-8884-v6.patch, hbase-8884-v7.patch, hbase-8884-v8.patch Today, the RPC scheduling mechanism is pretty simple: it execute requests in isolated thread-pools based on their priority. In the current implementation, all normal get/put requests are using the same pool. We'd like to add some per-user or per-region level isolation, so that a misbehaved user/region will not saturate the thread-pool and cause DoS to others easily. The idea is similar to FairScheduler in MR. The current scheduling code is not standalone and is mixed with others (Connection#processRequest). The issue is the first step to extract it to an interface, so that people are free to write and test their own implementations. This patch doesn't make it completely pluggable yet, as some parameters are pass from constructor. This is because HMaster and HRegionServer both use RpcServer and they have different thread-pool size config. Let me know if you have a solution to this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13761070#comment-13761070 ] stack commented on HBASE-8884: -- Looking more at this patch, we effectively double the number of threads we have running a Call. A handler, which is a thread does nothing but start the CallRunner thread added in this patch. And if I read this patch right, we actually have lost control of the bounding on the number of threads we had running at any one time. The handler does takes from the queue, starts a thread, then loops and there is nothing to prevent it taking another CallRunner from the queue and starting it, and so on. I would suggest that CallRunner NOT be a thread (before this patch Handler did the work, now it creates a new thread to do the work). Could we fix it so RpcScheduler does not have to reach back into RpcServer to get the CallRunner type? Maybe CallRunner should be outside of RpcServer. RpcServer has too many inner classes as it is. Its a pity this stuff is in the Interface: + /** Retrieves length of the general queue for metrics. */ + int getGeneralQueueLength(); + + /** Retrieves length of the priority queue for metrics. */ + int getPriorityQueueLength(); + + /** Retrieves length of the replication queue for metrics. */ + int getReplicationQueueLength(); It hardcodes three queue types thereby constraining the types of scheduler that can be implemented. The gymnastics w/ MONITORED_RPC, the thread local, and how we get it in CallRunner to set state is 'interesting'. It says that we are setting the state on the handler but that is not so, right, if the handler is running the CallRunner...we are setting state of a CallRunner... and it looks like a Handler could be running multiple of them. What is this doing? Making it so handler state in the UI shows properly? Thanks. Pluggable RpcScheduler -- Key: HBASE-8884 URL: https://issues.apache.org/jira/browse/HBASE-8884 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi Assignee: Chao Shi Fix For: 0.98.0 Attachments: hbase-8884.patch, hbase-8884-v2.patch, hbase-8884-v3.patch, hbase-8884-v4.patch, hbase-8884-v5.patch, hbase-8884-v6.patch, hbase-8884-v7.patch, hbase-8884-v8.patch Today, the RPC scheduling mechanism is pretty simple: it execute requests in isolated thread-pools based on their priority. In the current implementation, all normal get/put requests are using the same pool. We'd like to add some per-user or per-region level isolation, so that a misbehaved user/region will not saturate the thread-pool and cause DoS to others easily. The idea is similar to FairScheduler in MR. The current scheduling code is not standalone and is mixed with others (Connection#processRequest). The issue is the first step to extract it to an interface, so that people are free to write and test their own implementations. This patch doesn't make it completely pluggable yet, as some parameters are pass from constructor. This is because HMaster and HRegionServer both use RpcServer and they have different thread-pool size config. Let me know if you have a solution to this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13761073#comment-13761073 ] stack commented on HBASE-8884: -- I took a looksee and my claim above this patch doubling the count of threads seems wrong. CallRunner is a Runnable. Handler#run calls CallRunner#run so the body of CallRunner#run becomes the body of Handler#run. So we should be fine. Pluggable RpcScheduler -- Key: HBASE-8884 URL: https://issues.apache.org/jira/browse/HBASE-8884 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi Assignee: Chao Shi Fix For: 0.98.0 Attachments: hbase-8884.patch, hbase-8884-v2.patch, hbase-8884-v3.patch, hbase-8884-v4.patch, hbase-8884-v5.patch, hbase-8884-v6.patch, hbase-8884-v7.patch, hbase-8884-v8.patch Today, the RPC scheduling mechanism is pretty simple: it execute requests in isolated thread-pools based on their priority. In the current implementation, all normal get/put requests are using the same pool. We'd like to add some per-user or per-region level isolation, so that a misbehaved user/region will not saturate the thread-pool and cause DoS to others easily. The idea is similar to FairScheduler in MR. The current scheduling code is not standalone and is mixed with others (Connection#processRequest). The issue is the first step to extract it to an interface, so that people are free to write and test their own implementations. This patch doesn't make it completely pluggable yet, as some parameters are pass from constructor. This is because HMaster and HRegionServer both use RpcServer and they have different thread-pool size config. Let me know if you have a solution to this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13761074#comment-13761074 ] stack commented on HBASE-8884: -- Yeah, so why is CallRunner a Runnable at all? As you can tell, I am having a little trouble following what is going on here. Thanks. Pluggable RpcScheduler -- Key: HBASE-8884 URL: https://issues.apache.org/jira/browse/HBASE-8884 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi Assignee: Chao Shi Fix For: 0.98.0 Attachments: hbase-8884.patch, hbase-8884-v2.patch, hbase-8884-v3.patch, hbase-8884-v4.patch, hbase-8884-v5.patch, hbase-8884-v6.patch, hbase-8884-v7.patch, hbase-8884-v8.patch Today, the RPC scheduling mechanism is pretty simple: it execute requests in isolated thread-pools based on their priority. In the current implementation, all normal get/put requests are using the same pool. We'd like to add some per-user or per-region level isolation, so that a misbehaved user/region will not saturate the thread-pool and cause DoS to others easily. The idea is similar to FairScheduler in MR. The current scheduling code is not standalone and is mixed with others (Connection#processRequest). The issue is the first step to extract it to an interface, so that people are free to write and test their own implementations. This patch doesn't make it completely pluggable yet, as some parameters are pass from constructor. This is because HMaster and HRegionServer both use RpcServer and they have different thread-pool size config. Let me know if you have a solution to this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13761078#comment-13761078 ] stack commented on HBASE-8884: -- Shouldn't MONITORED_RPC be in Handler rather than kept as a thread local in RpcServer? Handler could give it to the CallRunner rather than have it jump hoops to get at its stashed instance? I do not see what benefit CallRunner adds? It only seems to complicate for no apparent benefit (it confuses with its CallRunner name though it is not a Runnable even though it implements the Interface). Pluggable RpcScheduler -- Key: HBASE-8884 URL: https://issues.apache.org/jira/browse/HBASE-8884 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi Assignee: Chao Shi Fix For: 0.98.0 Attachments: hbase-8884.patch, hbase-8884-v2.patch, hbase-8884-v3.patch, hbase-8884-v4.patch, hbase-8884-v5.patch, hbase-8884-v6.patch, hbase-8884-v7.patch, hbase-8884-v8.patch Today, the RPC scheduling mechanism is pretty simple: it execute requests in isolated thread-pools based on their priority. In the current implementation, all normal get/put requests are using the same pool. We'd like to add some per-user or per-region level isolation, so that a misbehaved user/region will not saturate the thread-pool and cause DoS to others easily. The idea is similar to FairScheduler in MR. The current scheduling code is not standalone and is mixed with others (Connection#processRequest). The issue is the first step to extract it to an interface, so that people are free to write and test their own implementations. This patch doesn't make it completely pluggable yet, as some parameters are pass from constructor. This is because HMaster and HRegionServer both use RpcServer and they have different thread-pool size config. Let me know if you have a solution to this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13752828#comment-13752828 ] stack commented on HBASE-8884: -- I commented over in hbase-9101 [~stepinto] Pluggable RpcScheduler -- Key: HBASE-8884 URL: https://issues.apache.org/jira/browse/HBASE-8884 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi Assignee: Chao Shi Fix For: 0.98.0 Attachments: hbase-8884.patch, hbase-8884-v2.patch, hbase-8884-v3.patch, hbase-8884-v4.patch, hbase-8884-v5.patch, hbase-8884-v6.patch, hbase-8884-v7.patch, hbase-8884-v8.patch Today, the RPC scheduling mechanism is pretty simple: it execute requests in isolated thread-pools based on their priority. In the current implementation, all normal get/put requests are using the same pool. We'd like to add some per-user or per-region level isolation, so that a misbehaved user/region will not saturate the thread-pool and cause DoS to others easily. The idea is similar to FairScheduler in MR. The current scheduling code is not standalone and is mixed with others (Connection#processRequest). The issue is the first step to extract it to an interface, so that people are free to write and test their own implementations. This patch doesn't make it completely pluggable yet, as some parameters are pass from constructor. This is because HMaster and HRegionServer both use RpcServer and they have different thread-pool size config. Let me know if you have a solution to this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725073#comment-13725073 ] Chao Shi commented on HBASE-8884: - Hi stack, I opened another issue (HBASE-9101) and posted a patch to fix the issue you mentioned. Please have a look. Thanks! Pluggable RpcScheduler -- Key: HBASE-8884 URL: https://issues.apache.org/jira/browse/HBASE-8884 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi Assignee: Chao Shi Fix For: 0.98.0 Attachments: hbase-8884.patch, hbase-8884-v2.patch, hbase-8884-v3.patch, hbase-8884-v4.patch, hbase-8884-v5.patch, hbase-8884-v6.patch, hbase-8884-v7.patch, hbase-8884-v8.patch Today, the RPC scheduling mechanism is pretty simple: it execute requests in isolated thread-pools based on their priority. In the current implementation, all normal get/put requests are using the same pool. We'd like to add some per-user or per-region level isolation, so that a misbehaved user/region will not saturate the thread-pool and cause DoS to others easily. The idea is similar to FairScheduler in MR. The current scheduling code is not standalone and is mixed with others (Connection#processRequest). The issue is the first step to extract it to an interface, so that people are free to write and test their own implementations. This patch doesn't make it completely pluggable yet, as some parameters are pass from constructor. This is because HMaster and HRegionServer both use RpcServer and they have different thread-pool size config. Let me know if you have a solution to this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721833#comment-13721833 ] stack commented on HBASE-8884: -- The commit needs a release note. Pluggable RpcScheduler -- Key: HBASE-8884 URL: https://issues.apache.org/jira/browse/HBASE-8884 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi Assignee: Chao Shi Fix For: 0.98.0 Attachments: hbase-8884.patch, hbase-8884-v2.patch, hbase-8884-v3.patch, hbase-8884-v4.patch, hbase-8884-v5.patch, hbase-8884-v6.patch, hbase-8884-v7.patch, hbase-8884-v8.patch Today, the RPC scheduling mechanism is pretty simple: it execute requests in isolated thread-pools based on their priority. In the current implementation, all normal get/put requests are using the same pool. We'd like to add some per-user or per-region level isolation, so that a misbehaved user/region will not saturate the thread-pool and cause DoS to others easily. The idea is similar to FairScheduler in MR. The current scheduling code is not standalone and is mixed with others (Connection#processRequest). The issue is the first step to extract it to an interface, so that people are free to write and test their own implementations. This patch doesn't make it completely pluggable yet, as some parameters are pass from constructor. This is because HMaster and HRegionServer both use RpcServer and they have different thread-pool size config. Let me know if you have a solution to this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721839#comment-13721839 ] stack commented on HBASE-8884: -- I added a few comments up on rb. Pluggable RpcScheduler -- Key: HBASE-8884 URL: https://issues.apache.org/jira/browse/HBASE-8884 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi Assignee: Chao Shi Fix For: 0.98.0 Attachments: hbase-8884.patch, hbase-8884-v2.patch, hbase-8884-v3.patch, hbase-8884-v4.patch, hbase-8884-v5.patch, hbase-8884-v6.patch, hbase-8884-v7.patch, hbase-8884-v8.patch Today, the RPC scheduling mechanism is pretty simple: it execute requests in isolated thread-pools based on their priority. In the current implementation, all normal get/put requests are using the same pool. We'd like to add some per-user or per-region level isolation, so that a misbehaved user/region will not saturate the thread-pool and cause DoS to others easily. The idea is similar to FairScheduler in MR. The current scheduling code is not standalone and is mixed with others (Connection#processRequest). The issue is the first step to extract it to an interface, so that people are free to write and test their own implementations. This patch doesn't make it completely pluggable yet, as some parameters are pass from constructor. This is because HMaster and HRegionServer both use RpcServer and they have different thread-pool size config. Let me know if you have a solution to this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13712606#comment-13712606 ] Ted Yu commented on HBASE-8884: --- Integrated to trunk. Thanks for the patch, Chao. Thanks for the reviews. Pluggable RpcScheduler -- Key: HBASE-8884 URL: https://issues.apache.org/jira/browse/HBASE-8884 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi Assignee: Chao Shi Attachments: hbase-8884.patch, hbase-8884-v2.patch, hbase-8884-v3.patch, hbase-8884-v4.patch, hbase-8884-v5.patch, hbase-8884-v6.patch, hbase-8884-v7.patch, hbase-8884-v8.patch Today, the RPC scheduling mechanism is pretty simple: it execute requests in isolated thread-pools based on their priority. In the current implementation, all normal get/put requests are using the same pool. We'd like to add some per-user or per-region level isolation, so that a misbehaved user/region will not saturate the thread-pool and cause DoS to others easily. The idea is similar to FairScheduler in MR. The current scheduling code is not standalone and is mixed with others (Connection#processRequest). The issue is the first step to extract it to an interface, so that people are free to write and test their own implementations. This patch doesn't make it completely pluggable yet, as some parameters are pass from constructor. This is because HMaster and HRegionServer both use RpcServer and they have different thread-pool size config. Let me know if you have a solution to this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13712867#comment-13712867 ] Hudson commented on HBASE-8884: --- FAILURE: Integrated in HBase-TRUNK #4269 (See [https://builds.apache.org/job/HBase-TRUNK/4269/]) HBASE-8884 Pluggable RpcScheduler (Chao Shi) (tedyu: rev 1504584) * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerWrapperImpl.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcScheduler.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServerInterface.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestDelayedRpc.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestIPC.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestProtoBufRpc.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java Pluggable RpcScheduler -- Key: HBASE-8884 URL: https://issues.apache.org/jira/browse/HBASE-8884 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi Assignee: Chao Shi Fix For: 0.98.0 Attachments: hbase-8884.patch, hbase-8884-v2.patch, hbase-8884-v3.patch, hbase-8884-v4.patch, hbase-8884-v5.patch, hbase-8884-v6.patch, hbase-8884-v7.patch, hbase-8884-v8.patch Today, the RPC scheduling mechanism is pretty simple: it execute requests in isolated thread-pools based on their priority. In the current implementation, all normal get/put requests are using the same pool. We'd like to add some per-user or per-region level isolation, so that a misbehaved user/region will not saturate the thread-pool and cause DoS to others easily. The idea is similar to FairScheduler in MR. The current scheduling code is not standalone and is mixed with others (Connection#processRequest). The issue is the first step to extract it to an interface, so that people are free to write and test their own implementations. This patch doesn't make it completely pluggable yet, as some parameters are pass from constructor. This is because HMaster and HRegionServer both use RpcServer and they have different thread-pool size config. Let me know if you have a solution to this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13713201#comment-13713201 ] Hudson commented on HBASE-8884: --- SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #623 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/623/]) HBASE-8884 Pluggable RpcScheduler (Chao Shi) (tedyu: rev 1504584) * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerWrapperImpl.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcScheduler.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServerInterface.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestDelayedRpc.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestIPC.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestProtoBufRpc.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java Pluggable RpcScheduler -- Key: HBASE-8884 URL: https://issues.apache.org/jira/browse/HBASE-8884 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi Assignee: Chao Shi Fix For: 0.98.0 Attachments: hbase-8884.patch, hbase-8884-v2.patch, hbase-8884-v3.patch, hbase-8884-v4.patch, hbase-8884-v5.patch, hbase-8884-v6.patch, hbase-8884-v7.patch, hbase-8884-v8.patch Today, the RPC scheduling mechanism is pretty simple: it execute requests in isolated thread-pools based on their priority. In the current implementation, all normal get/put requests are using the same pool. We'd like to add some per-user or per-region level isolation, so that a misbehaved user/region will not saturate the thread-pool and cause DoS to others easily. The idea is similar to FairScheduler in MR. The current scheduling code is not standalone and is mixed with others (Connection#processRequest). The issue is the first step to extract it to an interface, so that people are free to write and test their own implementations. This patch doesn't make it completely pluggable yet, as some parameters are pass from constructor. This is because HMaster and HRegionServer both use RpcServer and they have different thread-pool size config. Let me know if you have a solution to this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13713257#comment-13713257 ] Chao Shi commented on HBASE-8884: - Good! Thanks all reviewers. Pluggable RpcScheduler -- Key: HBASE-8884 URL: https://issues.apache.org/jira/browse/HBASE-8884 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi Assignee: Chao Shi Fix For: 0.98.0 Attachments: hbase-8884.patch, hbase-8884-v2.patch, hbase-8884-v3.patch, hbase-8884-v4.patch, hbase-8884-v5.patch, hbase-8884-v6.patch, hbase-8884-v7.patch, hbase-8884-v8.patch Today, the RPC scheduling mechanism is pretty simple: it execute requests in isolated thread-pools based on their priority. In the current implementation, all normal get/put requests are using the same pool. We'd like to add some per-user or per-region level isolation, so that a misbehaved user/region will not saturate the thread-pool and cause DoS to others easily. The idea is similar to FairScheduler in MR. The current scheduling code is not standalone and is mixed with others (Connection#processRequest). The issue is the first step to extract it to an interface, so that people are free to write and test their own implementations. This patch doesn't make it completely pluggable yet, as some parameters are pass from constructor. This is because HMaster and HRegionServer both use RpcServer and they have different thread-pool size config. Let me know if you have a solution to this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[
https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13710907#comment-13710907
]
Hadoop QA commented on HBASE-8884:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12592739/hbase-8884-v8.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/6374//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6374//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6374//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6374//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6374//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6374//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6374//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6374//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6374//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6374//console
This message is automatically generated.
Pluggable RpcScheduler
--
Key: HBASE-8884
URL: https://issues.apache.org/jira/browse/HBASE-8884
Project: HBase
Issue Type: Improvement
Components: IPC/RPC
Reporter: Chao Shi
Assignee: Chao Shi
Attachments: hbase-8884.patch, hbase-8884-v2.patch,
hbase-8884-v3.patch, hbase-8884-v4.patch, hbase-8884-v5.patch,
hbase-8884-v6.patch, hbase-8884-v7.patch, hbase-8884-v8.patch
Today, the RPC scheduling mechanism is pretty simple: it execute requests in
isolated thread-pools based on their priority. In the current implementation,
all normal get/put requests are using the same pool. We'd like to add some
per-user or per-region level isolation, so that a misbehaved user/region will
not saturate the thread-pool and cause DoS to others easily. The idea is
similar to FairScheduler in MR. The current scheduling code is not standalone
and is mixed with others (Connection#processRequest). The issue is the first
step to extract it to an interface, so that people are free to write and test
their own implementations.
This patch doesn't make it completely pluggable yet, as some parameters are
pass from constructor. This is because HMaster and HRegionServer both use
RpcServer and they have different thread-pool size config. Let me know if you
have a solution to this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13711210#comment-13711210 ] Jesse Yates commented on HBASE-8884: +1 LGTM Pluggable RpcScheduler -- Key: HBASE-8884 URL: https://issues.apache.org/jira/browse/HBASE-8884 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi Assignee: Chao Shi Attachments: hbase-8884.patch, hbase-8884-v2.patch, hbase-8884-v3.patch, hbase-8884-v4.patch, hbase-8884-v5.patch, hbase-8884-v6.patch, hbase-8884-v7.patch, hbase-8884-v8.patch Today, the RPC scheduling mechanism is pretty simple: it execute requests in isolated thread-pools based on their priority. In the current implementation, all normal get/put requests are using the same pool. We'd like to add some per-user or per-region level isolation, so that a misbehaved user/region will not saturate the thread-pool and cause DoS to others easily. The idea is similar to FairScheduler in MR. The current scheduling code is not standalone and is mixed with others (Connection#processRequest). The issue is the first step to extract it to an interface, so that people are free to write and test their own implementations. This patch doesn't make it completely pluggable yet, as some parameters are pass from constructor. This is because HMaster and HRegionServer both use RpcServer and they have different thread-pool size config. Let me know if you have a solution to this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13711266#comment-13711266 ] Ted Yu commented on HBASE-8884: --- Will wait for one more day for additional comments before integration. Pluggable RpcScheduler -- Key: HBASE-8884 URL: https://issues.apache.org/jira/browse/HBASE-8884 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi Assignee: Chao Shi Attachments: hbase-8884.patch, hbase-8884-v2.patch, hbase-8884-v3.patch, hbase-8884-v4.patch, hbase-8884-v5.patch, hbase-8884-v6.patch, hbase-8884-v7.patch, hbase-8884-v8.patch Today, the RPC scheduling mechanism is pretty simple: it execute requests in isolated thread-pools based on their priority. In the current implementation, all normal get/put requests are using the same pool. We'd like to add some per-user or per-region level isolation, so that a misbehaved user/region will not saturate the thread-pool and cause DoS to others easily. The idea is similar to FairScheduler in MR. The current scheduling code is not standalone and is mixed with others (Connection#processRequest). The issue is the first step to extract it to an interface, so that people are free to write and test their own implementations. This patch doesn't make it completely pluggable yet, as some parameters are pass from constructor. This is because HMaster and HRegionServer both use RpcServer and they have different thread-pool size config. Let me know if you have a solution to this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708814#comment-13708814 ] Jesse Yates commented on HBASE-8884: this is kind of a big patch... mind putting it up on RB? It would be good to get some other eyes on it (maybe [~andrew.purt...@gmail.com]?). Good stuff so far from my first pass though :) Pluggable RpcScheduler -- Key: HBASE-8884 URL: https://issues.apache.org/jira/browse/HBASE-8884 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi Assignee: Chao Shi Attachments: hbase-8884.patch, hbase-8884-v2.patch, hbase-8884-v3.patch, hbase-8884-v4.patch, hbase-8884-v5.patch, hbase-8884-v6.patch, hbase-8884-v7.patch Today, the RPC scheduling mechanism is pretty simple: it execute requests in isolated thread-pools based on their priority. In the current implementation, all normal get/put requests are using the same pool. We'd like to add some per-user or per-region level isolation, so that a misbehaved user/region will not saturate the thread-pool and cause DoS to others easily. The idea is similar to FairScheduler in MR. The current scheduling code is not standalone and is mixed with others (Connection#processRequest). The issue is the first step to extract it to an interface, so that people are free to write and test their own implementations. This patch doesn't make it completely pluggable yet, as some parameters are pass from constructor. This is because HMaster and HRegionServer both use RpcServer and they have different thread-pool size config. Let me know if you have a solution to this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13709371#comment-13709371 ] Chao Shi commented on HBASE-8884: - bq. this is kind of a big patch... mind putting it up on RB? It would be good to get some other eyes on it (maybe Andrew Purtell?). Good stuff so far from my first pass though Done. Please have a look at https://reviews.apache.org/r/12572/. Pluggable RpcScheduler -- Key: HBASE-8884 URL: https://issues.apache.org/jira/browse/HBASE-8884 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi Assignee: Chao Shi Attachments: hbase-8884.patch, hbase-8884-v2.patch, hbase-8884-v3.patch, hbase-8884-v4.patch, hbase-8884-v5.patch, hbase-8884-v6.patch, hbase-8884-v7.patch Today, the RPC scheduling mechanism is pretty simple: it execute requests in isolated thread-pools based on their priority. In the current implementation, all normal get/put requests are using the same pool. We'd like to add some per-user or per-region level isolation, so that a misbehaved user/region will not saturate the thread-pool and cause DoS to others easily. The idea is similar to FairScheduler in MR. The current scheduling code is not standalone and is mixed with others (Connection#processRequest). The issue is the first step to extract it to an interface, so that people are free to write and test their own implementations. This patch doesn't make it completely pluggable yet, as some parameters are pass from constructor. This is because HMaster and HRegionServer both use RpcServer and they have different thread-pool size config. Let me know if you have a solution to this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[
https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13707871#comment-13707871
]
Ted Yu commented on HBASE-8884:
---
Patch v6 looks good. Minor comments below.
For RpcScheduler interface, please add javadoc for the method parameters.
For SimpleRpcScheduler:
{code}
+ * A scheduler that maintains isolated handler pools for high-priority and
replication
+ * requests.
{code}
There are actually 3 queues. Better make the above comment clearer.
RpcServer.CallRunner can be package private.
Pluggable RpcScheduler
--
Key: HBASE-8884
URL: https://issues.apache.org/jira/browse/HBASE-8884
Project: HBase
Issue Type: Improvement
Components: IPC/RPC
Reporter: Chao Shi
Assignee: Chao Shi
Attachments: hbase-8884.patch, hbase-8884-v2.patch,
hbase-8884-v3.patch, hbase-8884-v4.patch, hbase-8884-v5.patch,
hbase-8884-v6.patch
Today, the RPC scheduling mechanism is pretty simple: it execute requests in
isolated thread-pools based on their priority. In the current implementation,
all normal get/put requests are using the same pool. We'd like to add some
per-user or per-region level isolation, so that a misbehaved user/region will
not saturate the thread-pool and cause DoS to others easily. The idea is
similar to FairScheduler in MR. The current scheduling code is not standalone
and is mixed with others (Connection#processRequest). The issue is the first
step to extract it to an interface, so that people are free to write and test
their own implementations.
This patch doesn't make it completely pluggable yet, as some parameters are
pass from constructor. This is because HMaster and HRegionServer both use
RpcServer and they have different thread-pool size config. Let me know if you
have a solution to this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[
https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13707951#comment-13707951
]
Hadoop QA commented on HBASE-8884:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12592181/hbase-8884-v7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/6328//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6328//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6328//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6328//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6328//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6328//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6328//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6328//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6328//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6328//console
This message is automatically generated.
Pluggable RpcScheduler
--
Key: HBASE-8884
URL: https://issues.apache.org/jira/browse/HBASE-8884
Project: HBase
Issue Type: Improvement
Components: IPC/RPC
Reporter: Chao Shi
Assignee: Chao Shi
Attachments: hbase-8884.patch, hbase-8884-v2.patch,
hbase-8884-v3.patch, hbase-8884-v4.patch, hbase-8884-v5.patch,
hbase-8884-v6.patch, hbase-8884-v7.patch
Today, the RPC scheduling mechanism is pretty simple: it execute requests in
isolated thread-pools based on their priority. In the current implementation,
all normal get/put requests are using the same pool. We'd like to add some
per-user or per-region level isolation, so that a misbehaved user/region will
not saturate the thread-pool and cause DoS to others easily. The idea is
similar to FairScheduler in MR. The current scheduling code is not standalone
and is mixed with others (Connection#processRequest). The issue is the first
step to extract it to an interface, so that people are free to write and test
their own implementations.
This patch doesn't make it completely pluggable yet, as some parameters are
pass from constructor. This is because HMaster and HRegionServer both use
RpcServer and they have different thread-pool size config. Let me know if you
have a solution to this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13707954#comment-13707954 ] Ted Yu commented on HBASE-8884: --- I don't have other comments. Pluggable RpcScheduler -- Key: HBASE-8884 URL: https://issues.apache.org/jira/browse/HBASE-8884 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi Assignee: Chao Shi Attachments: hbase-8884.patch, hbase-8884-v2.patch, hbase-8884-v3.patch, hbase-8884-v4.patch, hbase-8884-v5.patch, hbase-8884-v6.patch, hbase-8884-v7.patch Today, the RPC scheduling mechanism is pretty simple: it execute requests in isolated thread-pools based on their priority. In the current implementation, all normal get/put requests are using the same pool. We'd like to add some per-user or per-region level isolation, so that a misbehaved user/region will not saturate the thread-pool and cause DoS to others easily. The idea is similar to FairScheduler in MR. The current scheduling code is not standalone and is mixed with others (Connection#processRequest). The issue is the first step to extract it to an interface, so that people are free to write and test their own implementations. This patch doesn't make it completely pluggable yet, as some parameters are pass from constructor. This is because HMaster and HRegionServer both use RpcServer and they have different thread-pool size config. Let me know if you have a solution to this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13706603#comment-13706603 ] Chao Shi commented on HBASE-8884: - Hi [~stack] and [~yuzhih...@gmail.com], any comments on this patch v6? Pluggable RpcScheduler -- Key: HBASE-8884 URL: https://issues.apache.org/jira/browse/HBASE-8884 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi Assignee: Chao Shi Attachments: hbase-8884.patch, hbase-8884-v2.patch, hbase-8884-v3.patch, hbase-8884-v4.patch, hbase-8884-v5.patch, hbase-8884-v6.patch Today, the RPC scheduling mechanism is pretty simple: it execute requests in isolated thread-pools based on their priority. In the current implementation, all normal get/put requests are using the same pool. We'd like to add some per-user or per-region level isolation, so that a misbehaved user/region will not saturate the thread-pool and cause DoS to others easily. The idea is similar to FairScheduler in MR. The current scheduling code is not standalone and is mixed with others (Connection#processRequest). The issue is the first step to extract it to an interface, so that people are free to write and test their own implementations. This patch doesn't make it completely pluggable yet, as some parameters are pass from constructor. This is because HMaster and HRegionServer both use RpcServer and they have different thread-pool size config. Let me know if you have a solution to this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[
https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702975#comment-13702975
]
Hadoop QA commented on HBASE-8884:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12591369/hbase-8884-v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 4
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1 release
audit warnings (more than the trunk's current 0 warnings).
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/6264//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6264//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6264//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6264//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6264//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6264//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6264//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6264//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6264//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6264//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6264//console
This message is automatically generated.
Pluggable RpcScheduler
--
Key: HBASE-8884
URL: https://issues.apache.org/jira/browse/HBASE-8884
Project: HBase
Issue Type: Improvement
Components: IPC/RPC
Reporter: Chao Shi
Assignee: Chao Shi
Attachments: hbase-8884.patch, hbase-8884-v2.patch,
hbase-8884-v3.patch
Today, the RPC scheduling mechanism is pretty simple: it execute requests in
isolated thread-pools based on their priority. In the current implementation,
all normal get/put requests are using the same pool. We'd like to add some
per-user or per-region level isolation, so that a misbehaved user/region will
not saturate the thread-pool and cause DoS to others easily. The idea is
similar to FairScheduler in MR. The current scheduling code is not standalone
and is mixed with others (Connection#processRequest). The issue is the first
step to extract it to an interface, so that people are free to write and test
their own implementations.
This patch doesn't make it completely pluggable yet, as some parameters are
pass from constructor. This is because HMaster and HRegionServer both use
RpcServer and they have different thread-pool size config. Let me know if you
have a solution to this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[
https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13703119#comment-13703119
]
Hadoop QA commented on HBASE-8884:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12591393/hbase-8884-v4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.zookeeper.lock.TestZKInterProcessReadWriteLock
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/6269//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6269//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6269//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6269//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6269//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6269//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6269//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6269//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6269//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6269//console
This message is automatically generated.
Pluggable RpcScheduler
--
Key: HBASE-8884
URL: https://issues.apache.org/jira/browse/HBASE-8884
Project: HBase
Issue Type: Improvement
Components: IPC/RPC
Reporter: Chao Shi
Assignee: Chao Shi
Attachments: hbase-8884.patch, hbase-8884-v2.patch,
hbase-8884-v3.patch, hbase-8884-v4.patch
Today, the RPC scheduling mechanism is pretty simple: it execute requests in
isolated thread-pools based on their priority. In the current implementation,
all normal get/put requests are using the same pool. We'd like to add some
per-user or per-region level isolation, so that a misbehaved user/region will
not saturate the thread-pool and cause DoS to others easily. The idea is
similar to FairScheduler in MR. The current scheduling code is not standalone
and is mixed with others (Connection#processRequest). The issue is the first
step to extract it to an interface, so that people are free to write and test
their own implementations.
This patch doesn't make it completely pluggable yet, as some parameters are
pass from constructor. This is because HMaster and HRegionServer both use
RpcServer and they have different thread-pool size config. Let me know if you
have a solution to this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[
https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13703413#comment-13703413
]
Hadoop QA commented on HBASE-8884:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12591441/hbase-8884-v5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/6271//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6271//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6271//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6271//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6271//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6271//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6271//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6271//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6271//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6271//console
This message is automatically generated.
Pluggable RpcScheduler
--
Key: HBASE-8884
URL: https://issues.apache.org/jira/browse/HBASE-8884
Project: HBase
Issue Type: Improvement
Components: IPC/RPC
Reporter: Chao Shi
Assignee: Chao Shi
Attachments: hbase-8884.patch, hbase-8884-v2.patch,
hbase-8884-v3.patch, hbase-8884-v4.patch, hbase-8884-v5.patch
Today, the RPC scheduling mechanism is pretty simple: it execute requests in
isolated thread-pools based on their priority. In the current implementation,
all normal get/put requests are using the same pool. We'd like to add some
per-user or per-region level isolation, so that a misbehaved user/region will
not saturate the thread-pool and cause DoS to others easily. The idea is
similar to FairScheduler in MR. The current scheduling code is not standalone
and is mixed with others (Connection#processRequest). The issue is the first
step to extract it to an interface, so that people are free to write and test
their own implementations.
This patch doesn't make it completely pluggable yet, as some parameters are
pass from constructor. This is because HMaster and HRegionServer both use
RpcServer and they have different thread-pool size config. Let me know if you
have a solution to this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[
https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13704011#comment-13704011
]
Ted Yu commented on HBASE-8884:
---
{code}
+public class SimpleRpcScheduler implements RpcServer.RpcScheduler {
{code}
Please add annotation for stability and audience
RpcScheduler can be extracted out of RpcServer, right ?
For MetricsHBaseServerWrapperImpl.java:
{code}
+|| !(this.server.getScheduler() instanceof SimpleRpcScheduler)) {
return 0;
}
-return server.callQueue.size();
+SimpleRpcScheduler scheduler = (SimpleRpcScheduler) server.getScheduler();
+return scheduler.callQueue.size();
{code}
Can retrieval of callQueue be declared in RpcScheduler so that we don't use the
casting above ?
Pluggable RpcScheduler
--
Key: HBASE-8884
URL: https://issues.apache.org/jira/browse/HBASE-8884
Project: HBase
Issue Type: Improvement
Components: IPC/RPC
Reporter: Chao Shi
Assignee: Chao Shi
Attachments: hbase-8884.patch, hbase-8884-v2.patch,
hbase-8884-v3.patch, hbase-8884-v4.patch, hbase-8884-v5.patch
Today, the RPC scheduling mechanism is pretty simple: it execute requests in
isolated thread-pools based on their priority. In the current implementation,
all normal get/put requests are using the same pool. We'd like to add some
per-user or per-region level isolation, so that a misbehaved user/region will
not saturate the thread-pool and cause DoS to others easily. The idea is
similar to FairScheduler in MR. The current scheduling code is not standalone
and is mixed with others (Connection#processRequest). The issue is the first
step to extract it to an interface, so that people are free to write and test
their own implementations.
This patch doesn't make it completely pluggable yet, as some parameters are
pass from constructor. This is because HMaster and HRegionServer both use
RpcServer and they have different thread-pool size config. Let me know if you
have a solution to this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[
https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13704203#comment-13704203
]
Hadoop QA commented on HBASE-8884:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12591589/hbase-8884-v6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/6286//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6286//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6286//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6286//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6286//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6286//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6286//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6286//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6286//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6286//console
This message is automatically generated.
Pluggable RpcScheduler
--
Key: HBASE-8884
URL: https://issues.apache.org/jira/browse/HBASE-8884
Project: HBase
Issue Type: Improvement
Components: IPC/RPC
Reporter: Chao Shi
Assignee: Chao Shi
Attachments: hbase-8884.patch, hbase-8884-v2.patch,
hbase-8884-v3.patch, hbase-8884-v4.patch, hbase-8884-v5.patch,
hbase-8884-v6.patch
Today, the RPC scheduling mechanism is pretty simple: it execute requests in
isolated thread-pools based on their priority. In the current implementation,
all normal get/put requests are using the same pool. We'd like to add some
per-user or per-region level isolation, so that a misbehaved user/region will
not saturate the thread-pool and cause DoS to others easily. The idea is
similar to FairScheduler in MR. The current scheduling code is not standalone
and is mixed with others (Connection#processRequest). The issue is the first
step to extract it to an interface, so that people are free to write and test
their own implementations.
This patch doesn't make it completely pluggable yet, as some parameters are
pass from constructor. This is because HMaster and HRegionServer both use
RpcServer and they have different thread-pool size config. Let me know if you
have a solution to this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13701381#comment-13701381 ] Ted Yu commented on HBASE-8884: --- Why is SimpleRpcScheduler embedded in RpcServer ? I would expect RpcScheduler and SimpleRpcScheduler to have their own classes. In trunk, there is already RpcServerInterface. Can you utilize this interface to accommodate different thread-pool size config ? Pluggable RpcScheduler -- Key: HBASE-8884 URL: https://issues.apache.org/jira/browse/HBASE-8884 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi Attachments: hbase-8884.patch Today, the RPC scheduling mechanism is pretty simple: it execute requests in isolated thread-pools based on their priority. In the current implementation, all normal get/put requests are using the same pool. We'd like to add some per-user or per-region level isolation, so that a misbehaved user/region will not saturate the thread-pool and cause DoS to others easily. The idea is similar to FairScheduler in MR. The current scheduling code is not standalone and is mixed with others (Connection#processRequest). The issue is the first step to extract it to an interface, so that people are free to write and test their own implementations. This patch doesn't make it completely pluggable yet, as some parameters are pass from constructor. This is because HMaster and HRegionServer both use RpcServer and they have different thread-pool size config. Let me know if you have a solution to this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[
https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13701433#comment-13701433
]
stack commented on HBASE-8884:
--
Thank you for digging in on this issue.
Related, our Elliott has suggested that client's just pass over in the request
the priority they think a call should have. Server-side, a simple
implementation would just do what the client asks. Later the server could
respect or not dependent on context at time of request.
Regards RpcServerInterface, that is a mess -- a vestige left over how rpc used
to work so would suggest you keep away from it (See TODO at top of the class).
Maybe get a Scheduler implementation from Server rather than do this kind of
stuff:
{code}
-return server.callQueue.size();
+RpcServer.SimpleRpcScheduler scheduler = (RpcServer.SimpleRpcScheduler)
server.scheduler;
+return scheduler.callQueue.size();
{code}
This is kinda ugly: MONITORED_RPC -- it being static final in the class. Won't
it be shared by master and regionserver when we run standalone w/ all threads
in the one jvm?
I am not clear why we are passing in Header here:
-Call(int id, final BlockingService service, final MethodDescriptor md,
Message param,
+Call(int id, final BlockingService service, final MethodDescriptor md,
RequestHeader header, Message param,
I'd think we would be done w/ header when running Calls? (Doesn't Connection
object have what you need?)
Maybe write up a one pager on what your plan is. It'd be easier to decipher
your intent and be able to help instead of reading a patch.
Why we need a CallTask? Isn't a Call a Task? Would CallRunner be a better
name? Or what is it doing? It is executing/running the call?
Can Scheduler be an Interface that has implementations that the RpcServer makes
use of? Hmm I see you do this already:
+ * An interface for RPC request scheduling algorithm.
+ */
+ public interface RpcScheduler {
+
+void start();
+void stop();
+
+/** Dispatches an RPC request asynchronously. */
+void dispatch(CallTask task) throws IOException, InterruptedException;
+ }
Yeah, just have a protected interface called Scheduler in the RPC package.
What can't we dispatch a Call? Why we have to take a CallTask? What is
missing from Call that you need? Just add it?
Yeah, move out the Scheduler implemetation into its own class (protected) in
the rpc package
Patch is good.
The difference between master and regionserver RpcServer are scheduler
parameters -- the pool sizes? Could you make a SchedulerConfiguration object
that needs to be filled out and passed to RpcServer constructor? Or just have
a masterschedulerconfiguration and a regionserverschedulerconfiguration that
you pass to the RpcServer on construction?
Pluggable RpcScheduler
--
Key: HBASE-8884
URL: https://issues.apache.org/jira/browse/HBASE-8884
Project: HBase
Issue Type: Improvement
Components: IPC/RPC
Reporter: Chao Shi
Attachments: hbase-8884.patch
Today, the RPC scheduling mechanism is pretty simple: it execute requests in
isolated thread-pools based on their priority. In the current implementation,
all normal get/put requests are using the same pool. We'd like to add some
per-user or per-region level isolation, so that a misbehaved user/region will
not saturate the thread-pool and cause DoS to others easily. The idea is
similar to FairScheduler in MR. The current scheduling code is not standalone
and is mixed with others (Connection#processRequest). The issue is the first
step to extract it to an interface, so that people are free to write and test
their own implementations.
This patch doesn't make it completely pluggable yet, as some parameters are
pass from constructor. This is because HMaster and HRegionServer both use
RpcServer and they have different thread-pool size config. Let me know if you
have a solution to this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[
https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13701488#comment-13701488
]
Chao Shi commented on HBASE-8884:
-
Thanks for reviewing this patch so quickly.
{quote}
Why is SimpleRpcScheduler embedded in RpcServer ?
I would expect RpcScheduler and SimpleRpcScheduler to have their own classes.
{quote}
{quote}
Yeah, move out the Scheduler implemetation into its own class (protected) in
the rpc package
{quote}
It has to access some member of RpcServer (for now, getQosLevel and
highPriorityLevel). It would be nice to have it a separate class when we have a
approach to pass configuration to a scheduler. I'm trying to work on that.
{quote}
I'd think we would be done w/ header when running Calls? (Doesn't Connection
object have what you need?)
{quote}
header is used by SimpleRpcScheduler to determine qos level. How to get it from
connection?
{quote}
This is kinda ugly: MONITORED_RPC – it being static final in the class. Won't
it be shared by master and regionserver when we run standalone w/ all threads
in the one jvm?
{quote}
I agree. Yes, it is shared. It is only used for holder a thread-local
MonitoredRpcHandler per handler thread. Any better ideas?
{quote}
Why we need a CallTask? Isn't a Call a Task? Would CallRunner be a better name?
Or what is it doing? It is executing/running the call?
{quote}
CallRunner is OK for me. It is a Call plus the processing logic (a runnable),
thus a scheduler can simply put it into a thread pool for execution. In
addition, I would prefer adding a interface for Call (which exposes getters
that a scheduler needs), so we can simply mock it up in testing.
{quote}
The difference between master and regionserver RpcServer are scheduler
parameters – the pool sizes? Could you make a SchedulerConfiguration object
that needs to be filled out and passed to RpcServer constructor? Or just have a
masterschedulerconfiguration and a regionserverschedulerconfiguration that you
pass to the RpcServer on construction?
{quote}
Good. Please note that besides pool size, a scheduler may have its own things
configurable (implementation dependent), which may e between master and
regionserver. In master, we can rename all hbase.master.rpc.scheduler.\* to
hbase.rpc.scheduler.\*, so the scheduler can read it.
{quote}
Maybe write up a one pager on what your plan is. It'd be easier to decipher
your intent and be able to help instead of reading a patch.
{quote}
In fact, I haven't got idea what a fair scheduler exactly is. It is hard to
have a algorithm fit all people/scenario. This patch is the very first step.
With it, so we are able to test our implementation without modifying code
inside RpcServer. I'm glad to discuss with you about the scheduler
implementation in another jira issue (we're getting some testing data from the
prototype scheduler).
I don't have access to my devbox during weekend. I will submit a new patch in
Mon or Tue. Thanks!
Pluggable RpcScheduler
--
Key: HBASE-8884
URL: https://issues.apache.org/jira/browse/HBASE-8884
Project: HBase
Issue Type: Improvement
Components: IPC/RPC
Reporter: Chao Shi
Attachments: hbase-8884.patch
Today, the RPC scheduling mechanism is pretty simple: it execute requests in
isolated thread-pools based on their priority. In the current implementation,
all normal get/put requests are using the same pool. We'd like to add some
per-user or per-region level isolation, so that a misbehaved user/region will
not saturate the thread-pool and cause DoS to others easily. The idea is
similar to FairScheduler in MR. The current scheduling code is not standalone
and is mixed with others (Connection#processRequest). The issue is the first
step to extract it to an interface, so that people are free to write and test
their own implementations.
This patch doesn't make it completely pluggable yet, as some parameters are
pass from constructor. This is because HMaster and HRegionServer both use
RpcServer and they have different thread-pool size config. Let me know if you
have a solution to this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
