[jira] [Updated] (HBASE-14331) a single callQueue related improvements
[ https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-14331: Assignee: (was: Hiroshi Ikeda) Status: Open (was: Patch Available) > a single callQueue related improvements > --- > > Key: HBASE-14331 > URL: https://issues.apache.org/jira/browse/HBASE-14331 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, Performance, Scheduler >Reporter: Hiroshi Ikeda >Priority: Major > Attachments: BlockingQueuesPerformanceTestApp-output.pdf, > BlockingQueuesPerformanceTestApp-output.txt, > BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java, > HBASE-14331-V2.patch, HBASE-14331-V3.patch, HBASE-14331-V4.patch, > HBASE-14331-V5.patch, HBASE-14331-V6.patch, HBASE-14331-V6.patch, > HBASE-14331.patch, HBASE-14331.patch, SemaphoreBasedBlockingQueue.java, > SemaphoreBasedLinkedBlockingQueue.java, > SemaphoreBasedPriorityBlockingQueue.java > > > {{LinkedBlockingQueue}} well separates locks between the {{take}} method and > the {{put}} method, but not between takers, and not between putters. These > methods are implemented to take locks at the almost beginning of their logic. > HBASE-11355 introduces multiple call-queues to reduce such possible > congestion, but I doubt that it is required to stick to {{BlockingQueue}}. > There are the other shortcomings of using {{BlockingQueue}}. When using > multiple queues, since {{BlockingQueue}} blocks threads it is required to > prepare enough threads for each queue. It is possible that there is a queue > starving for threads while there is another queue where threads are idle. > Even if you can tune parameters to avoid such situations, the tuning is not > so trivial. > I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (HBASE-14331) a single callQueue related improvements
[ https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated HBASE-14331: -- Component/s: Scheduler > a single callQueue related improvements > --- > > Key: HBASE-14331 > URL: https://issues.apache.org/jira/browse/HBASE-14331 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, Performance, Scheduler >Reporter: Hiroshi Ikeda >Assignee: Hiroshi Ikeda >Priority: Major > Attachments: BlockingQueuesPerformanceTestApp-output.pdf, > BlockingQueuesPerformanceTestApp-output.txt, > BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java, > HBASE-14331-V2.patch, HBASE-14331-V3.patch, HBASE-14331-V4.patch, > HBASE-14331-V5.patch, HBASE-14331-V6.patch, HBASE-14331-V6.patch, > HBASE-14331.patch, HBASE-14331.patch, SemaphoreBasedBlockingQueue.java, > SemaphoreBasedLinkedBlockingQueue.java, > SemaphoreBasedPriorityBlockingQueue.java > > > {{LinkedBlockingQueue}} well separates locks between the {{take}} method and > the {{put}} method, but not between takers, and not between putters. These > methods are implemented to take locks at the almost beginning of their logic. > HBASE-11355 introduces multiple call-queues to reduce such possible > congestion, but I doubt that it is required to stick to {{BlockingQueue}}. > There are the other shortcomings of using {{BlockingQueue}}. When using > multiple queues, since {{BlockingQueue}} blocks threads it is required to > prepare enough threads for each queue. It is possible that there is a queue > starving for threads while there is another queue where threads are idle. > Even if you can tune parameters to avoid such situations, the tuning is not > so trivial. > I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-14331) a single callQueue related improvements
[ https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14331: -- Priority: Major (was: Minor) > a single callQueue related improvements > --- > > Key: HBASE-14331 > URL: https://issues.apache.org/jira/browse/HBASE-14331 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, Performance >Reporter: Hiroshi Ikeda >Assignee: Hiroshi Ikeda > Attachments: BlockingQueuesPerformanceTestApp-output.pdf, > BlockingQueuesPerformanceTestApp-output.txt, > BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java, > HBASE-14331-V2.patch, HBASE-14331-V3.patch, HBASE-14331-V4.patch, > HBASE-14331-V5.patch, HBASE-14331-V6.patch, HBASE-14331-V6.patch, > HBASE-14331.patch, HBASE-14331.patch, SemaphoreBasedBlockingQueue.java, > SemaphoreBasedLinkedBlockingQueue.java, > SemaphoreBasedPriorityBlockingQueue.java > > > {{LinkedBlockingQueue}} well separates locks between the {{take}} method and > the {{put}} method, but not between takers, and not between putters. These > methods are implemented to take locks at the almost beginning of their logic. > HBASE-11355 introduces multiple call-queues to reduce such possible > congestion, but I doubt that it is required to stick to {{BlockingQueue}}. > There are the other shortcomings of using {{BlockingQueue}}. When using > multiple queues, since {{BlockingQueue}} blocks threads it is required to > prepare enough threads for each queue. It is possible that there is a queue > starving for threads while there is another queue where threads are idle. > Even if you can tune parameters to avoid such situations, the tuning is not > so trivial. > I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14331) a single callQueue related improvements
[ https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hiroshi Ikeda updated HBASE-14331: -- Attachment: HBASE-14331-V6.patch Re-attached a patch to trigger QA run. > a single callQueue related improvements > --- > > Key: HBASE-14331 > URL: https://issues.apache.org/jira/browse/HBASE-14331 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, Performance >Reporter: Hiroshi Ikeda >Assignee: Hiroshi Ikeda >Priority: Minor > Attachments: BlockingQueuesPerformanceTestApp-output.pdf, > BlockingQueuesPerformanceTestApp-output.txt, > BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java, > HBASE-14331-V2.patch, HBASE-14331-V3.patch, HBASE-14331-V4.patch, > HBASE-14331-V5.patch, HBASE-14331-V6.patch, HBASE-14331-V6.patch, > HBASE-14331.patch, HBASE-14331.patch, SemaphoreBasedBlockingQueue.java, > SemaphoreBasedLinkedBlockingQueue.java, > SemaphoreBasedPriorityBlockingQueue.java > > > {{LinkedBlockingQueue}} well separates locks between the {{take}} method and > the {{put}} method, but not between takers, and not between putters. These > methods are implemented to take locks at the almost beginning of their logic. > HBASE-11355 introduces multiple call-queues to reduce such possible > congestion, but I doubt that it is required to stick to {{BlockingQueue}}. > There are the other shortcomings of using {{BlockingQueue}}. When using > multiple queues, since {{BlockingQueue}} blocks threads it is required to > prepare enough threads for each queue. It is possible that there is a queue > starving for threads while there is another queue where threads are idle. > Even if you can tune parameters to avoid such situations, the tuning is not > so trivial. > I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14331) a single callQueue related improvements
[ https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hiroshi Ikeda updated HBASE-14331: -- Attachment: HBASE-14331-V6.patch Attached a patch to trigger QA run, fixing a trivial typo of javadoc. My local environment is severe for building or testing. > a single callQueue related improvements > --- > > Key: HBASE-14331 > URL: https://issues.apache.org/jira/browse/HBASE-14331 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, Performance >Reporter: Hiroshi Ikeda >Assignee: Hiroshi Ikeda >Priority: Minor > Attachments: BlockingQueuesPerformanceTestApp-output.pdf, > BlockingQueuesPerformanceTestApp-output.txt, > BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java, > HBASE-14331-V2.patch, HBASE-14331-V3.patch, HBASE-14331-V4.patch, > HBASE-14331-V5.patch, HBASE-14331-V6.patch, HBASE-14331.patch, > HBASE-14331.patch, SemaphoreBasedBlockingQueue.java, > SemaphoreBasedLinkedBlockingQueue.java, > SemaphoreBasedPriorityBlockingQueue.java > > > {{LinkedBlockingQueue}} well separates locks between the {{take}} method and > the {{put}} method, but not between takers, and not between putters. These > methods are implemented to take locks at the almost beginning of their logic. > HBASE-11355 introduces multiple call-queues to reduce such possible > congestion, but I doubt that it is required to stick to {{BlockingQueue}}. > There are the other shortcomings of using {{BlockingQueue}}. When using > multiple queues, since {{BlockingQueue}} blocks threads it is required to > prepare enough threads for each queue. It is possible that there is a queue > starving for threads while there is another queue where threads are idle. > Even if you can tune parameters to avoid such situations, the tuning is not > so trivial. > I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14331) a single callQueue related improvements
[ https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hiroshi Ikeda updated HBASE-14331: -- Attachment: HBASE-14331-V4.patch Added a revised patch. - Added a new container class for an element with its deadline time. A comparator in a priority queue is repeatedly used for two elements until it finds a right insertion position, while I found the implementation of comparator for deadline queues was not so trivial (and it also has a little possibility to block threads via {{ConcurrentHashMap}}). I didn't change {{AnnotationReadingPriorityFunction.getDeadline}} because its intention is subtle (but I still think that is a bug...that even accesses a non-volatile field without synchronization). - Added some javadoc to introduced interfaces and classes around contracts of exceptions. > a single callQueue related improvements > --- > > Key: HBASE-14331 > URL: https://issues.apache.org/jira/browse/HBASE-14331 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, Performance >Reporter: Hiroshi Ikeda >Assignee: Hiroshi Ikeda >Priority: Minor > Attachments: BlockingQueuesPerformanceTestApp-output.pdf, > BlockingQueuesPerformanceTestApp-output.txt, > BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java, > HBASE-14331-V2.patch, HBASE-14331-V3.patch, HBASE-14331-V4.patch, > HBASE-14331.patch, HBASE-14331.patch, SemaphoreBasedBlockingQueue.java, > SemaphoreBasedLinkedBlockingQueue.java, > SemaphoreBasedPriorityBlockingQueue.java > > > {{LinkedBlockingQueue}} well separates locks between the {{take}} method and > the {{put}} method, but not between takers, and not between putters. These > methods are implemented to take locks at the almost beginning of their logic. > HBASE-11355 introduces multiple call-queues to reduce such possible > congestion, but I doubt that it is required to stick to {{BlockingQueue}}. > There are the other shortcomings of using {{BlockingQueue}}. When using > multiple queues, since {{BlockingQueue}} blocks threads it is required to > prepare enough threads for each queue. It is possible that there is a queue > starving for threads while there is another queue where threads are idle. > Even if you can tune parameters to avoid such situations, the tuning is not > so trivial. > I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14331) a single callQueue related improvements
[ https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hiroshi Ikeda updated HBASE-14331: -- Attachment: HBASE-14331-V5.patch Sorry, I missed some file. > a single callQueue related improvements > --- > > Key: HBASE-14331 > URL: https://issues.apache.org/jira/browse/HBASE-14331 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, Performance >Reporter: Hiroshi Ikeda >Assignee: Hiroshi Ikeda >Priority: Minor > Attachments: BlockingQueuesPerformanceTestApp-output.pdf, > BlockingQueuesPerformanceTestApp-output.txt, > BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java, > HBASE-14331-V2.patch, HBASE-14331-V3.patch, HBASE-14331-V4.patch, > HBASE-14331-V5.patch, HBASE-14331.patch, HBASE-14331.patch, > SemaphoreBasedBlockingQueue.java, SemaphoreBasedLinkedBlockingQueue.java, > SemaphoreBasedPriorityBlockingQueue.java > > > {{LinkedBlockingQueue}} well separates locks between the {{take}} method and > the {{put}} method, but not between takers, and not between putters. These > methods are implemented to take locks at the almost beginning of their logic. > HBASE-11355 introduces multiple call-queues to reduce such possible > congestion, but I doubt that it is required to stick to {{BlockingQueue}}. > There are the other shortcomings of using {{BlockingQueue}}. When using > multiple queues, since {{BlockingQueue}} blocks threads it is required to > prepare enough threads for each queue. It is possible that there is a queue > starving for threads while there is another queue where threads are idle. > Even if you can tune parameters to avoid such situations, the tuning is not > so trivial. > I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14331) a single callQueue related improvements
[ https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hiroshi Ikeda updated HBASE-14331: -- Attachment: HBASE-14331-V2.patch Added a revised patch. - Fixed a bug of using a wrong variable when starting handlers in RW-queue, and a bug of not adjusting the count of scan handlers. - Fixed a bug of the existing class for RW-queue. - Refactored. > a single callQueue related improvements > --- > > Key: HBASE-14331 > URL: https://issues.apache.org/jira/browse/HBASE-14331 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, Performance >Reporter: Hiroshi Ikeda >Assignee: Hiroshi Ikeda >Priority: Minor > Attachments: BlockingQueuesPerformanceTestApp-output.pdf, > BlockingQueuesPerformanceTestApp-output.txt, > BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java, > HBASE-14331-V2.patch, HBASE-14331.patch, HBASE-14331.patch, > SemaphoreBasedBlockingQueue.java, SemaphoreBasedLinkedBlockingQueue.java, > SemaphoreBasedPriorityBlockingQueue.java > > > {{LinkedBlockingQueue}} well separates locks between the {{take}} method and > the {{put}} method, but not between takers, and not between putters. These > methods are implemented to take locks at the almost beginning of their logic. > HBASE-11355 introduces multiple call-queues to reduce such possible > congestion, but I doubt that it is required to stick to {{BlockingQueue}}. > There are the other shortcomings of using {{BlockingQueue}}. When using > multiple queues, since {{BlockingQueue}} blocks threads it is required to > prepare enough threads for each queue. It is possible that there is a queue > starving for threads while there is another queue where threads are idle. > Even if you can tune parameters to avoid such situations, the tuning is not > so trivial. > I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14331) a single callQueue related improvements
[ https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hiroshi Ikeda updated HBASE-14331: -- Attachment: HBASE-14331-V3.patch Added a revised patch. Fixed a bug of reducing visibility of an existing protected method. BTW I'm not yet sure why zombie processes exist. > a single callQueue related improvements > --- > > Key: HBASE-14331 > URL: https://issues.apache.org/jira/browse/HBASE-14331 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, Performance >Reporter: Hiroshi Ikeda >Assignee: Hiroshi Ikeda >Priority: Minor > Attachments: BlockingQueuesPerformanceTestApp-output.pdf, > BlockingQueuesPerformanceTestApp-output.txt, > BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java, > HBASE-14331-V2.patch, HBASE-14331-V3.patch, HBASE-14331.patch, > HBASE-14331.patch, SemaphoreBasedBlockingQueue.java, > SemaphoreBasedLinkedBlockingQueue.java, > SemaphoreBasedPriorityBlockingQueue.java > > > {{LinkedBlockingQueue}} well separates locks between the {{take}} method and > the {{put}} method, but not between takers, and not between putters. These > methods are implemented to take locks at the almost beginning of their logic. > HBASE-11355 introduces multiple call-queues to reduce such possible > congestion, but I doubt that it is required to stick to {{BlockingQueue}}. > There are the other shortcomings of using {{BlockingQueue}}. When using > multiple queues, since {{BlockingQueue}} blocks threads it is required to > prepare enough threads for each queue. It is possible that there is a queue > starving for threads while there is another queue where threads are idle. > Even if you can tune parameters to avoid such situations, the tuning is not > so trivial. > I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14331) a single callQueue related improvements
[ https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hiroshi Ikeda updated HBASE-14331: -- Attachment: HBASE-14331.patch Added a patch. I did create a new interface {{CallQueue}} instead of adding broken {{BlockingQueue}} implementation classes. > a single callQueue related improvements > --- > > Key: HBASE-14331 > URL: https://issues.apache.org/jira/browse/HBASE-14331 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, Performance >Reporter: Hiroshi Ikeda >Priority: Minor > Attachments: BlockingQueuesPerformanceTestApp-output.pdf, > BlockingQueuesPerformanceTestApp-output.txt, > BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java, > HBASE-14331.patch, SemaphoreBasedBlockingQueue.java, > SemaphoreBasedLinkedBlockingQueue.java, > SemaphoreBasedPriorityBlockingQueue.java > > > {{LinkedBlockingQueue}} well separates locks between the {{take}} method and > the {{put}} method, but not between takers, and not between putters. These > methods are implemented to take locks at the almost beginning of their logic. > HBASE-11355 introduces multiple call-queues to reduce such possible > congestion, but I doubt that it is required to stick to {{BlockingQueue}}. > There are the other shortcomings of using {{BlockingQueue}}. When using > multiple queues, since {{BlockingQueue}} blocks threads it is required to > prepare enough threads for each queue. It is possible that there is a queue > starving for threads while there is another queue where threads are idle. > Even if you can tune parameters to avoid such situations, the tuning is not > so trivial. > I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14331) a single callQueue related improvements
[ https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hiroshi Ikeda updated HBASE-14331: -- Assignee: Hiroshi Ikeda Status: Patch Available (was: Open) > a single callQueue related improvements > --- > > Key: HBASE-14331 > URL: https://issues.apache.org/jira/browse/HBASE-14331 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, Performance >Reporter: Hiroshi Ikeda >Assignee: Hiroshi Ikeda >Priority: Minor > Attachments: BlockingQueuesPerformanceTestApp-output.pdf, > BlockingQueuesPerformanceTestApp-output.txt, > BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java, > HBASE-14331.patch, SemaphoreBasedBlockingQueue.java, > SemaphoreBasedLinkedBlockingQueue.java, > SemaphoreBasedPriorityBlockingQueue.java > > > {{LinkedBlockingQueue}} well separates locks between the {{take}} method and > the {{put}} method, but not between takers, and not between putters. These > methods are implemented to take locks at the almost beginning of their logic. > HBASE-11355 introduces multiple call-queues to reduce such possible > congestion, but I doubt that it is required to stick to {{BlockingQueue}}. > There are the other shortcomings of using {{BlockingQueue}}. When using > multiple queues, since {{BlockingQueue}} blocks threads it is required to > prepare enough threads for each queue. It is possible that there is a queue > starving for threads while there is another queue where threads are idle. > Even if you can tune parameters to avoid such situations, the tuning is not > so trivial. > I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14331) a single callQueue related improvements
[ https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14331: -- Attachment: HBASE-14331.patch > a single callQueue related improvements > --- > > Key: HBASE-14331 > URL: https://issues.apache.org/jira/browse/HBASE-14331 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, Performance >Reporter: Hiroshi Ikeda >Assignee: Hiroshi Ikeda >Priority: Minor > Attachments: BlockingQueuesPerformanceTestApp-output.pdf, > BlockingQueuesPerformanceTestApp-output.txt, > BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java, > HBASE-14331.patch, HBASE-14331.patch, SemaphoreBasedBlockingQueue.java, > SemaphoreBasedLinkedBlockingQueue.java, > SemaphoreBasedPriorityBlockingQueue.java > > > {{LinkedBlockingQueue}} well separates locks between the {{take}} method and > the {{put}} method, but not between takers, and not between putters. These > methods are implemented to take locks at the almost beginning of their logic. > HBASE-11355 introduces multiple call-queues to reduce such possible > congestion, but I doubt that it is required to stick to {{BlockingQueue}}. > There are the other shortcomings of using {{BlockingQueue}}. When using > multiple queues, since {{BlockingQueue}} blocks threads it is required to > prepare enough threads for each queue. It is possible that there is a queue > starving for threads while there is another queue where threads are idle. > Even if you can tune parameters to avoid such situations, the tuning is not > so trivial. > I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14331) a single callQueue related improvements
[ https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hiroshi Ikeda updated HBASE-14331: -- Attachment: BlockingQueuesPerformanceTestApp-output.pdf BlockingQueuesPerformanceTestApp-output.txt Added a result of the performance test in my environment, which has 8 cpu cores. It compares (1){{LinkedBlockingQueue}} in the standard Java library, (2){{BoundedPriorityBlockingQueue}} in HBase, and the newly implemented classes, (3){{SemaphoreBasedLinkedBlockingQueue}} and (4){{SemaphoreBasedPriorityBlockingQueue}}. I just changed the code of the test application to add (2){{BoundedPriorityBlockingQueue}}. The tasks are just executing {{Thread.yield}} and there should be excessively context switches, but the result says, the reading threads in the existing queues, (1){{LinkedBlockingQueue}} and (2){{BoundPriorityBlockingQueue}}, are rarely waken up. That means many reading threads are blocked in the queues. > a single callQueue related improvements > --- > > Key: HBASE-14331 > URL: https://issues.apache.org/jira/browse/HBASE-14331 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, Performance >Reporter: Hiroshi Ikeda >Priority: Minor > Attachments: BlockingQueuesPerformanceTestApp-output.pdf, > BlockingQueuesPerformanceTestApp-output.txt, > BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java, > SemaphoreBasedBlockingQueue.java, SemaphoreBasedLinkedBlockingQueue.java, > SemaphoreBasedPriorityBlockingQueue.java > > > {{LinkedBlockingQueue}} well separates locks between the {{take}} method and > the {{put}} method, but not between takers, and not between putters. These > methods are implemented to take locks at the almost beginning of their logic. > HBASE-11355 introduces multiple call-queues to reduce such possible > congestion, but I doubt that it is required to stick to {{BlockingQueue}}. > There are the other shortcomings of using {{BlockingQueue}}. When using > multiple queues, since {{BlockingQueue}} blocks threads it is required to > prepare enough threads for each queue. It is possible that there is a queue > starving for threads while there is another queue where threads are idle. > Even if you can tune parameters to avoid such situations, the tuning is not > so trivial. > I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14331) a single callQueue related improvements
[ https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hiroshi Ikeda updated HBASE-14331: -- Attachment: SemaphoreBasedBlockingQueue.java SemaphoreBasedLinkedBlockingQueue.java SemaphoreBasedPriorityBlockingQueue.java BlockingQueuesPerformanceTestApp.java Added trial semaphore based blocking classes, and a performance test application. Despite I said I doubted sticking to {{BlockingQueue}}, I tried to implement {{BlockingQueue}}. That is because the main logic around the call queue is just executing tasks with threads, and it seems preferable to replace it with {{ThreadPoolExecutor}}, which also accept only {{BlockingQueue}} as its task queue. But some methods can be just unsupported (around the {{Iterator.remove}} method) though the contract of the implemented interfaces allows it, and the implemented classes cannot be used with {{ThreadPoolExecutor}}. :-( a single callQueue related improvements --- Key: HBASE-14331 URL: https://issues.apache.org/jira/browse/HBASE-14331 Project: HBase Issue Type: Improvement Components: IPC/RPC, Performance Reporter: Hiroshi Ikeda Priority: Minor Attachments: BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java, SemaphoreBasedBlockingQueue.java, SemaphoreBasedLinkedBlockingQueue.java, SemaphoreBasedPriorityBlockingQueue.java {{LinkedBlockingQueue}} well separates locks between the {{take}} method and the {{put}} method, but not between takers, and not between putters. These methods are implemented to take locks at the almost beginning of their logic. HBASE-11355 introduces multiple call-queues to reduce such possible congestion, but I doubt that it is required to stick to {{BlockingQueue}}. There are the other shortcomings of using {{BlockingQueue}}. When using multiple queues, since {{BlockingQueue}} blocks threads it is required to prepare enough threads for each queue. It is possible that there is a queue starving for threads while there is another queue where threads are idle. Even if you can tune parameters to avoid such situations, the tuning is not so trivial. I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14331) a single callQueue related improvements
[ https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hiroshi Ikeda updated HBASE-14331: -- Attachment: CallQueuePerformanceTestApp.java Added a simple application to check performance between {{LinkedBlockingQueue}} and {{ConcurrentLinkedQueue}} with {{Semaphore}}. The callQueue here is not one in HBase but is defined inside as a mere simple interface. This application also show how many threads to read from the queue were simultaneously waken. In my environment using {{ConcurrentLinkedQueue}} is about 1.5-2 times faster, aside from whether the performance test practically makes sense. a single callQueue related improvements --- Key: HBASE-14331 URL: https://issues.apache.org/jira/browse/HBASE-14331 Project: HBase Issue Type: Improvement Components: IPC/RPC, Performance Reporter: Hiroshi Ikeda Priority: Minor Attachments: CallQueuePerformanceTestApp.java {{LinkedBlockingQueue}} well separates locks between the {{take}} method and the {{put}} method, but not between takers, and not between putters. These methods are implemented to take locks at the almost beginning of their logic. HBASE-11355 introduces multiple call-queues to reduce such possible congestion, but I doubt that it is required to stick to {{BlockingQueue}}. There are the other shortcomings of using {{BlockingQueue}}. When using multiple queues, since {{BlockingQueue}} blocks threads it is required to prepare enough threads for each queue. It is possible that there is a queue starving for threads while there is another queue where threads are idle. Even if you can tune parameters to avoid such situations, the tuning is not so trivial. I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)