[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539977#comment-13539977 ] Ted Yu commented on HBASE-5776: --- Integrated to trunk. Thanks for the patch, Liyin and lijin. Thanks for the review, Ram. HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: binlijin Fix For: 0.96.0 Attachments: 5776-trunk-V3.patch, 5776-trunk-V4.patch, 5776-trunk-V5.patch, 5776-trunk-V6.patch, 5776-trunk-V7.patch, ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.3.patch, ASF.LICENSE.NOT.GRANTED--D2775.4.patch, ASF.LICENSE.NOT.GRANTED--D2775.5.patch, HBASE-5776-trunk.patch, HBASE-5776-trunk-V2.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540047#comment-13540047 ] Hudson commented on HBASE-5776: --- Integrated in HBase-TRUNK #3660 (See [https://builds.apache.org/job/HBase-TRUNK/3660/]) HBASE-5776 HTableMultiplexer (Liyin, binlijin and Ted Yu) (Revision 1426221) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTable.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: binlijin Fix For: 0.96.0 Attachments: 5776-trunk-V3.patch, 5776-trunk-V4.patch, 5776-trunk-V5.patch, 5776-trunk-V6.patch, 5776-trunk-V7.patch, ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.3.patch, ASF.LICENSE.NOT.GRANTED--D2775.4.patch, ASF.LICENSE.NOT.GRANTED--D2775.5.patch, HBASE-5776-trunk.patch, HBASE-5776-trunk-V2.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540222#comment-13540222 ] Hudson commented on HBASE-5776: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #315 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/315/]) HBASE-5776 HTableMultiplexer (Liyin, binlijin and Ted Yu) (Revision 1426221) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTable.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: binlijin Fix For: 0.96.0 Attachments: 5776-trunk-V3.patch, 5776-trunk-V4.patch, 5776-trunk-V5.patch, 5776-trunk-V6.patch, 5776-trunk-V7.patch, ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.3.patch, ASF.LICENSE.NOT.GRANTED--D2775.4.patch, ASF.LICENSE.NOT.GRANTED--D2775.5.patch, HBASE-5776-trunk.patch, HBASE-5776-trunk-V2.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539793#comment-13539793 ] Ted Yu commented on HBASE-5776: --- Looks like Hadoop QA was not running test suite against patch: https://builds.apache.org/job/PreCommit-HBASE-Build/3707/parameters/ There is no effect on existing tests from patch v7 which is very close to patch v6 - there was no new modification to existing classes. HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: binlijin Fix For: 0.96.0 Attachments: 5776-trunk-V3.patch, 5776-trunk-V4.patch, 5776-trunk-V5.patch, 5776-trunk-V6.patch, 5776-trunk-V7.patch, ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.3.patch, ASF.LICENSE.NOT.GRANTED--D2775.4.patch, ASF.LICENSE.NOT.GRANTED--D2775.5.patch, HBASE-5776-trunk.patch, HBASE-5776-trunk-V2.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539832#comment-13539832 ] binlijin commented on HBASE-5776: - @Ted, thanks , the patch is fine, +1 on the Patch HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: binlijin Fix For: 0.96.0 Attachments: 5776-trunk-V3.patch, 5776-trunk-V4.patch, 5776-trunk-V5.patch, 5776-trunk-V6.patch, 5776-trunk-V7.patch, ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.3.patch, ASF.LICENSE.NOT.GRANTED--D2775.4.patch, ASF.LICENSE.NOT.GRANTED--D2775.5.patch, HBASE-5776-trunk.patch, HBASE-5776-trunk-V2.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[
https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539410#comment-13539410
]
ramkrishna.s.vasudevan commented on HBASE-5776:
---
{code}
queue = serverToBufferQueueMap.get(addr);
if (queue == null) {
// Create the queue for the new region server
queue = addNewRegionServer(addr, htable);
}
{code}
The above piece of code is in getBufferedQueue(). May be we can directly call
addNewRegionServer() only. Also addNewRegionServer is also synchronized.
HTableMultiplexer
--
Key: HBASE-5776
URL: https://issues.apache.org/jira/browse/HBASE-5776
Project: HBase
Issue Type: Improvement
Reporter: Liyin Tang
Assignee: binlijin
Fix For: 0.96.0
Attachments: 5776-trunk-V3.patch, 5776-trunk-V4.patch,
5776-trunk-V5.patch, 5776-trunk-V6.patch,
ASF.LICENSE.NOT.GRANTED--D2775.1.patch,
ASF.LICENSE.NOT.GRANTED--D2775.1.patch,
ASF.LICENSE.NOT.GRANTED--D2775.2.patch,
ASF.LICENSE.NOT.GRANTED--D2775.2.patch,
ASF.LICENSE.NOT.GRANTED--D2775.3.patch,
ASF.LICENSE.NOT.GRANTED--D2775.4.patch,
ASF.LICENSE.NOT.GRANTED--D2775.5.patch, HBASE-5776-trunk.patch,
HBASE-5776-trunk-V2.patch
There is a known issue in HBase client that single slow/dead region server
could slow down the multiput operations across all the region servers. So the
HBase client will be as slow as the slowest region server in the cluster.
To solve this problem, HTableMultiplexer will separate the multiput
submitting threads with the flush threads, which means the multiput operation
will be a nonblocking operation.
The submitting thread will shard all the puts into different queues based on
its destination region server and return immediately. The flush threads will
flush these puts from each queue to its destination region server.
Currently the HTableMultiplexer only supports the put operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539454#comment-13539454 ] binlijin commented on HBASE-5776: - @ramkrishna.s.vasudevan，the getBufferedQueue is not synchronized，so we can get the buffer queue fast and put the put request into the queue fast. The addNewRegionServer is synchronized, so if we switch to it directly ,the cost will be huge. HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: binlijin Fix For: 0.96.0 Attachments: 5776-trunk-V3.patch, 5776-trunk-V4.patch, 5776-trunk-V5.patch, 5776-trunk-V6.patch, ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.3.patch, ASF.LICENSE.NOT.GRANTED--D2775.4.patch, ASF.LICENSE.NOT.GRANTED--D2775.5.patch, HBASE-5776-trunk.patch, HBASE-5776-trunk-V2.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539277#comment-13539277 ] Ted Yu commented on HBASE-5776: --- Will integrate patch v6 if there is no further reviews. HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: binlijin Fix For: 0.96.0 Attachments: 5776-trunk-V3.patch, 5776-trunk-V4.patch, 5776-trunk-V5.patch, 5776-trunk-V6.patch, ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.3.patch, ASF.LICENSE.NOT.GRANTED--D2775.4.patch, ASF.LICENSE.NOT.GRANTED--D2775.5.patch, HBASE-5776-trunk.patch, HBASE-5776-trunk-V2.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539287#comment-13539287 ] ramkrishna.s.vasudevan commented on HBASE-5776: --- [~aoxiang] Did you compare this with asynchbase? Still i have not gone thro the patch closely. Some of the above comments says this one and asynchbase are similar. HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: binlijin Fix For: 0.96.0 Attachments: 5776-trunk-V3.patch, 5776-trunk-V4.patch, 5776-trunk-V5.patch, 5776-trunk-V6.patch, ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.3.patch, ASF.LICENSE.NOT.GRANTED--D2775.4.patch, ASF.LICENSE.NOT.GRANTED--D2775.5.patch, HBASE-5776-trunk.patch, HBASE-5776-trunk-V2.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539351#comment-13539351 ] binlijin commented on HBASE-5776: - @ramkrishna.s.vasudevan，no，i do not compare this with asynchbase，but this feature is simple compare with asynchbase. HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: binlijin Fix For: 0.96.0 Attachments: 5776-trunk-V3.patch, 5776-trunk-V4.patch, 5776-trunk-V5.patch, 5776-trunk-V6.patch, ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.3.patch, ASF.LICENSE.NOT.GRANTED--D2775.4.patch, ASF.LICENSE.NOT.GRANTED--D2775.5.patch, HBASE-5776-trunk.patch, HBASE-5776-trunk-V2.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[
https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538857#comment-13538857
]
Hadoop QA commented on HBASE-5776:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562236/5776-trunk-V3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 3
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 20 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestMultiParallel
org.apache.hadoop.hbase.client.TestHTableMultiplexer
{color:red}-1 core zombie tests{color}. There are zombie tests. See build
logs for details.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3670//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3670//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3670//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3670//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3670//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3670//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3670//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3670//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3670//console
This message is automatically generated.
HTableMultiplexer
--
Key: HBASE-5776
URL: https://issues.apache.org/jira/browse/HBASE-5776
Project: HBase
Issue Type: Improvement
Reporter: Liyin Tang
Assignee: binlijin
Fix For: 0.96.0
Attachments: 5776-trunk-V3.patch,
ASF.LICENSE.NOT.GRANTED--D2775.1.patch,
ASF.LICENSE.NOT.GRANTED--D2775.1.patch,
ASF.LICENSE.NOT.GRANTED--D2775.2.patch,
ASF.LICENSE.NOT.GRANTED--D2775.2.patch,
ASF.LICENSE.NOT.GRANTED--D2775.3.patch,
ASF.LICENSE.NOT.GRANTED--D2775.4.patch,
ASF.LICENSE.NOT.GRANTED--D2775.5.patch, HBASE-5776-trunk.patch,
HBASE-5776-trunk-V2.patch
There is a known issue in HBase client that single slow/dead region server
could slow down the multiput operations across all the region servers. So the
HBase client will be as slow as the slowest region server in the cluster.
To solve this problem, HTableMultiplexer will separate the multiput
submitting threads with the flush threads, which means the multiput operation
will be a nonblocking operation.
The submitting thread will shard all the puts into different queues based on
its destination region server and return immediately. The flush threads will
flush these puts from each queue to its destination region server.
Currently the HTableMultiplexer only supports the put operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[
https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538873#comment-13538873
]
Hadoop QA commented on HBASE-5776:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562239/5776-trunk-V4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 3
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 20 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestHTableMultiplexer
{color:red}-1 core zombie tests{color}. There are zombie tests. See build
logs for details.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3672//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3672//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3672//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3672//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3672//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3672//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3672//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3672//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3672//console
This message is automatically generated.
HTableMultiplexer
--
Key: HBASE-5776
URL: https://issues.apache.org/jira/browse/HBASE-5776
Project: HBase
Issue Type: Improvement
Reporter: Liyin Tang
Assignee: binlijin
Fix For: 0.96.0
Attachments: 5776-trunk-V3.patch, 5776-trunk-V4.patch,
ASF.LICENSE.NOT.GRANTED--D2775.1.patch,
ASF.LICENSE.NOT.GRANTED--D2775.1.patch,
ASF.LICENSE.NOT.GRANTED--D2775.2.patch,
ASF.LICENSE.NOT.GRANTED--D2775.2.patch,
ASF.LICENSE.NOT.GRANTED--D2775.3.patch,
ASF.LICENSE.NOT.GRANTED--D2775.4.patch,
ASF.LICENSE.NOT.GRANTED--D2775.5.patch, HBASE-5776-trunk.patch,
HBASE-5776-trunk-V2.patch
There is a known issue in HBase client that single slow/dead region server
could slow down the multiput operations across all the region servers. So the
HBase client will be as slow as the slowest region server in the cluster.
To solve this problem, HTableMultiplexer will separate the multiput
submitting threads with the flush threads, which means the multiput operation
will be a nonblocking operation.
The submitting thread will shard all the puts into different queues based on
its destination region server and return immediately. The flush threads will
flush these puts from each queue to its destination region server.
Currently the HTableMultiplexer only supports the put operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[
https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538911#comment-13538911
]
Hadoop QA commented on HBASE-5776:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562245/5776-trunk-V5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 3
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 20 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestHTableMultiplexer
org.apache.hadoop.hbase.ipc.TestDelayedRpc
org.apache.hadoop.hbase.TestZooKeeper
{color:red}-1 core zombie tests{color}. There are zombie tests. See build
logs for details.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3673//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3673//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3673//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3673//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3673//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3673//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3673//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3673//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3673//console
This message is automatically generated.
HTableMultiplexer
--
Key: HBASE-5776
URL: https://issues.apache.org/jira/browse/HBASE-5776
Project: HBase
Issue Type: Improvement
Reporter: Liyin Tang
Assignee: binlijin
Fix For: 0.96.0
Attachments: 5776-trunk-V3.patch, 5776-trunk-V4.patch,
5776-trunk-V5.patch, ASF.LICENSE.NOT.GRANTED--D2775.1.patch,
ASF.LICENSE.NOT.GRANTED--D2775.1.patch,
ASF.LICENSE.NOT.GRANTED--D2775.2.patch,
ASF.LICENSE.NOT.GRANTED--D2775.2.patch,
ASF.LICENSE.NOT.GRANTED--D2775.3.patch,
ASF.LICENSE.NOT.GRANTED--D2775.4.patch,
ASF.LICENSE.NOT.GRANTED--D2775.5.patch, HBASE-5776-trunk.patch,
HBASE-5776-trunk-V2.patch
There is a known issue in HBase client that single slow/dead region server
could slow down the multiput operations across all the region servers. So the
HBase client will be as slow as the slowest region server in the cluster.
To solve this problem, HTableMultiplexer will separate the multiput
submitting threads with the flush threads, which means the multiput operation
will be a nonblocking operation.
The submitting thread will shard all the puts into different queues based on
its destination region server and return immediately. The flush threads will
flush these puts from each queue to its destination region server.
Currently the HTableMultiplexer only supports the put operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[
https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538928#comment-13538928
]
Ted Yu commented on HBASE-5776:
---
From
https://builds.apache.org/job/PreCommit-HBASE-Build/3673/testReport/org.apache.hadoop.hbase.client/TestHTableMultiplexer/testHTableMultiplexer/:
{code}
2012-12-22 20:46:24,273 INFO [pool-1-thread-1]
client.TestHTableMultiplexer(93): Flushed put for @ iteration 1
...
2012-12-22 20:46:24,383 DEBUG
[HTableFlushWorker-asf011.sp2.ygridcore.net:35330-0]
client.HTableMultiplexer$HTableFlushWorker(553): Processed 1 put requests for
asf011.sp2.ygridcore.net:35330 and 0 failed, latency for this send: 208
{code}
We can see that client of HTableMultiplexer didn't get correct indication that
the put was flushed.
HTableMultiplexer
--
Key: HBASE-5776
URL: https://issues.apache.org/jira/browse/HBASE-5776
Project: HBase
Issue Type: Improvement
Reporter: Liyin Tang
Assignee: binlijin
Fix For: 0.96.0
Attachments: 5776-trunk-V3.patch, 5776-trunk-V4.patch,
5776-trunk-V5.patch, ASF.LICENSE.NOT.GRANTED--D2775.1.patch,
ASF.LICENSE.NOT.GRANTED--D2775.1.patch,
ASF.LICENSE.NOT.GRANTED--D2775.2.patch,
ASF.LICENSE.NOT.GRANTED--D2775.2.patch,
ASF.LICENSE.NOT.GRANTED--D2775.3.patch,
ASF.LICENSE.NOT.GRANTED--D2775.4.patch,
ASF.LICENSE.NOT.GRANTED--D2775.5.patch, HBASE-5776-trunk.patch,
HBASE-5776-trunk-V2.patch
There is a known issue in HBase client that single slow/dead region server
could slow down the multiput operations across all the region servers. So the
HBase client will be as slow as the slowest region server in the cluster.
To solve this problem, HTableMultiplexer will separate the multiput
submitting threads with the flush threads, which means the multiput operation
will be a nonblocking operation.
The submitting thread will shard all the puts into different queues based on
its destination region server and return immediately. The flush threads will
flush these puts from each queue to its destination region server.
Currently the HTableMultiplexer only supports the put operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[
https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538947#comment-13538947
]
Hadoop QA commented on HBASE-5776:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562251/5776-trunk-V6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 2
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:red}-1 core zombie tests{color}. There are zombie tests. See build
logs for details.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3674//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3674//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3674//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3674//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3674//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3674//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3674//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3674//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3674//console
This message is automatically generated.
HTableMultiplexer
--
Key: HBASE-5776
URL: https://issues.apache.org/jira/browse/HBASE-5776
Project: HBase
Issue Type: Improvement
Reporter: Liyin Tang
Assignee: binlijin
Fix For: 0.96.0
Attachments: 5776-trunk-V3.patch, 5776-trunk-V4.patch,
5776-trunk-V5.patch, 5776-trunk-V6.patch,
ASF.LICENSE.NOT.GRANTED--D2775.1.patch,
ASF.LICENSE.NOT.GRANTED--D2775.1.patch,
ASF.LICENSE.NOT.GRANTED--D2775.2.patch,
ASF.LICENSE.NOT.GRANTED--D2775.2.patch,
ASF.LICENSE.NOT.GRANTED--D2775.3.patch,
ASF.LICENSE.NOT.GRANTED--D2775.4.patch,
ASF.LICENSE.NOT.GRANTED--D2775.5.patch, HBASE-5776-trunk.patch,
HBASE-5776-trunk-V2.patch
There is a known issue in HBase client that single slow/dead region server
could slow down the multiput operations across all the region servers. So the
HBase client will be as slow as the slowest region server in the cluster.
To solve this problem, HTableMultiplexer will separate the multiput
submitting threads with the flush threads, which means the multiput operation
will be a nonblocking operation.
The submitting thread will shard all the puts into different queues based on
its destination region server and return immediately. The flush threads will
flush these puts from each queue to its destination region server.
Currently the HTableMultiplexer only supports the put operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538955#comment-13538955 ] Ted Yu commented on HBASE-5776: --- bq. [WARNING] javadoc: warning - No source files for package org.apache.hadoop.hbase.io.hfile.bucket The above was due to residue from application of HBASE-7404 patch. No javadoc warning after I removed the bucket directory. HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: binlijin Fix For: 0.96.0 Attachments: 5776-trunk-V3.patch, 5776-trunk-V4.patch, 5776-trunk-V5.patch, 5776-trunk-V6.patch, ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.3.patch, ASF.LICENSE.NOT.GRANTED--D2775.4.patch, ASF.LICENSE.NOT.GRANTED--D2775.5.patch, HBASE-5776-trunk.patch, HBASE-5776-trunk-V2.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538969#comment-13538969 ] binlijin commented on HBASE-5776: - @Ted, Thank you very much, I am +1 on the Patch v6. HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: binlijin Fix For: 0.96.0 Attachments: 5776-trunk-V3.patch, 5776-trunk-V4.patch, 5776-trunk-V5.patch, 5776-trunk-V6.patch, ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.3.patch, ASF.LICENSE.NOT.GRANTED--D2775.4.patch, ASF.LICENSE.NOT.GRANTED--D2775.5.patch, HBASE-5776-trunk.patch, HBASE-5776-trunk-V2.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13537759#comment-13537759 ] binlijin commented on HBASE-5776: - java.lang.NullPointerException at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:940) at org.apache.hadoop.hbase.client.TestHTableMultiplexer.testHTableMultiplexer(TestHTableMultiplexer.java:96) May be the put failed, I run the test local success. mvn test -P localTests -DHBasePatchProcess -Dtest=org.apache.hadoop.hbase.client.TestHTableMultiplexer HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: binlijin Attachments: ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.3.patch, ASF.LICENSE.NOT.GRANTED--D2775.4.patch, ASF.LICENSE.NOT.GRANTED--D2775.5.patch, HBASE-5776-trunk.patch, HBASE-5776-trunk-V2.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13537760#comment-13537760 ] binlijin commented on HBASE-5776: - Sorry，the put should be success, because there is no Exception, but get return no data, i don't known why. HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: binlijin Attachments: ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.3.patch, ASF.LICENSE.NOT.GRANTED--D2775.4.patch, ASF.LICENSE.NOT.GRANTED--D2775.5.patch, HBASE-5776-trunk.patch, HBASE-5776-trunk-V2.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13537648#comment-13537648 ] binlijin commented on HBASE-5776: - I have make a patch for the trunk HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.3.patch, ASF.LICENSE.NOT.GRANTED--D2775.4.patch, ASF.LICENSE.NOT.GRANTED--D2775.5.patch, HBASE-5776-trunk.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[
https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13537654#comment-13537654
]
Ted Yu commented on HBASE-5776:
---
{code}
+public class TestHTableMultiplexer {
{code}
Add annotation for test size. Is this a large test ?
{code}
+public class HTableMultiplexer {
{code}
Add annotation for audience and stability.
{code}
+ start = elapsed = System.currentTimeMillis();
{code}
Use EnvironmentEdgeManager above (and in other places).
After you attach patch v2, please press 'Submit Patch' so that Hadoop QA can
run test suite.
HTableMultiplexer
--
Key: HBASE-5776
URL: https://issues.apache.org/jira/browse/HBASE-5776
Project: HBase
Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang
Attachments: ASF.LICENSE.NOT.GRANTED--D2775.1.patch,
ASF.LICENSE.NOT.GRANTED--D2775.1.patch,
ASF.LICENSE.NOT.GRANTED--D2775.2.patch,
ASF.LICENSE.NOT.GRANTED--D2775.2.patch,
ASF.LICENSE.NOT.GRANTED--D2775.3.patch,
ASF.LICENSE.NOT.GRANTED--D2775.4.patch,
ASF.LICENSE.NOT.GRANTED--D2775.5.patch, HBASE-5776-trunk.patch
There is a known issue in HBase client that single slow/dead region server
could slow down the multiput operations across all the region servers. So the
HBase client will be as slow as the slowest region server in the cluster.
To solve this problem, HTableMultiplexer will separate the multiput
submitting threads with the flush threads, which means the multiput operation
will be a nonblocking operation.
The submitting thread will shard all the puts into different queues based on
its destination region server and return immediately. The flush threads will
flush these puts from each queue to its destination region server.
Currently the HTableMultiplexer only supports the put operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[
https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13537726#comment-13537726
]
Hadoop QA commented on HBASE-5776:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562042/HBASE-5776-trunk-V2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 29 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestFromClientSide
org.apache.hadoop.hbase.client.TestHTableMultiplexer
{color:red}-1 core zombie tests{color}. There are zombie tests. See build
logs for details.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3647//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3647//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3647//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3647//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3647//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3647//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3647//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3647//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3647//console
This message is automatically generated.
HTableMultiplexer
--
Key: HBASE-5776
URL: https://issues.apache.org/jira/browse/HBASE-5776
Project: HBase
Issue Type: Improvement
Reporter: Liyin Tang
Assignee: binlijin
Attachments: ASF.LICENSE.NOT.GRANTED--D2775.1.patch,
ASF.LICENSE.NOT.GRANTED--D2775.1.patch,
ASF.LICENSE.NOT.GRANTED--D2775.2.patch,
ASF.LICENSE.NOT.GRANTED--D2775.2.patch,
ASF.LICENSE.NOT.GRANTED--D2775.3.patch,
ASF.LICENSE.NOT.GRANTED--D2775.4.patch,
ASF.LICENSE.NOT.GRANTED--D2775.5.patch, HBASE-5776-trunk.patch,
HBASE-5776-trunk-V2.patch
There is a known issue in HBase client that single slow/dead region server
could slow down the multiput operations across all the region servers. So the
HBase client will be as slow as the slowest region server in the cluster.
To solve this problem, HTableMultiplexer will separate the multiput
submitting threads with the flush threads, which means the multiput operation
will be a nonblocking operation.
The submitting thread will shard all the puts into different queues based on
its destination region server and return immediately. The flush threads will
flush these puts from each queue to its destination region server.
Currently the HTableMultiplexer only supports the put operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13493477#comment-13493477 ] Otis Gospodnetic commented on HBASE-5776: - [~liangly] Any plans/ETA for getting this in trunk? HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.3.patch, ASF.LICENSE.NOT.GRANTED--D2775.4.patch, ASF.LICENSE.NOT.GRANTED--D2775.5.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13278600#comment-13278600 ] Phabricator commented on HBASE-5776: Kannan has accepted the revision [jira][89-fb][HBASE-5776] HTableMultiplexer. looks good Liyin. Remaining are cosmetic comments, hence accepting! INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1731 space after to src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:174 failed is unused src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:159 failed is unused src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:359 -1 - add space between - and 1. REVISION DETAIL https://reviews.facebook.net/D2775 BRANCH HBASE-5776 To: Kannan, Liyin Cc: JIRA, tedyu HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D2775.1.patch, D2775.1.patch, D2775.2.patch, D2775.2.patch, D2775.3.patch, D2775.4.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13279084#comment-13279084 ] Phabricator commented on HBASE-5776: Liyin has closed the revision [jira][89-fb][HBASE-5776] HTableMultiplexer. Close this 89-fb revision and will port to apache trunk soon. REVISION DETAIL https://reviews.facebook.net/D2775 To: Kannan, Liyin Cc: JIRA, tedyu HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D2775.1.patch, D2775.1.patch, D2775.2.patch, D2775.2.patch, D2775.3.patch, D2775.4.patch, D2775.5.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13274411#comment-13274411 ] Otis Gospodnetic commented on HBASE-5776: - I think asynchbase does the same thing - has a queue for each RS. Compatibility - don't know off hand, check its repo on github. HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D2775.1.patch, D2775.1.patch, D2775.2.patch, D2775.2.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13273631#comment-13273631 ] Otis Gospodnetic commented on HBASE-5776: - I read up on asynchbase yesterday. Doesn't asynchbase already solve the problem this issue is aimed at? See: http://search-hadoop.com/m/J6olJ11Idb http://search-hadoop.com/m/4fogb27wKWC HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D2775.1.patch, D2775.1.patch, D2775.2.patch, D2775.2.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[
https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13273652#comment-13273652
]
Liyin Tang commented on HBASE-5776:
---
Interesting :) Thanks @Otis for pointing the asynchbase out and I didn't know
that before.
I haven't read the asynchbase closely and it looks like both HTableMultiplexer
and asynchbase are attempting to provide async (deferred) api for applications.
However, I believe the HTableMulplixer mainly focuses on solving the slow
region server problem by partition the put requests based on its destination
region server {color:red} before deferring it {color} . So slow region server
will only hammer one shard of the requests.
BTW, is the asynchbase compatible with the current HBASE rpc protocol ?
HTableMultiplexer
--
Key: HBASE-5776
URL: https://issues.apache.org/jira/browse/HBASE-5776
Project: HBase
Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang
Attachments: D2775.1.patch, D2775.1.patch, D2775.2.patch,
D2775.2.patch
There is a known issue in HBase client that single slow/dead region server
could slow down the multiput operations across all the region servers. So the
HBase client will be as slow as the slowest region server in the cluster.
To solve this problem, HTableMultiplexer will separate the multiput
submitting threads with the flush threads, which means the multiput operation
will be a nonblocking operation.
The submitting thread will shard all the puts into different queues based on
its destination region server and return immediately. The flush threads will
flush these puts from each queue to its destination region server.
Currently the HTableMultiplexer only supports the put operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13270533#comment-13270533 ] Otis Gospodnetic commented on HBASE-5776: - What happens when some of the puts fail even after N attempts? Does the caller get notified that a failure happened and which puts failed? If not, how should one deal with such situations? What happens with puts that are in memory, not yet written to RS, and the app dies/stops for whatever reason. Are those puts lost? HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D2775.1.patch, D2775.1.patch, D2775.2.patch, D2775.2.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13270734#comment-13270734 ] Liyin Tang commented on HBASE-5776: --- @Otis, Good questions:) The motivation of HTableMultipler is NOT to replace the existing HTable but to provide a new alternative solutions for some high-throughput realtime applications. Usually, these applications can tolerate some data-loss during the region failover or other region-unavailable time instead of caching them in memory. In these cases, HTableMultiplexer tries its best to ONLY drop the data for these unavailable regions, instead of slowing down the overall multiput throughput which may cause more data dropping since the client fails to buffer any more incoming data across all the regions. Also, there is no call-back function if the put is finally failed after N attempts. And if client uses the multiput api in HTable, there is no such call back as well. However, we could have a way to config the HTableMultiplex to intelligently call back for some put operations. HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D2775.1.patch, D2775.1.patch, D2775.2.patch, D2775.2.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254901#comment-13254901 ] Kannan Muthukkaruppan commented on HBASE-5776: -- Ted: I think keeping Table in the name does not take way from the fact that it can support multiple tables. The multiplexer supports table operations-- so imo the name is ok as is. HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D2775.1.patch, D2775.1.patch, D2775.2.patch, D2775.2.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253768#comment-13253768 ] Liyin Tang commented on HBASE-5776: --- @Todd, The HTableMultiplexer is designed to process the puts requests across different tables. All the puts across the tables will be sharded into each different queues based on their destination region server. It will help to batch more puts for each region server before sending out the rpc request. HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D2775.1.patch, D2775.1.patch, D2775.2.patch, D2775.2.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253829#comment-13253829 ] Todd Lipcon commented on HBASE-5776: Gotcha. But is this a user-facing API? or would the HTables themselves write into an HTable multiplexer? It seems to me like this kind of behavior should happen automatically for any writes going into HBase, without expanding our API footprint. Is there some reason that that's impossible? HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D2775.1.patch, D2775.1.patch, D2775.2.patch, D2775.2.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253939#comment-13253939 ] Kannan Muthukkaruppan commented on HBASE-5776: -- Todd: This is API is user-facing. The HTables themselves aren't thread safe, so applications end up using a separate HTable per thread, and the per-RS batching happens only within the context of a single HTable request. The HTableMultiplexer was intended as a simple wrapper helper/utility that allows the app to get deferred put functionality, but with batching even the puts coming from different application threads in most effective manner. It also supports puts to multiple tables to be grouped in the same batch. Extending HTables which are not thread safe for this use case didn't make sense, and also painful from backward compat perspective. In this approach, a single instance of HTableMultiplexer can be use by all the app threads to shove puts, and let the multiplexer group commits per RS, and forward them along in batches. This non-blocking multiplexer API could get more sophisticated in future, e.g., supporting async gets (with callbacks and so on) and different config knobs (e.g., how much buffering memory to allow for all queues, retry/failure handling logic via callbacks, etc.). And maybe, in future, we can gradually deprecate the deferred put functionality in HTable, and recommend all the non-blocking use cases to go via the multiplexer. HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D2775.1.patch, D2775.1.patch, D2775.2.patch, D2775.2.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253976#comment-13253976 ] Zhihong Yu commented on HBASE-5776: --- Thanks for the explanation, Kannan. Since the Multiplexer isn't tied to any single table and it may support get's in the future, shall we remove the 'Table' in the class name ? HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D2775.1.patch, D2775.1.patch, D2775.2.patch, D2775.2.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13252966#comment-13252966 ] Phabricator commented on HBASE-5776: tedyu has commented on the revision [jira][89-fb][HBASE-5776] HTableMultiplexer. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1701 processSingleMultiPut would be a better name for this method. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1720 If this log is to be kept, please add method name. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1727 'and there is no succeeded put' - 'and there is no successful put' src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1730 Prepared - Prepare src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1702 Name this variable successfulPutCounter src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1738 Remove 'Set retry as true and' src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1745 Add a space between , and closing double quote. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1754 This is inconsistent with the handling on line 1760. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1766 'Catch the' - caught src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:1 License. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:20 javadoc for class, please. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:25 HServerAddress is deprecated in trunk. src/main/java/org/apache/hadoop/hbase/client/HTable.java:714 Package private access should be enough. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:63 A 'continue' after this line would allow you to omit else line below. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:66 loc.getHostnamePort() would allow you to identify the server. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:95 This can be package private since it is used by test only. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:122 flush worker is per region server, do we need poolID here ? src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:172 succeededCount - successfulCount src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:160 Give tmp a better name: how about putsUnderProcessing ? src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:198 ' put ' should precede 'requests' on line 199 src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:211 Should InterruptedException be handled specially ? src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:208 If this worker gets busy, there may not be sleeping. Should we introduce some counter for the above case? src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java:1 License, please. REVISION DETAIL https://reviews.facebook.net/D2775 HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D2775.1.patch, D2775.1.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13252965#comment-13252965 ] Phabricator commented on HBASE-5776: tedyu has commented on the revision [jira][89-fb][HBASE-5776] HTableMultiplexer. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1701 processSingleMultiPut would be a better name for this method. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1720 If this log is to be kept, please add method name. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1727 'and there is no succeeded put' - 'and there is no successful put' src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1730 Prepared - Prepare src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1702 Name this variable successfulPutCounter src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1738 Remove 'Set retry as true and' src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1745 Add a space between , and closing double quote. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1754 This is inconsistent with the handling on line 1760. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1766 'Catch the' - caught src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:1 License. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:20 javadoc for class, please. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:25 HServerAddress is deprecated in trunk. src/main/java/org/apache/hadoop/hbase/client/HTable.java:714 Package private access should be enough. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:63 A 'continue' after this line would allow you to omit else line below. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:66 loc.getHostnamePort() would allow you to identify the server. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:95 This can be package private since it is used by test only. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:122 flush worker is per region server, do we need poolID here ? src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:172 succeededCount - successfulCount src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:160 Give tmp a better name: how about putsUnderProcessing ? src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:198 ' put ' should precede 'requests' on line 199 src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:211 Should InterruptedException be handled specially ? src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:208 If this worker gets busy, there may not be sleeping. Should we introduce some counter for the above case? src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java:1 License, please. REVISION DETAIL https://reviews.facebook.net/D2775 HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D2775.1.patch, D2775.1.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253122#comment-13253122 ] Todd Lipcon commented on HBASE-5776: Can this be done without adding a new interface? eg can we make the existing queueing/flushing behavior of HTable use a threadpool? HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D2775.1.patch, D2775.1.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
