[jira] [Commented] (HBASE-6058) Use ZK 3.4 API 'multi' in bulk assignment
[ https://issues.apache.org/jira/browse/HBASE-6058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508963#comment-13508963 ] nkeywal commented on HBASE-6058: I've tested ZK#multi on assignment in master: no real improvement actually because we’re actually spending our time in the region server. We lower the load on ZK, but it would be visible only on a large cluster. As using multi would require to use ZK 3.4, it’s not compelling enough to do the move. Note that's because: - the master part I've changed is doing asynchronous writes, faster than synchronous writes - ZooKeeper does nothing else. On a large cluster, it would be more interesting. - there is no real bulk assign in the region server (i.e. a regionserver receives 20 regions simultaneously). So we don't need multi there today. Use ZK 3.4 API 'multi' in bulk assignment - Key: HBASE-6058 URL: https://issues.apache.org/jira/browse/HBASE-6058 Project: HBase Issue Type: Improvement Components: master, Zookeeper Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor We use async API today. This is already much much faster than the sync API. Still, it makes sense to use the 'multi' function: this will decrease the network zookeeper load at startup/rolling restart. On a 500 nodes cluster, we see 3 that 3 seconds are spent on updating ZK per bulk assignment. This should cut it in half (+ the benefits on the network/zk load). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6058) Use ZK 3.4 API 'multi' in bulk assignment
[ https://issues.apache.org/jira/browse/HBASE-6058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398282#comment-13398282 ] nkeywal commented on HBASE-6058: All the tests I made with multi were successful. But they were only tests :-). Note it's not totally trivial to use it in bulk assignment, because we have two levels of asynchronous calls (the callback calls another asynchronous function). So supporting both ZK version (with without multi) would be looking for issue imho. And fixing ZOOKEEPER-1381 would help on deployment, today we hang if we call multi on a 3.3 ZK server... Anyway, I will redo some perfo tests to see where we are now with the current implementation. Use ZK 3.4 API 'multi' in bulk assignment - Key: HBASE-6058 URL: https://issues.apache.org/jira/browse/HBASE-6058 Project: HBase Issue Type: Improvement Components: master, zookeeper Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor We use async API today. This is already much much faster than the sync API. Still, it makes sense to use the 'multi' function: this will decrease the network zookeeper load at startup/rolling restart. On a 500 nodes cluster, we see 3 that 3 seconds are spent on updating ZK per bulk assignment. This should cut it in half (+ the benefits on the network/zk load). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6058) Use ZK 3.4 API 'multi' in bulk assignment
[ https://issues.apache.org/jira/browse/HBASE-6058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398549#comment-13398549 ] Jesse Yates commented on HBASE-6058: bq. supporting both ZK version (with without multi) would be looking for issue imho Totally agree. bq. Anyway, I will redo some perfo tests to see where we are now with the current implementation Great! I'd love to see how big of a difference it makes. Use ZK 3.4 API 'multi' in bulk assignment - Key: HBASE-6058 URL: https://issues.apache.org/jira/browse/HBASE-6058 Project: HBase Issue Type: Improvement Components: master, zookeeper Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor We use async API today. This is already much much faster than the sync API. Still, it makes sense to use the 'multi' function: this will decrease the network zookeeper load at startup/rolling restart. On a 500 nodes cluster, we see 3 that 3 seconds are spent on updating ZK per bulk assignment. This should cut it in half (+ the benefits on the network/zk load). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6058) Use ZK 3.4 API 'multi' in bulk assignment
[ https://issues.apache.org/jira/browse/HBASE-6058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397917#comment-13397917 ] Jesse Yates commented on HBASE-6058: I think the consensus is that we don't move until 3.4 is considered stable. I've got a couple other patches up for other various zkutil things, for when we do move. Hopefully this happens around 0.96, especially since Patrick Hunt was saying that there haven't been any major issues around 3.4 (particularly around the multi stuff). Use ZK 3.4 API 'multi' in bulk assignment - Key: HBASE-6058 URL: https://issues.apache.org/jira/browse/HBASE-6058 Project: HBase Issue Type: Improvement Components: master, zookeeper Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor We use async API today. This is already much much faster than the sync API. Still, it makes sense to use the 'multi' function: this will decrease the network zookeeper load at startup/rolling restart. On a 500 nodes cluster, we see 3 that 3 seconds are spent on updating ZK per bulk assignment. This should cut it in half (+ the benefits on the network/zk load). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6058) Use ZK 3.4 API 'multi' in bulk assignment
[ https://issues.apache.org/jira/browse/HBASE-6058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397923#comment-13397923 ] Andrew Purtell commented on HBASE-6058: --- We run 3.4 in production and have had no issues. Multi has not had much use though, relative to the remainder of the ZK API. IMO, 0.96 is a good target for use of it, and we should get such usage in soon enough so we can have plenty of experience with it before cutting a release that uses it. Use ZK 3.4 API 'multi' in bulk assignment - Key: HBASE-6058 URL: https://issues.apache.org/jira/browse/HBASE-6058 Project: HBase Issue Type: Improvement Components: master, zookeeper Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor We use async API today. This is already much much faster than the sync API. Still, it makes sense to use the 'multi' function: this will decrease the network zookeeper load at startup/rolling restart. On a 500 nodes cluster, we see 3 that 3 seconds are spent on updating ZK per bulk assignment. This should cut it in half (+ the benefits on the network/zk load). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6058) Use ZK 3.4 API 'multi' in bulk assignment
[ https://issues.apache.org/jira/browse/HBASE-6058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13279559#comment-13279559 ] Zhihong Yu commented on HBASE-6058: --- There hasn't been consensus as to which HBase version would require zk 3.4 as the minimum supported version. Use ZK 3.4 API 'multi' in bulk assignment - Key: HBASE-6058 URL: https://issues.apache.org/jira/browse/HBASE-6058 Project: HBase Issue Type: Improvement Components: master, zookeeper Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor We use async API today. This is already much much faster than the sync API. Still, it makes sense to use the 'multi' function: this will decrease the network zookeeper load at startup/rolling restart. On a 500 nodes cluster, we see 3 that 3 seconds are spent on updating ZK per bulk assignment. This should cut it in half (+ the benefits on the network/zk load). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira