[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276470#comment-16276470 ] Hadoop QA commented on HBASE-19290: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271207#comment-16271207 ] Hudson commented on HBASE-19290: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4138 (See

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271002#comment-16271002 ] Hudson commented on HBASE-19290: FAILURE: Integrated in Jenkins build HBase-2.0 #936 (See

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270571#comment-16270571 ] binlijin commented on HBASE-19290: -- Push to master and branch-2 > Reduce zk request when doing split

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270476#comment-16270476 ] Hadoop QA commented on HBASE-19290: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270247#comment-16270247 ] Hadoop QA commented on HBASE-19290: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269984#comment-16269984 ] binlijin commented on HBASE-19290: -- HBASE-19290.master.006.patch fix test failed. > Reduce zk request

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268618#comment-16268618 ] Hadoop QA commented on HBASE-19290: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268414#comment-16268414 ] binlijin commented on HBASE-19290: -- bq.Many apologies for the wasted time! (will try to make up for it

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268365#comment-16268365 ] Appy commented on HBASE-19290: -- (facepalm) Okay, I accept my thick headedness here. Many apologies for the

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268045#comment-16268045 ] binlijin commented on HBASE-19290: -- bq. Can you please walk me through a full case explaining need of

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267542#comment-16267542 ] Appy commented on HBASE-19290: -- +1 I still don't get it. Am starting to feel thick head on this one :). But

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263840#comment-16263840 ] Hadoop QA commented on HBASE-19290: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263808#comment-16263808 ] Ted Yu commented on HBASE-19290: bq. The WAL split speed was stable at 0.2TB/minute The above is

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263773#comment-16263773 ] Yu Li commented on HBASE-19290: --- Let me try to add more information: Once when upgrading the HDFS version,

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263760#comment-16263760 ] binlijin commented on HBASE-19290: -- {quote} The patch is adding throttling, so it's certainly changing

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263748#comment-16263748 ] binlijin commented on HBASE-19290: -- bq. The above example lasted for almost an hour. With the

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263742#comment-16263742 ] Ted Yu commented on HBASE-19290: Lijin: The above example lasted for almost an hour. With the patch,

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263729#comment-16263729 ] binlijin commented on HBASE-19290: -- We observe that when the cluster is big, regionserver issue too much

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263720#comment-16263720 ] binlijin commented on HBASE-19290: -- [~tedyu] bq. Assuming patch v3 is very close to the version you run

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263699#comment-16263699 ] binlijin commented on HBASE-19290: -- [~appy] bq. Yeah, they are definitely clear, but Jingyun Tian's

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263187#comment-16263187 ] Ted Yu commented on HBASE-19290: Reading the code with patch, it seems calculateAvailableSplitters()

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263094#comment-16263094 ] Appy commented on HBASE-19290: -- [~tedyu] what kind of performance numbers are you looking for? > Reduce zk

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263089#comment-16263089 ] Appy commented on HBASE-19290: -- {quote} bq. I can see it's doing that. The question really meant - why such

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263086#comment-16263086 ] Ted Yu commented on HBASE-19290: Lijin: Assuming patch v3 is very close to the version you run in the

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263051#comment-16263051 ] Appy commented on HBASE-19290: -- {quote} bq. Since expectedTasksPerRS only relay on availableRSs and

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261933#comment-16261933 ] binlijin commented on HBASE-19290: -- bq. Since expectedTasksPerRS only relay on availableRSs and

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261925#comment-16261925 ] binlijin commented on HBASE-19290: -- bq. I can see it's doing that. The question really meant - why such

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261597#comment-16261597 ] Appy commented on HBASE-19290: -- {quote} bq. So there are 2 available splitters, and one grabbed task, we

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16260541#comment-16260541 ] Jingyun Tian commented on HBASE-19290: -- {code} 343 int expectedTasksPerRS = (numTasks /

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16260258#comment-16260258 ] Hadoop QA commented on HBASE-19290: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16260196#comment-16260196 ] binlijin commented on HBASE-19290: -- bq. Why randomize? Can be constant? No particular reason for

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16260121#comment-16260121 ] Appy commented on HBASE-19290: -- bq. int sleepTime = RandomUtils.nextInt(0, 100) + 500; Why randomize? Can be

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16259078#comment-16259078 ] Hadoop QA commented on HBASE-19290: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258913#comment-16258913 ] binlijin commented on HBASE-19290: -- bq. Add javadoc for availableRSs. OK. bq. Grabed -> Grabbed OK. bq.

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258912#comment-16258912 ] binlijin commented on HBASE-19290: -- bq. Please add a short commit message explaining 'idea' of the

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16257863#comment-16257863 ] Ted Yu commented on HBASE-19290: {code} + * @param numTasks current total number of available tasks +

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16257575#comment-16257575 ] Hadoop QA commented on HBASE-19290: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16256632#comment-16256632 ] Appy commented on HBASE-19290: -- - Please add a short commit message explaining 'idea' of the change (makes

[jira] [Commented] (HBASE-19290) Reduce zk request when doing split log

[ https://issues.apache.org/jira/browse/HBASE-19290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16256585#comment-16256585 ] binlijin commented on HBASE-19290: -- The patch is effective and we have test it on our 2000+ nodes