[jira] [Commented] (HBASE-20610) Procedure V2 - Distributed Log Splitting
[ https://issues.apache.org/jira/browse/HBASE-20610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17144456#comment-17144456 ] Michael Stack commented on HBASE-20610: --- I've been running ITBLLs with this feature enabled on a small cluster of 10 nodes and it looks as good as what we have in place by default. I think we should enable this as our default in hbase3. Let me file a ticket. > Procedure V2 - Distributed Log Splitting > > > Key: HBASE-20610 > URL: https://issues.apache.org/jira/browse/HBASE-20610 > Project: HBase > Issue Type: Umbrella > Components: proc-v2 >Reporter: Guanghao Zhang >Assignee: Jingyun Tian >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.1 > > Attachments: HBASE-20610.master.001.patch > > > Now master and regionserver use zk to coordinate log split tasks. The split > log manager manages all log files which need to be scanned and split. Then > the split log manager places all the logs into the ZooKeeper splitWAL node > (/hbase/splitWAL) as tasks and monitors these task nodes and waits for them > to be processed. Each regionserver watch splitWAL znode and grab task when > node children changed. And regionserver does the work to split the logs. > Open this umbrella issue to move this "coordinate" work to use new procedure > v2 framework and reduce zk depencency. Plan to finish this before 3.0 > release. Any suggestions are welcomed. Thanks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-20610) Procedure V2 - Distributed Log Splitting
[ https://issues.apache.org/jira/browse/HBASE-20610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17138057#comment-17138057 ] Guanghao Zhang commented on HBASE-20610: Yes. This issue should be resolved. All sub-tasks finished. > Procedure V2 - Distributed Log Splitting > > > Key: HBASE-20610 > URL: https://issues.apache.org/jira/browse/HBASE-20610 > Project: HBase > Issue Type: Umbrella > Components: proc-v2 >Reporter: Guanghao Zhang >Assignee: Jingyun Tian >Priority: Major > Fix For: 3.0.0-alpha-1 > > Attachments: HBASE-20610.master.001.patch > > > Now master and regionserver use zk to coordinate log split tasks. The split > log manager manages all log files which need to be scanned and split. Then > the split log manager places all the logs into the ZooKeeper splitWAL node > (/hbase/splitWAL) as tasks and monitors these task nodes and waits for them > to be processed. Each regionserver watch splitWAL znode and grab task when > node children changed. And regionserver does the work to split the logs. > Open this umbrella issue to move this "coordinate" work to use new procedure > v2 framework and reduce zk depencency. Plan to finish this before 3.0 > release. Any suggestions are welcomed. Thanks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-20610) Procedure V2 - Distributed Log Splitting
[ https://issues.apache.org/jira/browse/HBASE-20610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17138045#comment-17138045 ] Jingyun Tian commented on HBASE-20610: -- [~stack] Looks good to me for the split manager part. Sorry that I didn't follow HBase for a while, I don't know the context of getProcName(). > Procedure V2 - Distributed Log Splitting > > > Key: HBASE-20610 > URL: https://issues.apache.org/jira/browse/HBASE-20610 > Project: HBase > Issue Type: Umbrella > Components: proc-v2 >Reporter: Guanghao Zhang >Assignee: Jingyun Tian >Priority: Major > Fix For: 3.0.0-alpha-1 > > Attachments: HBASE-20610.master.001.patch > > > Now master and regionserver use zk to coordinate log split tasks. The split > log manager manages all log files which need to be scanned and split. Then > the split log manager places all the logs into the ZooKeeper splitWAL node > (/hbase/splitWAL) as tasks and monitors these task nodes and waits for them > to be processed. Each regionserver watch splitWAL znode and grab task when > node children changed. And regionserver does the work to split the logs. > Open this umbrella issue to move this "coordinate" work to use new procedure > v2 framework and reduce zk depencency. Plan to finish this before 3.0 > release. Any suggestions are welcomed. Thanks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-20610) Procedure V2 - Distributed Log Splitting
[ https://issues.apache.org/jira/browse/HBASE-20610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17137850#comment-17137850 ] Michael Stack commented on HBASE-20610: --- [~tianjingyun] Should we resolve this? Seems to basically work (I added a subtask w/ some cleanup A review would be appreciated if you have a moment... thanks). > Procedure V2 - Distributed Log Splitting > > > Key: HBASE-20610 > URL: https://issues.apache.org/jira/browse/HBASE-20610 > Project: HBase > Issue Type: Umbrella > Components: proc-v2 >Reporter: Guanghao Zhang >Priority: Major > Fix For: 3.0.0-alpha-1 > > Attachments: HBASE-20610.master.001.patch > > > Now master and regionserver use zk to coordinate log split tasks. The split > log manager manages all log files which need to be scanned and split. Then > the split log manager places all the logs into the ZooKeeper splitWAL node > (/hbase/splitWAL) as tasks and monitors these task nodes and waits for them > to be processed. Each regionserver watch splitWAL znode and grab task when > node children changed. And regionserver does the work to split the logs. > Open this umbrella issue to move this "coordinate" work to use new procedure > v2 framework and reduce zk depencency. Plan to finish this before 3.0 > release. Any suggestions are welcomed. Thanks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-20610) Procedure V2 - Distributed Log Splitting
[ https://issues.apache.org/jira/browse/HBASE-20610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737033#comment-16737033 ] Jingyun Tian commented on HBASE-20610: -- [~busbey] The doc is updated. Please check it out. > Procedure V2 - Distributed Log Splitting > > > Key: HBASE-20610 > URL: https://issues.apache.org/jira/browse/HBASE-20610 > Project: HBase > Issue Type: Umbrella > Components: proc-v2 >Reporter: Guanghao Zhang >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20610.master.001.patch > > > Now master and regionserver use zk to coordinate log split tasks. The split > log manager manages all log files which need to be scanned and split. Then > the split log manager places all the logs into the ZooKeeper splitWAL node > (/hbase/splitWAL) as tasks and monitors these task nodes and waits for them > to be processed. Each regionserver watch splitWAL znode and grab task when > node children changed. And regionserver does the work to split the logs. > Open this umbrella issue to move this "coordinate" work to use new procedure > v2 framework and reduce zk depencency. Plan to finish this before 3.0 > release. Any suggestions are welcomed. Thanks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20610) Procedure V2 - Distributed Log Splitting
[ https://issues.apache.org/jira/browse/HBASE-20610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736121#comment-16736121 ] Sean Busbey commented on HBASE-20610: - please add a section to the design doc describing how upgrades will work. > Procedure V2 - Distributed Log Splitting > > > Key: HBASE-20610 > URL: https://issues.apache.org/jira/browse/HBASE-20610 > Project: HBase > Issue Type: Umbrella > Components: proc-v2 >Reporter: Guanghao Zhang >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20610.master.001.patch > > > Now master and regionserver use zk to coordinate log split tasks. The split > log manager manages all log files which need to be scanned and split. Then > the split log manager places all the logs into the ZooKeeper splitWAL node > (/hbase/splitWAL) as tasks and monitors these task nodes and waits for them > to be processed. Each regionserver watch splitWAL znode and grab task when > node children changed. And regionserver does the work to split the logs. > Open this umbrella issue to move this "coordinate" work to use new procedure > v2 framework and reduce zk depencency. Plan to finish this before 3.0 > release. Any suggestions are welcomed. Thanks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20610) Procedure V2 - Distributed Log Splitting
[ https://issues.apache.org/jira/browse/HBASE-20610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16712761#comment-16712761 ] Jingyun Tian commented on HBASE-20610: -- [~Apache9] [~stack] [~zghaobac] Design doc is updated. Please check it out if you have time. > Procedure V2 - Distributed Log Splitting > > > Key: HBASE-20610 > URL: https://issues.apache.org/jira/browse/HBASE-20610 > Project: HBase > Issue Type: Umbrella > Components: proc-v2 >Reporter: Guanghao Zhang >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20610.master.001.patch > > > Now master and regionserver use zk to coordinate log split tasks. The split > log manager manages all log files which need to be scanned and split. Then > the split log manager places all the logs into the ZooKeeper splitWAL node > (/hbase/splitWAL) as tasks and monitors these task nodes and waits for them > to be processed. Each regionserver watch splitWAL znode and grab task when > node children changed. And regionserver does the work to split the logs. > Open this umbrella issue to move this "coordinate" work to use new procedure > v2 framework and reduce zk depencency. Plan to finish this before 3.0 > release. Any suggestions are welcomed. Thanks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20610) Procedure V2 - Distributed Log Splitting
[ https://issues.apache.org/jira/browse/HBASE-20610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709675#comment-16709675 ] stack commented on HBASE-20610: --- [~tianjingyun] Will do sir. Nice-looking feature! > Procedure V2 - Distributed Log Splitting > > > Key: HBASE-20610 > URL: https://issues.apache.org/jira/browse/HBASE-20610 > Project: HBase > Issue Type: Umbrella > Components: proc-v2 >Reporter: Guanghao Zhang >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20610.master.001.patch > > > Now master and regionserver use zk to coordinate log split tasks. The split > log manager manages all log files which need to be scanned and split. Then > the split log manager places all the logs into the ZooKeeper splitWAL node > (/hbase/splitWAL) as tasks and monitors these task nodes and waits for them > to be processed. Each regionserver watch splitWAL znode and grab task when > node children changed. And regionserver does the work to split the logs. > Open this umbrella issue to move this "coordinate" work to use new procedure > v2 framework and reduce zk depencency. Plan to finish this before 3.0 > release. Any suggestions are welcomed. Thanks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20610) Procedure V2 - Distributed Log Splitting
[ https://issues.apache.org/jira/browse/HBASE-20610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709580#comment-16709580 ] Jingyun Tian commented on HBASE-20610: -- [~stack] Thanks for your review. But my current implementation has some differences with the doc now. I'm still working on the patch and I will update the doc this week. Please help me review then. > Procedure V2 - Distributed Log Splitting > > > Key: HBASE-20610 > URL: https://issues.apache.org/jira/browse/HBASE-20610 > Project: HBase > Issue Type: Umbrella > Components: proc-v2 >Reporter: Guanghao Zhang >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20610.master.001.patch > > > Now master and regionserver use zk to coordinate log split tasks. The split > log manager manages all log files which need to be scanned and split. Then > the split log manager places all the logs into the ZooKeeper splitWAL node > (/hbase/splitWAL) as tasks and monitors these task nodes and waits for them > to be processed. Each regionserver watch splitWAL znode and grab task when > node children changed. And regionserver does the work to split the logs. > Open this umbrella issue to move this "coordinate" work to use new procedure > v2 framework and reduce zk depencency. Plan to finish this before 3.0 > release. Any suggestions are welcomed. Thanks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20610) Procedure V2 - Distributed Log Splitting
[ https://issues.apache.org/jira/browse/HBASE-20610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16707570#comment-16707570 ] stack commented on HBASE-20610: --- Outline design looks great. Left a few comments. Thanks [~tianjingyun] > Procedure V2 - Distributed Log Splitting > > > Key: HBASE-20610 > URL: https://issues.apache.org/jira/browse/HBASE-20610 > Project: HBase > Issue Type: Umbrella > Components: proc-v2 >Reporter: Guanghao Zhang >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20610.master.001.patch > > > Now master and regionserver use zk to coordinate log split tasks. The split > log manager manages all log files which need to be scanned and split. Then > the split log manager places all the logs into the ZooKeeper splitWAL node > (/hbase/splitWAL) as tasks and monitors these task nodes and waits for them > to be processed. Each regionserver watch splitWAL znode and grab task when > node children changed. And regionserver does the work to split the logs. > Open this umbrella issue to move this "coordinate" work to use new procedure > v2 framework and reduce zk depencency. Plan to finish this before 3.0 > release. Any suggestions are welcomed. Thanks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20610) Procedure V2 - Distributed Log Splitting
[ https://issues.apache.org/jira/browse/HBASE-20610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16704568#comment-16704568 ] Jingyun Tian commented on HBASE-20610: -- Upload a poc of this issue. It's a draft right now. Please feel free to comment. [~Apache9] > Procedure V2 - Distributed Log Splitting > > > Key: HBASE-20610 > URL: https://issues.apache.org/jira/browse/HBASE-20610 > Project: HBase > Issue Type: Umbrella > Components: proc-v2 >Reporter: Guanghao Zhang >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20610.master.001.patch > > > Now master and regionserver use zk to coordinate log split tasks. The split > log manager manages all log files which need to be scanned and split. Then > the split log manager places all the logs into the ZooKeeper splitWAL node > (/hbase/splitWAL) as tasks and monitors these task nodes and waits for them > to be processed. Each regionserver watch splitWAL znode and grab task when > node children changed. And regionserver does the work to split the logs. > Open this umbrella issue to move this "coordinate" work to use new procedure > v2 framework and reduce zk depencency. Plan to finish this before 3.0 > release. Any suggestions are welcomed. Thanks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20610) Procedure V2 - Distributed Log Splitting
[ https://issues.apache.org/jira/browse/HBASE-20610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16566628#comment-16566628 ] Jingyun Tian commented on HBASE-20610: -- Got some ideas in the design doc. Please check it out and feel free to comment. > Procedure V2 - Distributed Log Splitting > > > Key: HBASE-20610 > URL: https://issues.apache.org/jira/browse/HBASE-20610 > Project: HBase > Issue Type: Umbrella > Components: proc-v2 >Reporter: Guanghao Zhang >Priority: Major > Fix For: 3.0.0 > > > Now master and regionserver use zk to coordinate log split tasks. The split > log manager manages all log files which need to be scanned and split. Then > the split log manager places all the logs into the ZooKeeper splitWAL node > (/hbase/splitWAL) as tasks and monitors these task nodes and waits for them > to be processed. Each regionserver watch splitWAL znode and grab task when > node children changed. And regionserver does the work to split the logs. > Open this umbrella issue to move this "coordinate" work to use new procedure > v2 framework and reduce zk depencency. Plan to finish this before 3.0 > release. Any suggestions are welcomed. Thanks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20610) Procedure V2 - Distributed Log Splitting
[ https://issues.apache.org/jira/browse/HBASE-20610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509454#comment-16509454 ] Guanghao Zhang commented on HBASE-20610: Not start now. I thought we can't finish this in one big patch, so open this as a umbrella issue. Fell free to attach sub-task for this if you start working for this. :-) > Procedure V2 - Distributed Log Splitting > > > Key: HBASE-20610 > URL: https://issues.apache.org/jira/browse/HBASE-20610 > Project: HBase > Issue Type: Umbrella > Components: proc-v2 >Reporter: Guanghao Zhang >Priority: Major > Fix For: 3.0.0 > > > Now master and regionserver use zk to coordinate log split tasks. The split > log manager manages all log files which need to be scanned and split. Then > the split log manager places all the logs into the ZooKeeper splitWAL node > (/hbase/splitWAL) as tasks and monitors these task nodes and waits for them > to be processed. Each regionserver watch splitWAL znode and grab task when > node children changed. And regionserver does the work to split the logs. > Open this umbrella issue to move this "coordinate" work to use new procedure > v2 framework and reduce zk depencency. Plan to finish this before 3.0 > release. Any suggestions are welcomed. Thanks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20610) Procedure V2 - Distributed Log Splitting
[ https://issues.apache.org/jira/browse/HBASE-20610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509380#comment-16509380 ] Allan Yang commented on HBASE-20610: [~zghaobac], are you still working on this? > Procedure V2 - Distributed Log Splitting > > > Key: HBASE-20610 > URL: https://issues.apache.org/jira/browse/HBASE-20610 > Project: HBase > Issue Type: Umbrella > Components: proc-v2 >Reporter: Guanghao Zhang >Priority: Major > Fix For: 3.0.0 > > > Now master and regionserver use zk to coordinate log split tasks. The split > log manager manages all log files which need to be scanned and split. Then > the split log manager places all the logs into the ZooKeeper splitWAL node > (/hbase/splitWAL) as tasks and monitors these task nodes and waits for them > to be processed. Each regionserver watch splitWAL znode and grab task when > node children changed. And regionserver does the work to split the logs. > Open this umbrella issue to move this "coordinate" work to use new procedure > v2 framework and reduce zk depencency. Plan to finish this before 3.0 > release. Any suggestions are welcomed. Thanks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20610) Procedure V2 - Distributed Log Splitting
[ https://issues.apache.org/jira/browse/HBASE-20610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509381#comment-16509381 ] Allan Yang commented on HBASE-20610: [~zghaobac], are you still working on this? > Procedure V2 - Distributed Log Splitting > > > Key: HBASE-20610 > URL: https://issues.apache.org/jira/browse/HBASE-20610 > Project: HBase > Issue Type: Umbrella > Components: proc-v2 >Reporter: Guanghao Zhang >Priority: Major > Fix For: 3.0.0 > > > Now master and regionserver use zk to coordinate log split tasks. The split > log manager manages all log files which need to be scanned and split. Then > the split log manager places all the logs into the ZooKeeper splitWAL node > (/hbase/splitWAL) as tasks and monitors these task nodes and waits for them > to be processed. Each regionserver watch splitWAL znode and grab task when > node children changed. And regionserver does the work to split the logs. > Open this umbrella issue to move this "coordinate" work to use new procedure > v2 framework and reduce zk depencency. Plan to finish this before 3.0 > release. Any suggestions are welcomed. Thanks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
