[jira] [Commented] (HBASE-13773) Replication should not use ZooKeeper at all for coordination

2015-10-27 Thread Heng Chen (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-13773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14975802#comment-14975802
 ] 

Heng Chen commented on HBASE-13773:
---

I just make a suggestion how to implement  ReplicationQueues with hbase table.  
The files i uploaded is just an outline.
If needed, i can make a more detail patch about it. :)  

> Replication should not use ZooKeeper at all for coordination
> 
>
> Key: HBASE-13773
> URL: https://issues.apache.org/jira/browse/HBASE-13773
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0, 1.0.1, 1.1.0, 0.98.12, 1.2.0
>Reporter: Andrew Purtell
>Assignee: Maddineni Sukumar
>Priority: Critical
> Attachments: ReplicationQueueHBaseImpl.java, 
> ReplicationQueueHBaseImpl_v2.java
>
>
> Introduce a new system table for replication state and use this table for 
> coordination instead of znodes.​



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13773) Replication should not use ZooKeeper at all for coordination

2015-10-27 Thread Maddineni Sukumar (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-13773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14975805#comment-14975805
 ] 

Maddineni Sukumar commented on HBASE-13773:
---

[~chenheng]  :) 

Cool np anyhow I am already half way there, I will finish it.  Will take your 
help during code reviews. 
Any how thanks for that sample patch. 

> Replication should not use ZooKeeper at all for coordination
> 
>
> Key: HBASE-13773
> URL: https://issues.apache.org/jira/browse/HBASE-13773
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0, 1.0.1, 1.1.0, 0.98.12, 1.2.0
>Reporter: Andrew Purtell
>Assignee: Maddineni Sukumar
>Priority: Critical
> Attachments: ReplicationQueueHBaseImpl.java, 
> ReplicationQueueHBaseImpl_v2.java
>
>
> Introduce a new system table for replication state and use this table for 
> coordination instead of znodes.​



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13773) Replication should not use ZooKeeper at all for coordination

2015-10-27 Thread Heng Chen (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-13773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14975847#comment-14975847
 ] 

Heng Chen commented on HBASE-13773:
---

Go ahead please.  :)

> Replication should not use ZooKeeper at all for coordination
> 
>
> Key: HBASE-13773
> URL: https://issues.apache.org/jira/browse/HBASE-13773
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0, 1.0.1, 1.1.0, 0.98.12, 1.2.0
>Reporter: Andrew Purtell
>Assignee: Maddineni Sukumar
>Priority: Critical
> Attachments: ReplicationQueueHBaseImpl.java, 
> ReplicationQueueHBaseImpl_v2.java
>
>
> Introduce a new system table for replication state and use this table for 
> coordination instead of znodes.​



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13773) Replication should not use ZooKeeper at all for coordination

2015-10-27 Thread Maddineni Sukumar (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-13773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14975795#comment-14975795
 ] 

Maddineni Sukumar commented on HBASE-13773:
---


Sorry for delay in response, For last one week I was in a remote area in India 
where there is no internet at all. 

[~chenheng]  Thanks for the initial patch. where is the logic to create those 
tables? 

I am also working on it parallel. But my work is still in progress, my 
intention is to remove zk as much as possible. 
Below are the areas .

List of Peers
--List of RS
   --List of Files With Status
if we move above information from zk to HBase table completely then this will 
give us performance benefit as well as no data loss when zk is gone 
completely(we can say loosing ZK has no impact on replication flow)


> Replication should not use ZooKeeper at all for coordination
> 
>
> Key: HBASE-13773
> URL: https://issues.apache.org/jira/browse/HBASE-13773
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0, 1.0.1, 1.1.0, 0.98.12, 1.2.0
>Reporter: Andrew Purtell
>Assignee: Maddineni Sukumar
>Priority: Critical
> Attachments: ReplicationQueueHBaseImpl.java, 
> ReplicationQueueHBaseImpl_v2.java
>
>
> Introduce a new system table for replication state and use this table for 
> coordination instead of znodes.​



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13773) Replication should not use ZooKeeper at all for coordination

2015-10-26 Thread Ashish Singhi (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-13773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14974030#comment-14974030
 ] 

Ashish Singhi commented on HBASE-13773:
---

If a cluster having 100 RS, will it create 100 tables ?

> Replication should not use ZooKeeper at all for coordination
> 
>
> Key: HBASE-13773
> URL: https://issues.apache.org/jira/browse/HBASE-13773
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0, 1.0.1, 1.1.0, 0.98.12, 1.2.0
>Reporter: Andrew Purtell
>Assignee: Maddineni Sukumar
>Priority: Critical
> Attachments: ReplicationQueueHBaseImpl.java
>
>
> Introduce a new system table for replication state and use this table for 
> coordination instead of znodes.​



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13773) Replication should not use ZooKeeper at all for coordination

2015-10-26 Thread Heng Chen (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-13773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14974016#comment-14974016
 ] 

Heng Chen commented on HBASE-13773:
---

{quote}
If zk removed, how the slaves get notified by master if there are some new wals 
to replicate? Polling by slave?
{quote}
Sorry, i got it now.  master do polling and push wals to slave.

I make a ReplicationQueues implementation with system table,  and upload it.  
Any suggestions?

> Replication should not use ZooKeeper at all for coordination
> 
>
> Key: HBASE-13773
> URL: https://issues.apache.org/jira/browse/HBASE-13773
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0, 1.0.1, 1.1.0, 0.98.12, 1.2.0
>Reporter: Andrew Purtell
>Assignee: Maddineni Sukumar
>Priority: Critical
> Attachments: ReplicationQueueHBaseImpl.java
>
>
> Introduce a new system table for replication state and use this table for 
> coordination instead of znodes.​



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13773) Replication should not use ZooKeeper at all for coordination

2015-10-26 Thread Heng Chen (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-13773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14974168#comment-14974168
 ] 

Heng Chen commented on HBASE-13773:
---

{quote}
If a cluster having 100 RS, will it create 100 tables ?
{quote} 
You are right. we update the v2, any suggestions? 
{code}
* We use two tables to do it.
 *
 * The first one is 'RSManager' which used to store RS and queueId information
 * RowKey is 'RS name', table only has one columnFamily 'CF', and columnKey is 
queueId, columnValue is null
 *
 * The secondary table is 'RSWals' which used to store RS wals information.
 * RowKey is 'RS name + queueId', table only has one columnFamily 'CF', and 
columnKey is wal file, columnValue is position
{code}


> Replication should not use ZooKeeper at all for coordination
> 
>
> Key: HBASE-13773
> URL: https://issues.apache.org/jira/browse/HBASE-13773
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0, 1.0.1, 1.1.0, 0.98.12, 1.2.0
>Reporter: Andrew Purtell
>Assignee: Maddineni Sukumar
>Priority: Critical
> Attachments: ReplicationQueueHBaseImpl.java, 
> ReplicationQueueHBaseImpl_v2.java
>
>
> Introduce a new system table for replication state and use this table for 
> coordination instead of znodes.​



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13773) Replication should not use ZooKeeper at all for coordination

2015-10-24 Thread Heng Chen (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-13773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972979#comment-14972979
 ] 

Heng Chen commented on HBASE-13773:
---

What about {{ReplicationQueues}}?  

If zk removed, how the slaves get notified by master if there are some new wals 
to replicate?  Polling by slave?

> Replication should not use ZooKeeper at all for coordination
> 
>
> Key: HBASE-13773
> URL: https://issues.apache.org/jira/browse/HBASE-13773
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0, 1.0.1, 1.1.0, 0.98.12, 1.2.0
>Reporter: Andrew Purtell
>Assignee: Maddineni Sukumar
>Priority: Critical
>
> Introduce a new system table for replication state and use this table for 
> coordination instead of znodes.​



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13773) Replication should not use ZooKeeper at all for coordination

2015-05-26 Thread Ashish Singhi (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-13773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14559006#comment-14559006
 ] 

Ashish Singhi commented on HBASE-13773:
---

Duplicate of HBASE-10295 ?

 Replication should not use ZooKeeper at all for coordination
 

 Key: HBASE-13773
 URL: https://issues.apache.org/jira/browse/HBASE-13773
 Project: HBase
  Issue Type: Improvement
Affects Versions: 2.0.0, 1.0.1, 1.1.0, 0.98.12, 1.2.0
Reporter: Andrew Purtell
Priority: Critical

 Introduce a new system table for replication state and use this table for 
 coordination instead of znodes.​



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13773) Replication should not use ZooKeeper at all for coordination

2015-05-26 Thread Andrew Purtell (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-13773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14559322#comment-14559322
 ] 

Andrew Purtell commented on HBASE-13773:


Not sure HBASE-10295 goes far enough so let's keep them both for now. 
HBASE-10295 is stuck, btw, no activity since October 2014.

 Replication should not use ZooKeeper at all for coordination
 

 Key: HBASE-13773
 URL: https://issues.apache.org/jira/browse/HBASE-13773
 Project: HBase
  Issue Type: Improvement
Affects Versions: 2.0.0, 1.0.1, 1.1.0, 0.98.12, 1.2.0
Reporter: Andrew Purtell
Priority: Critical

 Introduce a new system table for replication state and use this table for 
 coordination instead of znodes.​



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13773) Replication should not use ZooKeeper at all for coordination

2015-05-25 Thread Andrew Purtell (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-13773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14558398#comment-14558398
 ] 

Andrew Purtell commented on HBASE-13773:


Feel free to make this a subtask of any umbrella for a major replication 
overhaul aka Replication V2

 Replication should not use ZooKeeper at all for coordination
 

 Key: HBASE-13773
 URL: https://issues.apache.org/jira/browse/HBASE-13773
 Project: HBase
  Issue Type: Improvement
Affects Versions: 2.0.0, 1.0.1, 1.1.0, 0.98.12, 1.2.0
Reporter: Andrew Purtell
Priority: Critical

 Introduce a new system table for replication state and use this table for 
 coordination instead of znodes.​



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)