[ 
https://issues.apache.org/jira/browse/YARN-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14790987#comment-14790987
 ] 

Joep Rottinghuis commented on YARN-4062:
----------------------------------------

While discussing flush and compaction with [~vrushalic] I just realized that 
there might be a complication with cross-dc replication.

Potentially the RS in two different datacenters might decide to flush/compact 
values for one row at the same time. We need to think through the consequences 
what happens if they make a different decision (because one DC might have later 
information that hasn't been replicated across such as app completion for 
example). Even if the order and the decisions are deterministic, we need to 
consider what happens if two regions modify the same row.
With hRaven we have been able to make master-master replication work because we 
were guaranteed that every row is "owned" and therefore manipulated only 
locally.

Perhaps we can do the same here, where flush and compactions happen only in the 
HBase cluster located in the datacenter where the row is owned. For example, 
only if the rowkey starts with the same datacenter as where the copro runs. 
This would ensure that each row is flushed/compacted only in one DC and the 
other DCs would be followers.

This would have to be configurable and disabled for installations with a single 
HBase instance that are written to remotely by multiple datacenters, otherwise 
no compaction will happen at all (at least perhaps functionally correct even if 
not optimal for space usage).


> Add the flush and compaction functionality via coprocessors and scanners for 
> flow run table
> -------------------------------------------------------------------------------------------
>
>                 Key: YARN-4062
>                 URL: https://issues.apache.org/jira/browse/YARN-4062
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Vrushali C
>            Assignee: Vrushali C
>
> As part of YARN-3901, coprocessor and scanner is being added for storing into 
> the flow_run table. It also needs a flush & compaction processing in the 
> coprocessor and perhaps a new scanner to deal with the data during flushing 
> and compaction stages. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to