You can see my design doc for async dfs output https://docs.google.com/document/d/11AyWtGhItQs6vsLRIx32PwTxmBY3libXwGXI25obVEY/edit#heading=h.2jvw6cxnmirr
See the footnote below section 3.4. For the current HDFS pipeline implementation, it could be a problem for replication in HBase, though it rarely happens. And now HBase has its own AsyncFSWAL implementation, HBASE-14004 is used to resolve the problem(although later we make things wrong and HBASE-24625 is the fix). And for WAL recovery, it will not be a problem. We will only return success to client after all the replicas have been successfully committed, so if DN2 goes offline, we will close the current file and commit it, and open a new file to write WAL. Thanks. Paul Carey <[email protected]> 于2020年7月3日周五 下午7:40写道： > > If the hdfs write succeeded while u had only one DN available, then the > other replica on the offline DN would be invalid now. > > Interesting, I wasn't aware of this. Are there any docs you could > point me towards where this is described? I've had a look in Hadoop: > The Definitive Guide and the official docs, but hadn't come across > this. > > On Fri, Jul 3, 2020 at 11:19 AM Wellington Chevreuil > <[email protected]> wrote: > > > > This is actually an hdfs consistency question, not hbase. If the hdfs > write > > succeeded while u had only one DN available, then the other replica on > the > > offline DN would be invalid now. Then what u have is an under replicated > > block, and of your only available DN goes offline before it could be > > replicated, the file that block belongs to now is corrupt. If I turn on > the > > previous offline DN, it would still be corrupt as the replica it has is > not > > valid anymore (NN knows which is the last valid version of the replica), > so > > unless u can bring back the DN that has the only valid replica, your > hfilr > > is corrupt and your data is lost. > > > > On Fri, 3 Jul 2020, 09:12 Paul Carey, <[email protected]> wrote: > > > > > Hi > > > > > > I'd like to understand how HBase deals with the situation where the > > > only available DataNodes for a given offline Region contain stale > > > data. Will HBase allow the Region to be brought online again, > > > effectively making the inconsistency permanent, or will it refuse to > > > do so? > > > > > > My question is motivated from seeing how Kafka and Elasticsearch > > > handle this scenario. They both allow the inconsistency to become > > > permanent, Kafka via unclean leader election, and Elasticsearch via > > > the allocate_stale_primary command. > > > > > > To better understand my question, please consider the following > example: > > > > > > - HDFS is configured with `dfs.replication=2` and > > > `dfs.namenode.replication.min=1` > > > - DataNodes DN1 and DN2 contain the blocks for Region R1 > > > - DN2 goes offline > > > - R1 receives a writes which succeeds as it can be written > successfully to > > > DN1 > > > - DN1 goes offline before the NameNode can replicate the > > > under-replicated block containing the write to another DataNode > > > - At this point the R1 is offline > > > - DN2 comes back online, but it does not contain the missed write > > > > > > There are now two options: > > > > > > - R1 is brought back online, violating consistency > > > - R1 remains offline, indefinitely, until DN1 is brought back online > > > > > > How does HBase deal with this situation? > > > > > > Many thanks > > > > > > Paul > > > >
