junfei liang created HBASE-23160:
------------------------------------
Summary: The Dead RS May Remove compacted files after recover
from full gc
Key: HBASE-23160
URL: https://issues.apache.org/jira/browse/HBASE-23160
Project: HBase
Issue Type: Bug
Components: regionserver
Reporter: junfei liang
in our online cluster, we found daughter region's reference file can point to
a nonexistent hfile. so when is region is balanced, the region open will be
failed as FileNotFoundException, and a lot of errors thrown.
how the problem happen
1. Region R1 is on server S1, and it's has a compaction, say storefile sf1
is compacted into another file at time t1.
2. S1 has a long full gc (in our case about 470s) at t2 (t1 + 300s)
3. R1 is offline from S1 after t2 + 180s, rs zk session expired , so
master thought the RS is dead and reassign the R1 to S2.
4. S2 found R1 is too large so it make a split request, and R1 split into
R2 + R3, both hold a reference to sf1.
5 . the S1 finish the fullgc t2 + 470s , and before it report to master,
CompactedHFilesDischarger remove compacted file sf1 from R1 (R1 is still online
on Server S1 )
6. so R2、R3 hold a reference to not exists storefile，and lead to the error
we came across。
possible solutions：
1. write WAL Marker before remove hfile from store
as in SSH, the dead rs log dir is deleted, so write wal marker will be
failed.
but is not absolutely reliable， because rs can fullgc after write the
marker. there is not way we do these two action ** atomically.
it's not 100% reliable , but it's simple...
2. a possible reliable solution
when remove hfile from store dir, first move it to a RS-Level special
DIR, and then move to archived dir.
and we delete the DIR in the SSH，so the remove compacted files will be
failed in the first step, it's reliable but complicated.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
