[
https://issues.apache.org/jira/browse/HBASE-14949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15149675#comment-15149675
]
Duo Zhang commented on HBASE-14949:
-----------------------------------
There are two places where we might have file name conflict, one is the
intermediate file when spliting, the other is the final recovered edits file.
For final file, it is named using the highest sequence id of the WAL entries in
it. Since both the files are completed, we could just compare the lowest
sequence id of the WAL entries in each, the lower one should have more entries
so we could safely delete the other one.
And for the intermediate file, it is named using the sequence id of the first
WAL entry that written into it. If conflict happens at that time, the file is
not completed yet so we do not know which one could be deleted, so we should
avoid conflict here. One simple way is to use a UUID, but if split failed with
an RS crash, the file will left there forever so we need to add a cleanup
chore. Using a fixed rule to generate the file name does not have this problem
because we always need to split the WAL file and every time we will using the
same intermediate file name. If we find a conflict, and since we know that
other file's split processing will not use the same name, we can make sure this
is a failed split intermediate file and delete it. That's why I append a
fileBeingWritten name after it.
{quote}
If we do find two files the same, we just take the one with most edits without
regard to sequenceids?
{quote}
Oh, this is a good question. We have a multi WAL implementation now, I do not
know if it is still safe to delete the one with fewer entries when multi WAL is
on... But we should not change the final name if we want to do rolling
upgrade... And if we found that it is not safe when multi WAL is on, we could
do a merge on the two files? It is a little slow but I think the conflict is
rarely happen?
Thanks.
> Resolve name conflict when splitting if there are duplicated WAL entries
> ------------------------------------------------------------------------
>
> Key: HBASE-14949
> URL: https://issues.apache.org/jira/browse/HBASE-14949
> Project: HBase
> Issue Type: Sub-task
> Reporter: Heng Chen
> Assignee: Duo Zhang
> Attachments: HBASE-14949-v3.patch, HBASE-14949-v4.patch,
> HBASE-14949.patch, HBASE-14949_v1.patch, HBASE-14949_v2.patch
>
>
> The AsyncFSHLog introduced in HBASE-14790 may write same WAL entries to
> different WAL files. WAL entry itself is idempotent so replay is not a
> problem but the intermediate file name and final name when splitting is
> constructed using the lowest or highest sequence id of the WAL entries
> written, so it is possible that different WAL files will have same
> intermediate or final file name when splitting. In the currentm
> implementation, this will cause split fail or data loss. We need to solve
> this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
