[
https://issues.apache.org/jira/browse/HBASE-26658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17473262#comment-17473262
]
chenglei edited comment on HBASE-26658 at 1/12/22, 3:43 AM:
------------------------------------------------------------
[~zhangduo]，thank you very much for reply.
??The toWriteAppends will be cleared in appendAndAsync, so if you clear
unackedAppends after transferring to toWriteAppends, if we face a HDFS failure
again, there will be data loss...??
I do not know why there is data loss ? In appendAndAsync, we indeed clear
toWriteAppends after we successfully send them, but we also put them in
unackedAppends at the same time, so if we face a HDFS again, unackedAppends
would be transferred to the toWriteAppends to send them again, it seems no
data loss.
??If we fail all the time, we just keep transferring the data from
unackedAppends to toWriteAppends, how could this operation introduce repetitive
data in unackedAppends???
yes , that is my mistake, I ignore only new FSWALEntry could be appended to
unackedAppends, I have omitted it.
was (Author: comnetwork):
[~zhangduo]，thank you very much for reply.
??The toWriteAppends will be cleared in appendAndAsync, so if you clear
unackedAppends after transferring to toWriteAppends, if we face a HDFS failure
again, there will be data loss...??
I do not know why there is data loss ? In appendAndAsync, we indeed clear
toWriteAppends after we successfully send them, but we also put them in
unackedAppends at the same time, so if we face a HDFS again, unackedAppends
would be transferred to the toWriteAppends to send them again, it seems no
data loss.
??If we fail all the time, we just keep transferring the data from
unackedAppends to toWriteAppends, how could this operation introduce repetitive
data in unackedAppends?
??
yes , that is my mistake, I ignore only new FSWALEntry could be appended to
unackedAppends, I have omitted it.
> AsyncFSWAL.unackedAppends should clear after transfered to
> AsyncFSWAL.toWriteAppends
> --------------------------------------------------------------------------------------
>
> Key: HBASE-26658
> URL: https://issues.apache.org/jira/browse/HBASE-26658
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 3.0.0-alpha-2, 2.4.9
> Reporter: chenglei
> Assignee: chenglei
> Priority: Major
>
> When {{ASyncFSWAL}} syncing to HDFS failed, {{AsyncFSWAL.unackedAppends}}
> are transfered to {{AsyncFSWAL.toWriteAppends}} to avoid data loss, but
> {{AsyncFSWAL.unackedAppends}} itself is not cleared. I think there is no need
> to continue retain them in {{AsyncFSWAL.unackedAppends}} because we would
> open a new HDFS pipeline to resend the {{AsyncFSWAL.unackedAppends}}.
> BTW : It would also simplify the logic for fixing HBASE-25905, current fix
> for HBASE-25905 is somewhat hard to understand. I think the problem to cause
> HBASE-25905 is that {{AsyncFSWAL.unackedAppends}} could not exactly reflect
> the *unacked* for current HDFS pipeline. If we clear
> {{AsyncFSWAL.unackedAppends}} after transferring them to
> {{AsyncFSWAL.toWriteAppends}}, HBASE-25905 could also avoid.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
