[
https://issues.apache.org/jira/browse/HBASE-26658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17474286#comment-17474286
]
chenglei edited comment on HBASE-26658 at 1/12/22, 10:15 AM:
-------------------------------------------------------------
[~zhangduo],thank you for feedback,
bq.We need to track all the entries which have already been sent out without
ack, as we do not know the state of these entries. If we just clear them after
transferring them back to toWriteAppends, and then there is a shutdown, we may
report to the upper layer that these entries have not been written out
I do not very understand...
1. How we using unackedAppends to report the upper layer when there is
shutdown?seems I can not find existing code in the master for using
unackedAppends to report entries not written out or not received response to
the upper layer.
2. Why the upper layer should differentiate the entry is not written out
or not received response? I do not see any existing code use ASyncFSWAL take
care it, do you mean in the future? seems upper layer should just know it is
successful or failed. For distribute system, seems it is not very meaningful
to report the unacked intermediate state to upper layer , just as you said we
could not make sure any thing. For upper layer, an entry is in unackedAppends
could just state that it has once in FanOutOneBlockAsyncDFSOutput.buf, even
could not make sure it is sent out, so seems that only report to the upper
layer successful or failed is suitable, just as AsyncFSWAL does now. Clearing
unackedAppends after transferring to toWriteAppends does not affect the
successful or failed result.
Well, now it is not a bug. I just find that seems clearing unackedAppends
after transferring to toWriteAppends is a more simpler and easier to understand
way to fix
HBASE-25905 from my view，just for your reference. Because the data we want to
resend is already in toWriteAppends, clearing seems natural and harmless(not
make the data loss or logic error) .
was (Author: comnetwork):
[~zhangduo],thank you for feedback,
bq.We need to track all the entries which have already been sent out without
ack, as we do not know the state of these entries. If we just clear them after
transferring them back to toWriteAppends, and then there is a shutdown, we may
report to the upper layer that these entries have not been written out
I do not very understand...
1. How we using unackedAppends to report the upper layer when there is
shutdown?seems I can not find existing code in the master for using
unackedAppends to report entries not written out or not received response to
the upper layer.
2. Why the upper layer should differentiate the entry is not written out
or not received response? seems upper layer should just know it is successful
or failed. For distribute system, seems it is not very meaningful to report
the unacked intermediate state to upper layer , just as you said we could not
make sure any thing. For upper layer, an entry is in unackedAppends could just
state that it has once in FanOutOneBlockAsyncDFSOutput.buf, even could not make
sure it is sent out, so seems that only report to the upper layer successful or
failed is suitable, just as AsyncFSWAL does now. Clearing unackedAppends
after transferring to toWriteAppends does not affect the successful or failed
result.
Well, now it is not a bug. I just find that seems clearing unackedAppends
after transferring to toWriteAppends is a more simpler and easier to understand
way to fix
HBASE-25905 from my view，just for your reference. Because the data we want to
resend is already in toWriteAppends, clearing seems natural and harmless(not
make the data loss or logic error) .
> AsyncFSWAL.unackedAppends should clear after transfered to
> AsyncFSWAL.toWriteAppends
> --------------------------------------------------------------------------------------
>
> Key: HBASE-26658
> URL: https://issues.apache.org/jira/browse/HBASE-26658
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 3.0.0-alpha-2, 2.4.9
> Reporter: chenglei
> Assignee: chenglei
> Priority: Major
>
> When {{ASyncFSWAL}} syncing to HDFS failed, {{AsyncFSWAL.unackedAppends}}
> are transfered to {{AsyncFSWAL.toWriteAppends}} to avoid data loss, but
> {{AsyncFSWAL.unackedAppends}} itself is not cleared. I think there is no need
> to continue retain them in {{AsyncFSWAL.unackedAppends}} because we would
> open a new HDFS pipeline to resend the {{AsyncFSWAL.unackedAppends}}.
> BTW : It would also simplify the logic for fixing HBASE-25905, current fix
> for HBASE-25905 is somewhat hard to understand. I think the problem to cause
> HBASE-25905 is that {{AsyncFSWAL.unackedAppends}} could not exactly reflect
> the *unacked* for current HDFS pipeline. If we clear
> {{AsyncFSWAL.unackedAppends}} after transferring them to
> {{AsyncFSWAL.toWriteAppends}}, HBASE-25905 could also avoid.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
