[
https://issues.apache.org/jira/browse/HBASE-26271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17414985#comment-17414985
]
Duo Zhang edited comment on HBASE-26271 at 9/15/21, 3:01 PM:
-------------------------------------------------------------
Do you guys have any plans on this topic? [~elserj][~wchevreuil]
I think here we need to list the data directory to find out the untracked files
and delete them, since this is a background task, I do not think performance is
critical so listing is acceptable.
For me, there are basically 3 problems:
1. Do we need to scan periodically, or just schedule a background task once
when opening a new region? In general, we will delete the broken files when
flush/compaction fails, so typically, we can only have broken files when RS
crashes. But I'm not sure whether this is a strong enough assumption, as we
could still fail to delete the broken files when rollback a flush/compaction,
and maybe a failed flush/compaction will not always crash the RS, so maybe
scanning periodically is safer.
2. How to prevent deleting newly added files which are not in track yet. Use
the generate time of the HFile to determine?
3. Do not need to schedule the background task if the store file tracker does
not write to data directory directly, but how? We do not expose this
information to upper layer.
Thanks.
was (Author: apache9):
Do you guys have any plans on this topic? [~elserj][~wchevreuil]
I think here we need to list the data directory to find out the untracked files
and delete them, since this is a background task, I do think performance is
critical so listing is acceptable.
For me, there are basically 3 problems:
1. Do we need to scan periodically, or just schedule a background task once
when opening a new region? In general, we will delete the broken files when
flush/compaction fails, so typically, we can only have broken files when RS
crashes. But I'm not sure whether this is a strong enough assumption, as we
could still fail to delete the broken files when rollback a flush/compaction,
and maybe a failed flush/compaction will not always crash the RS, so maybe
scanning periodically is safer.
2. How to prevent deleting newly added files which are not in track yet. Use
the generate time of the HFile to determine?
3. Do not need to schedule the background task if the store file tracker does
not write to data directory directly, but how? We do not expose this
information to upper layer.
Thanks.
> Cleanup the broken store files under data directory
> ---------------------------------------------------
>
> Key: HBASE-26271
> URL: https://issues.apache.org/jira/browse/HBASE-26271
> Project: HBase
> Issue Type: Sub-task
> Components: HFile
> Reporter: Duo Zhang
> Priority: Major
>
> As for some new store file tracker implementation, we allow flush/compaction
> to write directly to data directory, so if we crash in the middle, there will
> be broken store files left in the data directory.
> We should find a proper way to delete these broken files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
