[
https://issues.apache.org/jira/browse/HBASE-2437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12871931#action_12871931
]
HBase Review Board commented on HBASE-2437:
-------------------------------------------
Message from: "Todd Lipcon" <[email protected]>
Oh, it's much much improved! Thanks! I still think a "high level overview"
would be good to see.
why .regionserver.? I'd say just hbase.hlog.split.batch.size or something
if this applies only to hlog splitting, maybe hbase.hlog.split.skip.errors
Which do you mean by "writers" here? I'd support factoring this function out
into an HdfsUtil class somewhere.
I agree if it has no records (I think - do we syncfs after writing the
sequencefile header?). But there's the case where inside SequenceFile we call
create, but never actually write any bytes. This is still worth recovering.
In general I think a corrupt tail means we should drop that record (incomplete
written record) but not shut down. This is only true if it's the tail record,
though.
Can't find it now... does my above comment make sense?
This seems really voodoo.. if anything we're probably masking a real bug by
doing this. Can you write a unit test which shows this problem (even if it
takes 30 minutes to run, would be good to have in our arsenal)
The case where this happens is if you crash in the middle of appending a long
edit. Consider the case where a single edit might have 1MB of data (large
rows). We can easily crash in the middle of transferring it, before we call
sync on the edit. In this case, the client never received an "ack" for the
write, so we can feel free to throw it away (this isn't data loss, it's correct
operation).
- Todd
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.hbase.org/r/74/#review40
-----------------------------------------------------------
On 2010-05-21 12:11:59, stack wrote:
> Refactor HLog splitLog
> ----------------------
>
> Key: HBASE-2437
> URL: https://issues.apache.org/jira/browse/HBASE-2437
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.21.0
> Reporter: Cosmin Lehene
> Assignee: Cosmin Lehene
> Fix For: 0.21.0
>
> Attachments: 2437-v2.txt, 2437-v3.txt, 2437-v4.patch, 2437.txt,
> HBASE-2437_for_HBase-0.21_with_unit_tests_for_HDFS-0.21.patch
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> the HLog.splitLog got really long and complex and hard to verify for
> correctness.
> I started to refactor it and also ported changes from hbase-2337 that deals
> with premature deletion of log files in case of errors. Further improvements
> will be possible, however the scope of this issue is to clean the code and
> make it behave correctly (i.e. not lose any edits)
> Added a suite of unit tests that might be ported to 0.20 as well.
> Added a setting (hbase.skip.errors - feel free to suggest a better name)
> that, when set to false will make the process less tolerant to failures or
> corrupted files: in case a log file is corrupted or an error stops the
> process from consistently splitting the log, will abort the entire operation
> to avoid losing any edits. When hbase.skip.errors is on any corrupted files
> will be partially parsed and then moved to the corrupted logs archive (see
> hbase-2337).
> Like hbase-2337 the splitLog method will first split all the logs and then
> proceed to archive them. If any splitted log file (oldlogfile.log) that is
> the result of an earlier splitLog attempt is found in the region directory,
> it will be deleted - this is safe since we won't move the original log files
> until the splitLog process completes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
