[
https://issues.apache.org/jira/browse/HBASE-5128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13234581#comment-13234581
]
[email protected] commented on HBASE-5128:
------------------------------------------------------
bq. On 2012-03-11 14:37:01, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 1689
bq. > <https://reviews.apache.org/r/4280/diff/1/?file=90977#file90977line1689>
bq. >
bq. > Precede regioninfo with a dot.
done
bq. On 2012-03-11 14:37:01, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 2879
bq. > <https://reviews.apache.org/r/4280/diff/1/?file=90977#file90977line2879>
bq. >
bq. > Can we name this option fixRegionHolesOnHdfs ?
bq. > It would be better to note which options can be run with cluster
offline.
at the moment, hbck can only be run while hbase is online. This has not been
unified with OfflineMetaRebuild yet.
bq. On 2012-03-11 14:37:01, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 2880
bq. > <https://reviews.apache.org/r/4280/diff/1/?file=90977#file90977line2880>
bq. >
bq. > Name this fixRegionOverlapsOnHdfs ?
I'm not sure what making the flag even long buys us. I was thinking about
making it even more concise: -fixHoles, -fixOverlaps. The assumption in this
tool is that the data in the file system is golden and to reconstruct
everything from there (previous version trusted meta table only).
bq. On 2012-03-11 14:37:01, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 2949
bq. > <https://reviews.apache.org/r/4280/diff/1/?file=90977#file90977line2949>
bq. >
bq. > white space.
fixed
bq. On 2012-03-11 14:37:01, Ted Yu wrote:
bq. >
src/main/java/org/apache/hadoop/hbase/util/hbck/TableIntegrityErrorHandler.java,
line 28
bq. > <https://reviews.apache.org/r/4280/diff/1/?file=90980#file90980line28>
bq. >
bq. > Should read 'callbacks for handling particular table integrity
invariant violations detected.'
updated to be in english. :)
bq. On 2012-03-11 14:37:01, Ted Yu wrote:
bq. >
src/main/java/org/apache/hadoop/hbase/util/hbck/TableIntegrityErrorHandler.java,
line 33
bq. > <https://reviews.apache.org/r/4280/diff/1/?file=90980#file90980line33>
bq. >
bq. > Please add javadoc for the handleXXX methods on what scenario each
fixes.
done
bq. On 2012-03-11 14:37:01, Ted Yu wrote:
bq. >
src/main/java/org/apache/hadoop/hbase/util/hbck/TableIntegrityErrorHandler.java,
line 52
bq. > <https://reviews.apache.org/r/4280/diff/1/?file=90980#file90980line52>
bq. >
bq. > This class should be abstract.
bq. > It is better to put it in its own file.
done.
bq. On 2012-03-11 14:37:01, Ted Yu wrote:
bq. >
src/main/java/org/apache/hadoop/hbase/util/hbck/TableIntegrityErrorHandler.java,
line 38
bq. > <https://reviews.apache.org/r/4280/diff/1/?file=90980#file90980line38>
bq. >
bq. > Since region always belongs to some table, I suggest naming this
method handleNonEmptyRegionStartKey()
renamed to handleRegionStartKeyNotEmpty
bq. On 2012-03-11 14:37:01, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 2878
bq. > <https://reviews.apache.org/r/4280/diff/1/?file=90977#file90977line2878>
bq. >
bq. > More than one option modifies .META. table.
bq. > Shall we name this option fixMetaUsingRegionInfoOnHdfs ?
there are two kinds of flags here -- individual flags like -fixAssignments,
-fixMeta, -fixHdfs*, and combo flags that enable a few such as -fixAll. I
going to change the combo flags to make them more distinct; I'll change
-fixAll to be -allFix or something like that to make it clearer. I also need
to update the usage info to be more accurate.
bq. On 2012-03-11 14:37:01, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 2876
bq. > <https://reviews.apache.org/r/4280/diff/1/?file=90977#file90977line2876>
bq. >
bq. > Looking at code @ line 2835 below, it seems -fixAssignments and -fix
are equivalent.
bq. > What was the reason for deprecating -fix ?
-fix and -fixAssigments are equivalent to the original application's behavior.
I didn't want our front line supporters to use -fix assuming old behavior and
have it fix durable state (hdfs modifications), so I added other flags to
enable those modifications. With the other options it seemed like changing the
name to be consistent made sense.
- jmhsieh
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4280/#review5826
-----------------------------------------------------------
On 2012-03-10 01:04:58, jmhsieh wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4280/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-10 01:04:58)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, and Lars Hofhansl.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. This version is similar to the 0.90.x version posted a few months back,
but has a few new features and some minor differences.
bq.
bq. 1) No trackHTD method needed since we can read from the file system.
bq. 2) Added safeguards to prevent mega merges, and to isolate repairs to
particular tables.
bq. 3) Fixed comparator in HRegionInfo
bq. 4) Fixed TestRegionObserverInterface so that it doesn't rely on bug in
HRegionInfo comparator.
bq.
bq. I'll backport to 0.94/0.92 (which should be very similar) and update the
0.90 versions after this patch has mostly cleared.
bq.
bq. This version is not perfect (there are definitely cases not covered) but
it think it is worth trying to get this in so that future reviews are more
manageable.
bq.
bq.
bq. This addresses bug HBASE-5128.
bq. https://issues.apache.org/jira/browse/HBASE-5128
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 98f79fc
bq. src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 3bcf899
bq. src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
ae468ca
bq. src/main/java/org/apache/hadoop/hbase/master/HMaster.java e2bbbd0
bq. src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 720841c
bq. src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java 5916d9c
bq. src/main/java/org/apache/hadoop/hbase/util/hbck/OfflineMetaRepair.java
d57bb6b
bq.
src/main/java/org/apache/hadoop/hbase/util/hbck/TableIntegrityErrorHandler.java
PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java 38eb6a8
bq.
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
1b3b6df
bq. src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java 937781d
bq. src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsckComparator.java
0599da1
bq. src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java
dbb97f8
bq.
src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildBase.java
2b4cac8
bq.
src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildHole.java
ebbeead
bq.
src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildOverlap.java
b175548
bq.
bq. Diff: https://reviews.apache.org/r/4280/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Unit tests cover many many situations and pass. Most "live" testing has
been done on 0.90.x versions. Many improvements and features added from
experience. Not much testing live on the trunk versions.
bq.
bq.
bq. Thanks,
bq.
bq. jmhsieh
bq.
bq.
> [uber hbck] Enable hbck to automatically repair table integrity problems as
> well as region consistency problems while online.
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-5128
> URL: https://issues.apache.org/jira/browse/HBASE-5128
> Project: HBase
> Issue Type: New Feature
> Components: hbck
> Affects Versions: 0.90.5, 0.92.0, 0.94.0, 0.96.0
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: hbase-5128-trunk.patch
>
>
> The current (0.90.5, 0.92.0rc2) versions of hbck detects most of region
> consistency and table integrity invariant violations. However with '-fix' it
> can only automatically repair region consistency cases having to do with
> deployment problems. This updated version should be able to handle all cases
> (including a new orphan regiondir case). When complete will likely deprecate
> the OfflineMetaRepair tool and subsume several open META-hole related issue.
> Here's the approach (from the comment of at the top of the new version of the
> file).
> {code}
> /**
> * HBaseFsck (hbck) is a tool for checking and repairing region consistency
> and
> * table integrity.
> *
> * Region consistency checks verify that META, region deployment on
> * region servers and the state of data in HDFS (.regioninfo files) all are in
> * accordance.
> *
> * Table integrity checks verify that that all possible row keys can resolve
> to
> * exactly one region of a table. This means there are no individual
> degenerate
> * or backwards regions; no holes between regions; and that there no
> overlapping
> * regions.
> *
> * The general repair strategy works in these steps.
> * 1) Repair Table Integrity on HDFS. (merge or fabricate regions)
> * 2) Repair Region Consistency with META and assignments
> *
> * For table integrity repairs, the tables their region directories are
> scanned
> * for .regioninfo files. Each table's integrity is then verified. If there
> * are any orphan regions (regions with no .regioninfo files), or holes, new
> * regions are fabricated. Backwards regions are sidelined as well as empty
> * degenerate (endkey==startkey) regions. If there are any overlapping
> regions,
> * a new region is created and all data is merged into the new region.
> *
> * Table integrity repairs deal solely with HDFS and can be done offline --
> the
> * hbase region servers or master do not need to be running. These phase can
> be
> * use to completely reconstruct the META table in an offline fashion.
> *
> * Region consistency requires three conditions -- 1) valid .regioninfo file
> * present in an hdfs region dir, 2) valid row with .regioninfo data in META,
> * and 3) a region is deployed only at the regionserver that is was assigned
> to.
> *
> * Region consistency requires hbck to contact the HBase master and region
> * servers, so the connect() must first be called successfully. Much of the
> * region consistency information is transient and less risky to repair.
> */
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
