Repository: hbase Updated Branches: refs/heads/branch-2.0 1fe783925 -> 213a95c7c
HBASE-21497 Copy down docs for 2.0.3 release Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/213a95c7 Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/213a95c7 Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/213a95c7 Branch: refs/heads/branch-2.0 Commit: 213a95c7c5a032f101f35877f97b13e53aa6e0f3 Parents: 1fe7839 Author: stack <st...@apache.org> Authored: Sun Nov 18 13:23:06 2018 -0800 Committer: stack <st...@apache.org> Committed: Sun Nov 18 13:23:25 2018 -0800 ---------------------------------------------------------------------- src/main/asciidoc/_chapters/developer.adoc | 162 ++++++++++++++++++------ src/main/asciidoc/_chapters/ops_mgt.adoc | 51 +++++--- src/main/asciidoc/_chapters/security.adoc | 5 +- src/main/asciidoc/_chapters/upgrading.adoc | 5 +- src/main/asciidoc/book.adoc | 1 - 5 files changed, 158 insertions(+), 66 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/hbase/blob/213a95c7/src/main/asciidoc/_chapters/developer.adoc ---------------------------------------------------------------------- diff --git a/src/main/asciidoc/_chapters/developer.adoc b/src/main/asciidoc/_chapters/developer.adoc index 935d6e6..51ed461 100644 --- a/src/main/asciidoc/_chapters/developer.adoc +++ b/src/main/asciidoc/_chapters/developer.adoc @@ -2040,30 +2040,97 @@ For more information on how to use ReviewBoard, see link:http://www.reviewboard. ==== Guide for HBase Committers +===== Becoming a committer + +Committers are responsible for reviewing and integrating code changes, testing +and voting on release candidates, weighing in on design discussions, as well as +other types of project contributions. The PMC votes to make a contributor a +committer based on an assessment of their contributions to the project. It is +expected that committers demonstrate a sustained history of high-quality +contributions to the project and community involvement. + +Contributions can be made in many ways. There is no single path to becoming a +committer, nor any expected timeline. Submitting features, improvements, and bug +fixes is the most common avenue, but other methods are both recognized and +encouraged (and may be even more important to the health of HBase as a project and a +community). A non-exhaustive list of potential contributions (in no particular +order): + +* <<appendix_contributing_to_documentation,Update the documentation>> for new + changes, best practices, recipes, and other improvements. +* Keep the website up to date. +* Perform testing and report the results. For instance, scale testing and + testing non-standard configurations is always appreciated. +* Maintain the shared Jenkins testing environment and other testing + infrastructure. +* <<hbase.rc.voting,Vote on release candidates>> after performing validation, even if non-binding. + A non-binding vote is a vote by a non-committer. +* Provide input for discussion threads on the link:/mail-lists.html[mailing lists] (which usually have + `[DISCUSS]` in the subject line). +* Answer questions questions on the user or developer mailing lists and on + Slack. +* Make sure the HBase community is a welcoming one and that we adhere to our + link:/coc.html[Code of conduct]. Alert the PMC if you + have concerns. +* Review other people's work (both code and non-code) and provide public + feedback. +* Report bugs that are found, or file new feature requests. +* Triage issues and keep JIRA organized. This includes closing stale issues, + labeling new issues, updating metadata, and other tasks as needed. +* Mentor new contributors of all sorts. +* Give talks and write blogs about HBase. Add these to the link:/[News] section + of the website. +* Provide UX feedback about HBase, the web UI, the CLI, APIs, and the website. +* Write demo applications and scripts. +* Help attract and retain a diverse community. +* Interact with other projects in ways that benefit HBase and those other + projects. + +Not every individual is able to do all (or even any) of the items on this list. +If you think of other ways to contribute, go for it (and add them to the list). +A pleasant demeanor and willingness to contribute are all you need to make a +positive impact on the HBase project. Invitations to become a committer are the +result of steady interaction with the community over the long term, which builds +trust and recognition. + ===== New committers -New committers are encouraged to first read Apache's generic committer documentation: +New committers are encouraged to first read Apache's generic committer +documentation: * link:https://www.apache.org/dev/new-committers-guide.html[Apache New Committer Guide] * link:https://www.apache.org/dev/committers.html[Apache Committer FAQ] ===== Review -HBase committers should, as often as possible, attempt to review patches submitted by others. -Ideally every submitted patch will get reviewed by a committer _within a few days_. -If a committer reviews a patch they have not authored, and believe it to be of sufficient quality, then they can commit the patch, otherwise the patch should be cancelled with a clear explanation for why it was rejected. - -The list of submitted patches is in the link:https://issues.apache.org/jira/secure/IssueNavigator.jspa?mode=hide&requestId=12312392[HBase Review Queue], which is ordered by time of last modification. -Committers should scan the list from top to bottom, looking for patches that they feel qualified to review and possibly commit. - -For non-trivial changes, it is required to get another committer to review your own patches before commit. -Use the btn:[Submit Patch] button in JIRA, just like other contributors, and then wait for a `+1` response from another committer before committing. +HBase committers should, as often as possible, attempt to review patches +submitted by others. Ideally every submitted patch will get reviewed by a +committer _within a few days_. If a committer reviews a patch they have not +authored, and believe it to be of sufficient quality, then they can commit the +patch. Otherwise the patch should be cancelled with a clear explanation for why +it was rejected. + +The list of submitted patches is in the +link:https://issues.apache.org/jira/secure/IssueNavigator.jspa?mode=hide&requestId=12312392[HBase Review Queue], +which is ordered by time of last modification. Committers should scan the list +from top to bottom, looking for patches that they feel qualified to review and +possibly commit. If you see a patch you think someone else is better qualified +to review, you can mention them by username in the JIRA. + +For non-trivial changes, it is required that another committer review your +patches before commit. **Self-commits of non-trivial patches are not allowed.** +Use the btn:[Submit Patch] button in JIRA, just like other contributors, and +then wait for a `+1` response from another committer before committing. ===== Reject -Patches which do not adhere to the guidelines in link:https://hbase.apache.org/book.html#developer[HowToContribute] and to the link:https://wiki.apache.org/hadoop/CodeReviewChecklist[code review checklist] should be rejected. -Committers should always be polite to contributors and try to instruct and encourage them to contribute better patches. -If a committer wishes to improve an unacceptable patch, then it should first be rejected, and a new patch should be attached by the committer for review. +Patches which do not adhere to the guidelines in +link:https://hbase.apache.org/book.html#developer[HowToContribute] and to the +link:https://wiki.apache.org/hadoop/CodeReviewChecklist[code review checklist] +should be rejected. Committers should always be polite to contributors and try +to instruct and encourage them to contribute better patches. If a committer +wishes to improve an unacceptable patch, then it should first be rejected, and a +new patch should be attached by the committer for further review. [[committing.patches]] ===== Commit @@ -2074,29 +2141,34 @@ Committers commit patches to the Apache HBase GIT repository. [NOTE] ==== Make sure your local configuration is correct, especially your identity and email. -Examine the output of the +$ git config - --list+ command and be sure it is correct. -See this GitHub article, link:https://help.github.com/articles/set-up-git[Set Up Git] if you need pointers. +Examine the output of the +$ git config --list+ command and be sure it is correct. +See link:https://help.github.com/articles/set-up-git[Set Up Git] if you need +pointers. ==== -When you commit a patch, please: - -. Include the Jira issue id in the commit message along with a short description of the change. Try - to add something more than just the Jira title so that someone looking at git log doesn't - have to go to Jira to discern what the change is about. - Be sure to get the issue ID right, as this causes Jira to link to the change in Git (use the - issue's "All" tab to see these). -. Commit the patch to a new branch based off master or other intended branch. - It's a good idea to call this branch by the JIRA ID. - Then check out the relevant target branch where you want to commit, make sure your local branch has all remote changes, by doing a +git pull --rebase+ or another similar command, cherry-pick the change into each relevant branch (such as master), and do +git push <remote-server> - <remote-branch>+. +When you commit a patch: + +. Include the Jira issue ID in the commit message along with a short description + of the change. Try to add something more than just the Jira title so that + someone looking at `git log` output doesn't have to go to Jira to discern what + the change is about. Be sure to get the issue ID right, because this causes + Jira to link to the change in Git (use the issue's "All" tab to see these + automatic links). +. Commit the patch to a new branch based off `master` or the other intended + branch. It's a good idea to include the JIRA ID in the name of this branch. + Check out the relevant target branch where you want to commit, and make sure + your local branch has all remote changes, by doing a +git pull --rebase+ or + another similar command. Next, cherry-pick the change into each relevant + branch (such as master), and push the changes to the remote branch using + a command such as +git push <remote-server> <remote-branch>+. + WARNING: If you do not have all remote changes, the push will fail. If the push fails for any reason, fix the problem or ask for help. Do not do a +git push --force+. + Before you can commit a patch, you need to determine how the patch was created. -The instructions and preferences around the way to create patches have changed, and there will be a transition period. +The instructions and preferences around the way to create patches have changed, +and there will be a transition period. + .Determine How a Patch Was Created * If the first few lines of the patch look like the headers of an email, with a From, Date, and @@ -2123,16 +2195,18 @@ diff --git src/main/asciidoc/_chapters/developer.adoc src/main/asciidoc/_chapter + .Example of committing a Patch ==== -One thing you will notice with these examples is that there are a lot of +git pull+ commands. -The only command that actually writes anything to the remote repository is +git push+, and you need to make absolutely sure you have the correct versions of everything and don't have any conflicts before pushing. -The extra +git - pull+ commands are usually redundant, but better safe than sorry. +One thing you will notice with these examples is that there are a lot of ++git pull+ commands. The only command that actually writes anything to the +remote repository is +git push+, and you need to make absolutely sure you have +the correct versions of everything and don't have any conflicts before pushing. +The extra +git pull+ commands are usually redundant, but better safe than sorry. -The first example shows how to apply a patch that was generated with +git format-patch+ and apply it to the `master` and `branch-1` branches. +The first example shows how to apply a patch that was generated with +git +format-patch+ and apply it to the `master` and `branch-1` branches. -The directive to use +git format-patch+ rather than +git diff+, and not to use `--no-prefix`, is a new one. -See the second example for how to apply a patch created with +git - diff+, and educate the person who created the patch. +The directive to use +git format-patch+ rather than +git diff+, and not to use +`--no-prefix`, is a new one. See the second example for how to apply a patch +created with +git diff+, and educate the person who created the patch. ---- $ git checkout -b HBASE-XXXX @@ -2154,13 +2228,13 @@ $ git push origin branch-1 $ git branch -D HBASE-XXXX ---- -This example shows how to commit a patch that was created using +git diff+ without `--no-prefix`. -If the patch was created with `--no-prefix`, add `-p0` to the +git apply+ command. +This example shows how to commit a patch that was created using +git diff+ +without `--no-prefix`. If the patch was created with `--no-prefix`, add `-p0` to +the +git apply+ command. ---- $ git apply ~/Downloads/HBASE-XXXX-v2.patch -$ git commit -m "HBASE-XXXX Really Good Code Fix (Joe Schmo)" --author=<contributor> -a # This -and next command is needed for patches created with 'git diff' +$ git commit -m "HBASE-XXXX Really Good Code Fix (Joe Schmo)" --author=<contributor> -a # This and next command is needed for patches created with 'git diff' $ git commit --amend --signoff $ git checkout master $ git pull --rebase @@ -2181,7 +2255,9 @@ $ git branch -D HBASE-XXXX ==== . Resolve the issue as fixed, thanking the contributor. - Always set the "Fix Version" at this point, but please only set a single fix version for each branch where the change was committed, the earliest release in that branch in which the change will appear. + Always set the "Fix Version" at this point, but only set a single fix version + for each branch where the change was committed, the earliest release in that + branch in which the change will appear. ====== Commit Message Format @@ -2196,7 +2272,9 @@ The preferred commit message format is: HBASE-12345 Fix All The Things (j...@example.com) ---- -If the contributor used +git format-patch+ to generate the patch, their commit message is in their patch and you can use that, but be sure the JIRA ID is at the front of the commit message, even if the contributor left it out. +If the contributor used +git format-patch+ to generate the patch, their commit +message is in their patch and you can use that, but be sure the JIRA ID is at +the front of the commit message, even if the contributor left it out. [[committer.amending.author]] ====== Add Amending-Author when a conflict cherrypick backporting http://git-wip-us.apache.org/repos/asf/hbase/blob/213a95c7/src/main/asciidoc/_chapters/ops_mgt.adoc ---------------------------------------------------------------------- diff --git a/src/main/asciidoc/_chapters/ops_mgt.adoc b/src/main/asciidoc/_chapters/ops_mgt.adoc index ca9cab6..473418f 100644 --- a/src/main/asciidoc/_chapters/ops_mgt.adoc +++ b/src/main/asciidoc/_chapters/ops_mgt.adoc @@ -51,7 +51,8 @@ Options: Commands: Some commands take arguments. Pass no args or -h for usage. shell Run the HBase shell - hbck Run the hbase 'fsck' tool + hbck Run the HBase 'fsck' tool. Defaults read-only hbck1. + Pass '-j /path/to/HBCK2.jar' to run hbase-2.x HBCK2. snapshot Tool for managing snapshots wal Write-ahead-log analyzer hfile Store file analyzer @@ -386,12 +387,33 @@ Each command except `RowCounter` and `CellCounter` accept a single `--help` argu [[hbck]] === HBase `hbck` -To run `hbck` against your HBase cluster run `$./bin/hbase hbck`. At the end of the command's output it prints `OK` or `INCONSISTENCY`. -If your cluster reports inconsistencies, pass `-details` to see more detail emitted. -If inconsistencies, run `hbck` a few times because the inconsistency may be transient (e.g. cluster is starting up or a region is splitting). - Passing `-fix` may correct the inconsistency (This is an experimental feature). +The `hbck` tool that shipped with hbase-1.x has been made read-only in hbase-2.x. It is not able to repair +hbase-2.x clusters as hbase internals have changed. Nor should its assessments in read-only mode be +trusted as it does not understand hbase-2.x operation. -For more information, see <<hbck.in.depth>>. +A new tool, <<HBCK2>>, described in the next section, replaces `hbck`. + +[[HBCK2]] +=== HBase `HBCK2` + +`HBCK2` is the successor to <<hbck>>, the hbase-1.x fix tool (A.K.A `hbck1`). Use it in place of `hbck1` +making repairs against hbase-2.x installs. + +`HBCK2` does not ship as part of hbase. It can be found as a subproject of the companion +link:https://github.com/apache/hbase-operator-tools[hbase-operator-tools] repository at +link:https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2[Apache HBase HBCK2 Tool]. +`HBCK2` was moved out of hbase so it could evolve at a cadence apart from that of hbase core. + +See the [https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2](HBCK2) Home Page +for how `HBCK2` differs from `hbck1`, and for how to build and use it. + +Once built, you can run `HBCK2` as follows: + +``` +$ hbase hbck -j /path/to/HBCK2.jar +``` + +This will generate `HBCK2` usage describing commands and options. [[hfile_tool2]] === HFile Tool @@ -1272,15 +1294,6 @@ Monitor the output of the _/tmp/log.txt_ file to follow the progress of the scri Use the following guidelines if you want to create your own rolling restart script. . Extract the new release, verify its configuration, and synchronize it to all nodes of your cluster using `rsync`, `scp`, or another secure synchronization mechanism. -. Use the hbck utility to ensure that the cluster is consistent. -+ ----- - -$ ./bin/hbck ----- -+ -Perform repairs if required. -See <<hbck,hbck>> for details. . Restart the master first. You may need to modify these commands if your new HBase directory is different from the old one, such as for an upgrade. @@ -1312,7 +1325,6 @@ $ for i in `cat conf/regionservers|sort`; do ./bin/graceful_stop.sh --restart -- ---- . Restart the Master again, to clear out the dead servers list and re-enable the load balancer. -. Run the `hbck` utility again, to be sure the cluster is consistent. [[adding.new.node]] === Adding a New Node @@ -1658,7 +1670,7 @@ How your application builds on top of the HBase API matters when replication is The combination of these two properties (at-least-once delivery and the lack of message ordering) means that some destination clusters may end up in a different state if your application makes use of operations that are not idempotent, e.g. Increments. -To solve the problem, HBase now supports serial replication, which sends edits to destination cluster as the order of requests from client. It is available in hbase-2.1.x, not in this version (hbase-2.0.x). +To solve the problem, HBase now supports serial replication, which sends edits to destination cluster as the order of requests from client. See <<Serial Replication,Serial Replication>>. ==== @@ -1700,6 +1712,9 @@ Instead of SQL statements, entire WALEdits (consisting of multiple cell inserts LOG.info("Replicating "+clusterId + " -> " + peerClusterId); ---- +.Serial Replication Configuration +See <<Serial Replication,Serial Replication>> + .Cluster Management Commands add_peer <ID> <CLUSTER_KEY>:: Adds a replication relationship between two clusters. + @@ -2675,8 +2690,6 @@ HDFS replication factor only affects your disk usage and is invisible to most HB You can view the current number of regions for a given table using the HMaster UI. In the [label]#Tables# section, the number of online regions for each table is listed in the [label]#Online Regions# column. This total only includes the in-memory state and does not include disabled or offline regions. -If you do not want to use the HMaster UI, you can determine the number of regions by counting the number of subdirectories of the /hbase/<table>/ subdirectories in HDFS, or by running the `bin/hbase hbck` command. -Each of these methods may return a slightly different number, depending on the status of each region. [[ops.capacity.regions.count]] ==== Number of regions per RS - upper bound http://git-wip-us.apache.org/repos/asf/hbase/blob/213a95c7/src/main/asciidoc/_chapters/security.adoc ---------------------------------------------------------------------- diff --git a/src/main/asciidoc/_chapters/security.adoc b/src/main/asciidoc/_chapters/security.adoc index 53e8dbf..56f6566 100644 --- a/src/main/asciidoc/_chapters/security.adoc +++ b/src/main/asciidoc/_chapters/security.adoc @@ -1739,7 +1739,7 @@ All options have been discussed separately in the sections above. <!-- HBase Superuser --> <property> <name>hbase.superuser</name> - <value>hbase, admin</value> + <value>hbase,admin</value> </property> <!-- Coprocessors for ACLs and Visibility Tags --> <property> @@ -1759,8 +1759,7 @@ All options have been discussed separately in the sections above. </property> <property> <name>hbase.coprocessor.regionserver.classes</name> - <value>org.apache.hadoop/hbase.security.access.AccessController, - org.apache.hadoop.hbase.security.access.VisibilityController</value> + <value>org.apache.hadoop.hbase.security.access.AccessController</value> </property> <!-- Executable ACL for Coprocessor Endpoints --> <property> http://git-wip-us.apache.org/repos/asf/hbase/blob/213a95c7/src/main/asciidoc/_chapters/upgrading.adoc ---------------------------------------------------------------------- diff --git a/src/main/asciidoc/_chapters/upgrading.adoc b/src/main/asciidoc/_chapters/upgrading.adoc index a556123..da0dac0 100644 --- a/src/main/asciidoc/_chapters/upgrading.adoc +++ b/src/main/asciidoc/_chapters/upgrading.adoc @@ -331,7 +331,10 @@ As noted in the section <<basic.prerequisites>>, HBase 2.0+ requires a minimum o .HBCK must match HBase server version You *must not* use an HBase 1.x version of HBCK against an HBase 2.0+ cluster. HBCK is strongly tied to the HBase server version. Using the HBCK tool from an earlier release against an HBase 2.0+ cluster will destructively alter said cluster in unrecoverable ways. -As of HBase 2.0, HBCK is a read-only tool that can report the status of some non-public system internals. You should not rely on the format nor content of these internals to remain consistent across HBase releases. +As of HBase 2.0, HBCK (A.K.A _HBCK1_ or _hbck1_) is a read-only tool that can report the status of some non-public system internals. You should not rely on the format nor content of these internals to remain consistent across HBase releases. + +To read about HBCK's replacement, see <<HBCK2>> in <<ops_mgt>>. + //// Link to a ref guide section on HBCK in 2.0 that explains use and calls out the inability of clients and server sides to detect version of each other. http://git-wip-us.apache.org/repos/asf/hbase/blob/213a95c7/src/main/asciidoc/book.adoc ---------------------------------------------------------------------- diff --git a/src/main/asciidoc/book.adoc b/src/main/asciidoc/book.adoc index 764d7b4..667e5c6 100644 --- a/src/main/asciidoc/book.adoc +++ b/src/main/asciidoc/book.adoc @@ -83,7 +83,6 @@ include::_chapters/community.adoc[] include::_chapters/appendix_contributing_to_documentation.adoc[] include::_chapters/faq.adoc[] -include::_chapters/hbck_in_depth.adoc[] include::_chapters/appendix_acl_matrix.adoc[] include::_chapters/compression.adoc[] include::_chapters/sql.adoc[]
