[jira] [Updated] (HBASE-18262) name of parameter quote need update in hbase-default.xml

2017-06-25 Thread Dongtao Zhang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-18262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dongtao Zhang updated HBASE-18262:
--
Affects Version/s: (was: 2.0.0-alpha-1)
   1.2.0

> name of parameter quote need update in hbase-default.xml
> 
>
> Key: HBASE-18262
> URL: https://issues.apache.org/jira/browse/HBASE-18262
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 1.2.0
>Reporter: Dongtao Zhang
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HBASE-18262-v001.patch
>
>
> In description of parameter  "hbase.zookeeper.quorum", old name 
> "hbase.zookeeper.clientPort " should be replaced to 
> "hbase.zookeeper.property.clientPort".



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18262) name of parameter quote need update in hbase-default.xml

2017-06-25 Thread Dongtao Zhang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-18262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dongtao Zhang updated HBASE-18262:
--
Fix Version/s: 3.0.0

> name of parameter quote need update in hbase-default.xml
> 
>
> Key: HBASE-18262
> URL: https://issues.apache.org/jira/browse/HBASE-18262
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 1.2.0
>Reporter: Dongtao Zhang
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HBASE-18262-v001.patch
>
>
> In description of parameter  "hbase.zookeeper.quorum", old name 
> "hbase.zookeeper.clientPort " should be replaced to 
> "hbase.zookeeper.property.clientPort".



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18262) name of parameter quote need update in hbase-default.xml

2017-06-25 Thread Dongtao Zhang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-18262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dongtao Zhang updated HBASE-18262:
--
Fix Version/s: (was: 2.0.0-alpha-1)

> name of parameter quote need update in hbase-default.xml
> 
>
> Key: HBASE-18262
> URL: https://issues.apache.org/jira/browse/HBASE-18262
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0-alpha-1
>Reporter: Dongtao Zhang
>Priority: Minor
> Attachments: HBASE-18262-v001.patch
>
>
> In description of parameter  "hbase.zookeeper.quorum", old name 
> "hbase.zookeeper.clientPort " should be replaced to 
> "hbase.zookeeper.property.clientPort".



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18161) Incremental Load support for Multiple-Table HFileOutputFormat

2017-06-25 Thread Jerry He (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-18161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062550#comment-16062550
 ] 

Jerry He commented on HBASE-18161:
--

Added more comments on the RB.  I think it is before you v10 patch.

> Incremental Load support for Multiple-Table HFileOutputFormat
> -
>
> Key: HBASE-18161
> URL: https://issues.apache.org/jira/browse/HBASE-18161
> Project: HBase
>  Issue Type: New Feature
>Reporter: Densel Santhmayor
>Priority: Minor
> Attachments: MultiHFileOutputFormatSupport_HBASE_18161.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v10.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v2.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v3.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v4.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v5.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v6.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v7.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v8.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v9.patch
>
>
> h2. Introduction
> MapReduce currently supports the ability to write HBase records in bulk to 
> HFiles for a single table. The file(s) can then be uploaded to the relevant 
> RegionServers information with reasonable latency. This feature is useful to 
> make a large set of data available for queries at the same time as well as 
> provides a way to efficiently process very large input into HBase without 
> affecting query latencies.
> There is, however, no support to write variations of the same record key to 
> HFiles belonging to multiple HBase tables from within the same MapReduce job. 
>  
> h2. Goal
> The goal of this JIRA is to extend HFileOutputFormat2 to support writing to 
> HFiles for different tables within the same MapReduce job while single-table 
> HFile features backwards-compatible. 
> For our use case, we needed to write a record key to a smaller HBase table 
> for quicker access, and the same record key with a date appended to a larger 
> table for longer term storage with chronological access. Each of these tables 
> would have different TTL and other settings to support their respective 
> access patterns. We also needed to be able to bulk write records to multiple 
> tables with different subsets of very large input as efficiently as possible. 
> Rather than run the MapReduce job multiple times (one for each table or 
> record structure), it would be useful to be able to parse the input a single 
> time and write to multiple tables simultaneously.
> Additionally, we'd like to maintain backwards compatibility with the existing 
> heavily-used HFileOutputFormat2 interface to allow benefits such as locality 
> sensitivity (that was introduced long after we implemented support for 
> multiple tables) to support both single table and multi table hfile writes. 
> h2. Proposal
> * Backwards compatibility for existing single table support in 
> HFileOutputFormat2 will be maintained and in this case, mappers will need to 
> emit the table rowkey as before. However, a new class - 
> MultiHFileOutputFormat - will provide a helper function to generate a rowkey 
> for mappers that prefixes the desired tablename to the existing rowkey as 
> well as provides configureIncrementalLoad support for multiple tables.
> * HFileOutputFormat2 will be updated in the following way:
> ** configureIncrementalLoad will now accept multiple table descriptor and 
> region locator pairs, analogous to the single pair currently accepted by 
> HFileOutputFormat2. 
> ** Compression, Block Size, Bloom Type and Datablock settings PER column 
> family that are set in the Configuration object are now indexed and retrieved 
> by tablename AND column family
> ** getRegionStartKeys will now support multiple regionlocators and calculate 
> split points and therefore partitions collectively for all tables. Similarly, 
> now the eventual number of Reducers will be equal to the total number of 
> partitions across all tables. 
> ** The RecordWriter class will be able to process rowkeys either with or 
> without the tablename prepended depending on how configureIncrementalLoad was 
> configured with MultiHFileOutputFormat or HFileOutputFormat2.
> * The use of MultiHFileOutputFormat will write the output into HFiles which 
> will match the output format of HFileOutputFormat2. However, while the 
> default use case will keep the existing directory structure with column 
> family name as the directory and HFiles within that directory, in the case of 
> MultiHFileOutputFormat, it will output HFiles in the output directory with 
> the following relative paths: 
> {noformat}
>  --table1 
>--family1 
>  --HFiles 
>  --table2 
>--family1 
>--family2 
>  --HFiles
>

[jira] [Updated] (HBASE-18234) Revisit the async admin api

2017-06-25 Thread Guanghao Zhang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-18234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang updated HBASE-18234:
---
Attachment: HBASE-18234.master.010.patch

The failed ut passed locally. Retry for Hadoop QA.

> Revisit the async admin api
> ---
>
> Key: HBASE-18234
> URL: https://issues.apache.org/jira/browse/HBASE-18234
> Project: HBase
>  Issue Type: Sub-task
>  Components: Client
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-18234.master.001.patch, 
> HBASE-18234.master.002.patch, HBASE-18234.master.003.patch, 
> HBASE-18234.master.004.patch, HBASE-18234.master.005.patch, 
> HBASE-18234.master.006.patch, HBASE-18234.master.006.patch, 
> HBASE-18234.master.006.patch, HBASE-18234.master.007.patch, 
> HBASE-18234.master.008.patch, HBASE-18234.master.009.patch, 
> HBASE-18234.master.010.patch, HBASE-18234.master.010.patch
>
>
> 1. Update the balance method name. 
> balancer -> balance
> setBalancerRunning -> setBalancerOn
> isBalancerEnabled -> isBalancerOn
> 2. Use HRegionLocation instead of Pair
> 3. Remove the closeRegionWithEncodedRegionName method. Because all other api 
> can handle region name or encoded region name both. So don't need a method 
> for encoded name.
> 4. Unify the region name parameter's type to byte[]. And region name may be 
> full name or encoded name.
> 5. Unify the server name parameter's type to ServerName. For smoe api, it 
> support null for server name. So use Optional instead.
> 6. Unify the table name parameter's type to TableName.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18161) Incremental Load support for Multiple-Table HFileOutputFormat

2017-06-25 Thread Densel Santhmayor (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-18161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062542#comment-16062542
 ] 

Densel Santhmayor commented on HBASE-18161:
---

Filled out release notes. 

Final patch attached: Added javadoc comments for public apis in 
MultiHFileOutputFormat and renamed createMapperKey to createCompositeKey (See 
release notes on https://issues.apache.org/jira/browse/HBASE-18161 for how 
createCompositeKey can also be used to set storage policy).

> Incremental Load support for Multiple-Table HFileOutputFormat
> -
>
> Key: HBASE-18161
> URL: https://issues.apache.org/jira/browse/HBASE-18161
> Project: HBase
>  Issue Type: New Feature
>Reporter: Densel Santhmayor
>Priority: Minor
> Attachments: MultiHFileOutputFormatSupport_HBASE_18161.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v10.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v2.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v3.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v4.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v5.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v6.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v7.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v8.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v9.patch
>
>
> h2. Introduction
> MapReduce currently supports the ability to write HBase records in bulk to 
> HFiles for a single table. The file(s) can then be uploaded to the relevant 
> RegionServers information with reasonable latency. This feature is useful to 
> make a large set of data available for queries at the same time as well as 
> provides a way to efficiently process very large input into HBase without 
> affecting query latencies.
> There is, however, no support to write variations of the same record key to 
> HFiles belonging to multiple HBase tables from within the same MapReduce job. 
>  
> h2. Goal
> The goal of this JIRA is to extend HFileOutputFormat2 to support writing to 
> HFiles for different tables within the same MapReduce job while single-table 
> HFile features backwards-compatible. 
> For our use case, we needed to write a record key to a smaller HBase table 
> for quicker access, and the same record key with a date appended to a larger 
> table for longer term storage with chronological access. Each of these tables 
> would have different TTL and other settings to support their respective 
> access patterns. We also needed to be able to bulk write records to multiple 
> tables with different subsets of very large input as efficiently as possible. 
> Rather than run the MapReduce job multiple times (one for each table or 
> record structure), it would be useful to be able to parse the input a single 
> time and write to multiple tables simultaneously.
> Additionally, we'd like to maintain backwards compatibility with the existing 
> heavily-used HFileOutputFormat2 interface to allow benefits such as locality 
> sensitivity (that was introduced long after we implemented support for 
> multiple tables) to support both single table and multi table hfile writes. 
> h2. Proposal
> * Backwards compatibility for existing single table support in 
> HFileOutputFormat2 will be maintained and in this case, mappers will need to 
> emit the table rowkey as before. However, a new class - 
> MultiHFileOutputFormat - will provide a helper function to generate a rowkey 
> for mappers that prefixes the desired tablename to the existing rowkey as 
> well as provides configureIncrementalLoad support for multiple tables.
> * HFileOutputFormat2 will be updated in the following way:
> ** configureIncrementalLoad will now accept multiple table descriptor and 
> region locator pairs, analogous to the single pair currently accepted by 
> HFileOutputFormat2. 
> ** Compression, Block Size, Bloom Type and Datablock settings PER column 
> family that are set in the Configuration object are now indexed and retrieved 
> by tablename AND column family
> ** getRegionStartKeys will now support multiple regionlocators and calculate 
> split points and therefore partitions collectively for all tables. Similarly, 
> now the eventual number of Reducers will be equal to the total number of 
> partitions across all tables. 
> ** The RecordWriter class will be able to process rowkeys either with or 
> without the tablename prepended depending on how configureIncrementalLoad was 
> configured with MultiHFileOutputFormat or HFileOutputFormat2.
> * The use of MultiHFileOutputFormat will write the output into HFiles which 
> will match the output format of HFileOutputFormat2. However, while the 
> default use case will keep the existing directory structure with column 
> family name as the directory and HFiles within that directory, in the case of 
>

[jira] [Updated] (HBASE-18161) Incremental Load support for Multiple-Table HFileOutputFormat

2017-06-25 Thread Densel Santhmayor (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-18161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Densel Santhmayor updated HBASE-18161:
--
Attachment: MultiHFileOutputFormatSupport_HBASE_18161_v10.patch

> Incremental Load support for Multiple-Table HFileOutputFormat
> -
>
> Key: HBASE-18161
> URL: https://issues.apache.org/jira/browse/HBASE-18161
> Project: HBase
>  Issue Type: New Feature
>Reporter: Densel Santhmayor
>Priority: Minor
> Attachments: MultiHFileOutputFormatSupport_HBASE_18161.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v10.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v2.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v3.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v4.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v5.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v6.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v7.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v8.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v9.patch
>
>
> h2. Introduction
> MapReduce currently supports the ability to write HBase records in bulk to 
> HFiles for a single table. The file(s) can then be uploaded to the relevant 
> RegionServers information with reasonable latency. This feature is useful to 
> make a large set of data available for queries at the same time as well as 
> provides a way to efficiently process very large input into HBase without 
> affecting query latencies.
> There is, however, no support to write variations of the same record key to 
> HFiles belonging to multiple HBase tables from within the same MapReduce job. 
>  
> h2. Goal
> The goal of this JIRA is to extend HFileOutputFormat2 to support writing to 
> HFiles for different tables within the same MapReduce job while single-table 
> HFile features backwards-compatible. 
> For our use case, we needed to write a record key to a smaller HBase table 
> for quicker access, and the same record key with a date appended to a larger 
> table for longer term storage with chronological access. Each of these tables 
> would have different TTL and other settings to support their respective 
> access patterns. We also needed to be able to bulk write records to multiple 
> tables with different subsets of very large input as efficiently as possible. 
> Rather than run the MapReduce job multiple times (one for each table or 
> record structure), it would be useful to be able to parse the input a single 
> time and write to multiple tables simultaneously.
> Additionally, we'd like to maintain backwards compatibility with the existing 
> heavily-used HFileOutputFormat2 interface to allow benefits such as locality 
> sensitivity (that was introduced long after we implemented support for 
> multiple tables) to support both single table and multi table hfile writes. 
> h2. Proposal
> * Backwards compatibility for existing single table support in 
> HFileOutputFormat2 will be maintained and in this case, mappers will need to 
> emit the table rowkey as before. However, a new class - 
> MultiHFileOutputFormat - will provide a helper function to generate a rowkey 
> for mappers that prefixes the desired tablename to the existing rowkey as 
> well as provides configureIncrementalLoad support for multiple tables.
> * HFileOutputFormat2 will be updated in the following way:
> ** configureIncrementalLoad will now accept multiple table descriptor and 
> region locator pairs, analogous to the single pair currently accepted by 
> HFileOutputFormat2. 
> ** Compression, Block Size, Bloom Type and Datablock settings PER column 
> family that are set in the Configuration object are now indexed and retrieved 
> by tablename AND column family
> ** getRegionStartKeys will now support multiple regionlocators and calculate 
> split points and therefore partitions collectively for all tables. Similarly, 
> now the eventual number of Reducers will be equal to the total number of 
> partitions across all tables. 
> ** The RecordWriter class will be able to process rowkeys either with or 
> without the tablename prepended depending on how configureIncrementalLoad was 
> configured with MultiHFileOutputFormat or HFileOutputFormat2.
> * The use of MultiHFileOutputFormat will write the output into HFiles which 
> will match the output format of HFileOutputFormat2. However, while the 
> default use case will keep the existing directory structure with column 
> family name as the directory and HFiles within that directory, in the case of 
> MultiHFileOutputFormat, it will output HFiles in the output directory with 
> the following relative paths: 
> {noformat}
>  --table1 
>--family1 
>  --HFiles 
>  --table2 
>--family1 
>--family2 
>  --HFiles
> {noformat}
> This aims to be a

[jira] [Commented] (HBASE-18234) Revisit the async admin api

2017-06-25 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-18234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062536#comment-16062536
 ] 

Hadoop QA commented on HBASE-18234:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 9 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 3m 11s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 52s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 
57s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 
22s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 49s 
{color} | {color:red} hbase-client in master has 4 extant Findbugs warnings. 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 45s 
{color} | {color:red} hbase-client in master has 4 extant Findbugs warnings. 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 5m 31s 
{color} | {color:red} hbase-server in master has 10 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 9s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 24s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 24s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 
5s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
58m 3s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha3. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m 
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s 
{color} | {color:green} hbase-client generated 0 new + 1 unchanged - 1 fixed = 
1 total (was 2) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s 
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s 
{color} | {color:green} hbase-endpoint in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 7s 
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 34m 27s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 8s 
{color} | {color:green} hbase-endpoint in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
55s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 151m 45s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.locking.TestLockProcedure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:757bf37 |
| JIRA Patch URL |

[jira] [Updated] (HBASE-18265) Correct the link unuseful in regionServer's region state UI

2017-06-25 Thread Shibin Zhang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-18265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shibin Zhang updated HBASE-18265:
-
Attachment: HBASE-18265.patch

> Correct the link unuseful in regionServer's region state UI
> ---
>
> Key: HBASE-18265
> URL: https://issues.apache.org/jira/browse/HBASE-18265
> Project: HBase
>  Issue Type: Bug
>  Components: UI
>Affects Versions: 3.0.0
>Reporter: Shibin Zhang
>Priority: Trivial
> Attachments: HBASE-18265.patch
>
>
> when i watch for region state ,i found this link maybe update in 
> regionserver's region state UI
> IP:60030/rs-status  is in  RegionListTmpl.jamon
> See http://hbase.org;>HBase Home for further explication.
> http://hbase.org   is unuseful  , should update to  http://hbase.apache.org



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18265) Correct the link unuseful in regionServer's region state UI

2017-06-25 Thread Shibin Zhang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-18265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shibin Zhang updated HBASE-18265:
-
Assignee: Shibin Zhang
  Status: Patch Available  (was: Open)

> Correct the link unuseful in regionServer's region state UI
> ---
>
> Key: HBASE-18265
> URL: https://issues.apache.org/jira/browse/HBASE-18265
> Project: HBase
>  Issue Type: Bug
>  Components: UI
>Affects Versions: 3.0.0
>Reporter: Shibin Zhang
>Assignee: Shibin Zhang
>Priority: Trivial
> Attachments: HBASE-18265.patch
>
>
> when i watch for region state ,i found this link maybe update in 
> regionserver's region state UI
> IP:60030/rs-status  is in  RegionListTmpl.jamon
> See http://hbase.org;>HBase Home for further explication.
> http://hbase.org   is unuseful  , should update to  http://hbase.apache.org



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-18265) Correct the link unuseful in regionServer's region state UI

2017-06-25 Thread Shibin Zhang (JIRA) 
Shibin Zhang created HBASE-18265:


 Summary: Correct the link unuseful in regionServer's region state 
UI
 Key: HBASE-18265
 URL: https://issues.apache.org/jira/browse/HBASE-18265
 Project: HBase
  Issue Type: Bug
  Components: UI
Affects Versions: 3.0.0
Reporter: Shibin Zhang
Priority: Trivial


when i watch for region state ,i found this link maybe update in regionserver's 
region state UI

IP:60030/rs-status  is in  RegionListTmpl.jamon

See http://hbase.org;>HBase Home for further explication.

http://hbase.org   is unuseful  , should update to  http://hbase.apache.org



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18161) Incremental Load support for Multiple-Table HFileOutputFormat

2017-06-25 Thread Densel Santhmayor (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-18161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Densel Santhmayor updated HBASE-18161:
--
Release Note: 
In order to use this feature, a user must 

1. Register their tables when configuring their job

2. Create a composite key of the tablename and original rowkey to send as the 
mapper output key. 



To register their tables (and configure their job for incremental load into 
multiple tables), a user must call the static 
MultiHFileOutputFormat.configureIncrementalLoad function to register the HBase 
tables that will be ingested into. 



To create the composite key, a helper function 
MultiHFileOutputFormat2.createCompositeKey should be called with the 
destination tablename and rowkey as arguments, and the result should be output 
as the mapper key. 


Before this JIRA, for HFileOutputFormat2 a configuration for the storage 
policy was set per Column Family. This was set manually by the user. In this 
JIRA, this is unchanged when using HFileOutputFormat2. However, when 
specifically using MultiHFileOutputFormat2, the user now has to manually set 
the prefix by creating a composite of the table name and the column family. The 
user can create the new composite value by calling 
MultiHFileOutputFormat2.createCompositeKey with the tablename and column family 
as arguments.

Changes added through this JIRA are backwards compatible with existing 
HFileOutputFormat2 apis and functionality.

> Incremental Load support for Multiple-Table HFileOutputFormat
> -
>
> Key: HBASE-18161
> URL: https://issues.apache.org/jira/browse/HBASE-18161
> Project: HBase
>  Issue Type: New Feature
>Reporter: Densel Santhmayor
>Priority: Minor
> Attachments: MultiHFileOutputFormatSupport_HBASE_18161.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v2.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v3.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v4.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v5.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v6.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v7.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v8.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v9.patch
>
>
> h2. Introduction
> MapReduce currently supports the ability to write HBase records in bulk to 
> HFiles for a single table. The file(s) can then be uploaded to the relevant 
> RegionServers information with reasonable latency. This feature is useful to 
> make a large set of data available for queries at the same time as well as 
> provides a way to efficiently process very large input into HBase without 
> affecting query latencies.
> There is, however, no support to write variations of the same record key to 
> HFiles belonging to multiple HBase tables from within the same MapReduce job. 
>  
> h2. Goal
> The goal of this JIRA is to extend HFileOutputFormat2 to support writing to 
> HFiles for different tables within the same MapReduce job while single-table 
> HFile features backwards-compatible. 
> For our use case, we needed to write a record key to a smaller HBase table 
> for quicker access, and the same record key with a date appended to a larger 
> table for longer term storage with chronological access. Each of these tables 
> would have different TTL and other settings to support their respective 
> access patterns. We also needed to be able to bulk write records to multiple 
> tables with different subsets of very large input as efficiently as possible. 
> Rather than run the MapReduce job multiple times (one for each table or 
> record structure), it would be useful to be able to parse the input a single 
> time and write to multiple tables simultaneously.
> Additionally, we'd like to maintain backwards compatibility with the existing 
> heavily-used HFileOutputFormat2 interface to allow benefits such as locality 
> sensitivity (that was introduced long after we implemented support for 
> multiple tables) to support both single table and multi table hfile writes. 
> h2. Proposal
> * Backwards compatibility for existing single table support in 
> HFileOutputFormat2 will be maintained and in this case, mappers will need to 
> emit the table rowkey as before. However, a new class - 
> MultiHFileOutputFormat - will provide a helper function to generate a rowkey 
> for mappers that prefixes the desired tablename to the existing rowkey as 
> well as provides configureIncrementalLoad support for multiple tables.
> * HFileOutputFormat2 will be updated in the following way:
> ** configureIncrementalLoad will now accept multiple table descriptor and 
> region locator pairs, analogous to the single pair currently accepted by 
> HFileOutputFormat2. 
> ** Compression, Block Size, Bloom Type and Datablock settings PER column 
>

[jira] [Updated] (HBASE-18234) Revisit the async admin api

2017-06-25 Thread Guanghao Zhang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-18234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang updated HBASE-18234:
---
Attachment: HBASE-18234.master.010.patch

Fixed javadoc.

> Revisit the async admin api
> ---
>
> Key: HBASE-18234
> URL: https://issues.apache.org/jira/browse/HBASE-18234
> Project: HBase
>  Issue Type: Sub-task
>  Components: Client
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-18234.master.001.patch, 
> HBASE-18234.master.002.patch, HBASE-18234.master.003.patch, 
> HBASE-18234.master.004.patch, HBASE-18234.master.005.patch, 
> HBASE-18234.master.006.patch, HBASE-18234.master.006.patch, 
> HBASE-18234.master.006.patch, HBASE-18234.master.007.patch, 
> HBASE-18234.master.008.patch, HBASE-18234.master.009.patch, 
> HBASE-18234.master.010.patch
>
>
> 1. Update the balance method name. 
> balancer -> balance
> setBalancerRunning -> setBalancerOn
> isBalancerEnabled -> isBalancerOn
> 2. Use HRegionLocation instead of Pair
> 3. Remove the closeRegionWithEncodedRegionName method. Because all other api 
> can handle region name or encoded region name both. So don't need a method 
> for encoded name.
> 4. Unify the region name parameter's type to byte[]. And region name may be 
> full name or encoded name.
> 5. Unify the server name parameter's type to ServerName. For smoe api, it 
> support null for server name. So use Optional instead.
> 6. Unify the table name parameter's type to TableName.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-17125) Inconsistent result when use filter to read data

2017-06-25 Thread Ted Yu (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-17125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062382#comment-16062382
 ] 

Ted Yu commented on HBASE-17125:


Since existing unit tests didn't uncover the defect in 17125-slack-13.txt, 
please add new test to show that patch v17 is correct in this regard.

> Inconsistent result when use filter to read data
> 
>
> Key: HBASE-17125
> URL: https://issues.apache.org/jira/browse/HBASE-17125
> Project: HBase
>  Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: 17125-slack-13.txt, example.diff, 
> HBASE-17125.master.001.patch, HBASE-17125.master.002.patch, 
> HBASE-17125.master.002.patch, HBASE-17125.master.003.patch, 
> HBASE-17125.master.004.patch, HBASE-17125.master.005.patch, 
> HBASE-17125.master.006.patch, HBASE-17125.master.007.patch, 
> HBASE-17125.master.008.patch, HBASE-17125.master.009.patch, 
> HBASE-17125.master.009.patch, HBASE-17125.master.010.patch, 
> HBASE-17125.master.011.patch, HBASE-17125.master.011.patch, 
> HBASE-17125.master.012.patch, HBASE-17125.master.013.patch, 
> HBASE-17125.master.014.patch, HBASE-17125.master.015.patch, 
> HBASE-17125.master.016.patch, HBASE-17125.master.017.patch, 
> HBASE-17125.master.checkReturnedVersions.patch, 
> HBASE-17125.master.no-specified-filter.patch
>
>
> Assume a cloumn's max versions is 3, then we write 4 versions of this column. 
> The oldest version doesn't remove immediately. But from the user view, the 
> oldest version has gone. When user use a filter to query, if the filter skip 
> a new version, then the oldest version will be seen again. But after compact 
> the region, then the oldest version will never been seen. So it is weird for 
> user. The query will get inconsistent result before and after region 
> compaction.
> The reason is matchColumn method of UserScanQueryMatcher. It first check the 
> cell by filter, then check the number of versions needed. So if the filter 
> skip the new version, then the oldest version will be seen again when it is 
> not removed.
> Have a discussion offline with [~Apache9] and [~fenghh], now we have two 
> solution for this problem. The first idea is check the number of versions 
> first, then check the cell by filter. As the comment of setFilter, the filter 
> is called after all tests for ttl, column match, deletes and max versions 
> have been run.
> {code}
>   /**
>* Apply the specified server-side filter when performing the Query.
>* Only {@link Filter#filterKeyValue(Cell)} is called AFTER all tests
>* for ttl, column match, deletes and max versions have been run.
>* @param filter filter to run on the server
>* @return this for invocation chaining
>*/
>   public Query setFilter(Filter filter) {
> this.filter = filter;
> return this;
>   }
> {code}
> But this idea has another problem, if a column's max version is 5 and the 
> user query only need 3 versions. It first check the version's number, then 
> check the cell by filter. So the cells number of the result may less than 3. 
> But there are 2 versions which don't read anymore.
> So the second idea has three steps.
> 1. check by the max versions of this column
> 2. check the kv by filter
> 3. check the versions which user need.
> But this will lead the ScanQueryMatcher more complicated. And this will break 
> the javadoc of Query.setFilter.
> Now we don't have a final solution for this problem. Suggestions are welcomed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (HBASE-17125) Inconsistent result when use filter to read data

2017-06-25 Thread Ted Yu (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-17125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062138#comment-16062138
 ] 

Ted Yu edited comment on HBASE-17125 at 6/25/17 5:20 PM:
-

{code}
+public class ColumnTrackerWrapper implements ColumnTracker {
{code}
There may be more wrapper for ColumnTracker in the future.
How about calling this wrapper FilterAwareColumnTracker (or something like 
that) ?


was (Author: yuzhih...@gmail.com):
{code}
+public class ColumnTrackerWrapper implements ColumnTracker {
{code}
There may be more wrapper for ColumnTracker in the future.
How about calling this wrapper ColumnTrackerWithVersions (or something like 
that) ?

> Inconsistent result when use filter to read data
> 
>
> Key: HBASE-17125
> URL: https://issues.apache.org/jira/browse/HBASE-17125
> Project: HBase
>  Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: 17125-slack-13.txt, example.diff, 
> HBASE-17125.master.001.patch, HBASE-17125.master.002.patch, 
> HBASE-17125.master.002.patch, HBASE-17125.master.003.patch, 
> HBASE-17125.master.004.patch, HBASE-17125.master.005.patch, 
> HBASE-17125.master.006.patch, HBASE-17125.master.007.patch, 
> HBASE-17125.master.008.patch, HBASE-17125.master.009.patch, 
> HBASE-17125.master.009.patch, HBASE-17125.master.010.patch, 
> HBASE-17125.master.011.patch, HBASE-17125.master.011.patch, 
> HBASE-17125.master.012.patch, HBASE-17125.master.013.patch, 
> HBASE-17125.master.014.patch, HBASE-17125.master.015.patch, 
> HBASE-17125.master.016.patch, HBASE-17125.master.017.patch, 
> HBASE-17125.master.checkReturnedVersions.patch, 
> HBASE-17125.master.no-specified-filter.patch
>
>
> Assume a cloumn's max versions is 3, then we write 4 versions of this column. 
> The oldest version doesn't remove immediately. But from the user view, the 
> oldest version has gone. When user use a filter to query, if the filter skip 
> a new version, then the oldest version will be seen again. But after compact 
> the region, then the oldest version will never been seen. So it is weird for 
> user. The query will get inconsistent result before and after region 
> compaction.
> The reason is matchColumn method of UserScanQueryMatcher. It first check the 
> cell by filter, then check the number of versions needed. So if the filter 
> skip the new version, then the oldest version will be seen again when it is 
> not removed.
> Have a discussion offline with [~Apache9] and [~fenghh], now we have two 
> solution for this problem. The first idea is check the number of versions 
> first, then check the cell by filter. As the comment of setFilter, the filter 
> is called after all tests for ttl, column match, deletes and max versions 
> have been run.
> {code}
>   /**
>* Apply the specified server-side filter when performing the Query.
>* Only {@link Filter#filterKeyValue(Cell)} is called AFTER all tests
>* for ttl, column match, deletes and max versions have been run.
>* @param filter filter to run on the server
>* @return this for invocation chaining
>*/
>   public Query setFilter(Filter filter) {
> this.filter = filter;
> return this;
>   }
> {code}
> But this idea has another problem, if a column's max version is 5 and the 
> user query only need 3 versions. It first check the version's number, then 
> check the cell by filter. So the cells number of the result may less than 3. 
> But there are 2 versions which don't read anymore.
> So the second idea has three steps.
> 1. check by the max versions of this column
> 2. check the kv by filter
> 3. check the versions which user need.
> But this will lead the ScanQueryMatcher more complicated. And this will break 
> the javadoc of Query.setFilter.
> Now we don't have a final solution for this problem. Suggestions are welcomed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18234) Revisit the async admin api

2017-06-25 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-18234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062361#comment-16062361
 ] 

Hadoop QA commented on HBASE-18234:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 25s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 9 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 39s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 
34s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 56s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 
55s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 
23s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 47s 
{color} | {color:red} hbase-client in master has 4 extant Findbugs warnings. 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 47s 
{color} | {color:red} hbase-client in master has 4 extant Findbugs warnings. 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 5m 34s 
{color} | {color:red} hbase-server in master has 10 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 13s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 25s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 25s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 
6s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
57m 59s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha3. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m 
51s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 31s 
{color} | {color:red} hbase-client generated 8 new + 1 unchanged - 1 fixed = 9 
total (was 2) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 41s 
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 33m 40s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 47s 
{color} | {color:green} hbase-endpoint in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
40s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 147m 45s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.locking.TestLockProcedure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:757bf37 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12874408/HBASE-18234.master.009.patch
 |
| JIRA Issue | HBASE-18234 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux acf5c8b6b398 4.8.3-std-1 #1 SMP Fri Oct 21 11:15:43 UTC 2016

[jira] [Updated] (HBASE-18234) Revisit the async admin api

2017-06-25 Thread Guanghao Zhang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-18234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang updated HBASE-18234:
---
Attachment: HBASE-18234.master.009.patch

Retry 009 patch.

> Revisit the async admin api
> ---
>
> Key: HBASE-18234
> URL: https://issues.apache.org/jira/browse/HBASE-18234
> Project: HBase
>  Issue Type: Sub-task
>  Components: Client
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-18234.master.001.patch, 
> HBASE-18234.master.002.patch, HBASE-18234.master.003.patch, 
> HBASE-18234.master.004.patch, HBASE-18234.master.005.patch, 
> HBASE-18234.master.006.patch, HBASE-18234.master.006.patch, 
> HBASE-18234.master.006.patch, HBASE-18234.master.007.patch, 
> HBASE-18234.master.008.patch, HBASE-18234.master.009.patch
>
>
> 1. Update the balance method name. 
> balancer -> balance
> setBalancerRunning -> setBalancerOn
> isBalancerEnabled -> isBalancerOn
> 2. Use HRegionLocation instead of Pair
> 3. Remove the closeRegionWithEncodedRegionName method. Because all other api 
> can handle region name or encoded region name both. So don't need a method 
> for encoded name.
> 4. Unify the region name parameter's type to byte[]. And region name may be 
> full name or encoded name.
> 5. Unify the server name parameter's type to ServerName. For smoe api, it 
> support null for server name. So use Optional instead.
> 6. Unify the table name parameter's type to TableName.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18234) Revisit the async admin api

2017-06-25 Thread Guanghao Zhang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-18234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang updated HBASE-18234:
---
Attachment: (was: HBASE-18234.master.009.patch)

> Revisit the async admin api
> ---
>
> Key: HBASE-18234
> URL: https://issues.apache.org/jira/browse/HBASE-18234
> Project: HBase
>  Issue Type: Sub-task
>  Components: Client
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-18234.master.001.patch, 
> HBASE-18234.master.002.patch, HBASE-18234.master.003.patch, 
> HBASE-18234.master.004.patch, HBASE-18234.master.005.patch, 
> HBASE-18234.master.006.patch, HBASE-18234.master.006.patch, 
> HBASE-18234.master.006.patch, HBASE-18234.master.007.patch, 
> HBASE-18234.master.008.patch
>
>
> 1. Update the balance method name. 
> balancer -> balance
> setBalancerRunning -> setBalancerOn
> isBalancerEnabled -> isBalancerOn
> 2. Use HRegionLocation instead of Pair
> 3. Remove the closeRegionWithEncodedRegionName method. Because all other api 
> can handle region name or encoded region name both. So don't need a method 
> for encoded name.
> 4. Unify the region name parameter's type to byte[]. And region name may be 
> full name or encoded name.
> 5. Unify the server name parameter's type to ServerName. For smoe api, it 
> support null for server name. So use Optional instead.
> 6. Unify the table name parameter's type to TableName.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18164) Much faster locality cost function and candidate generator

2017-06-25 Thread Chia-Ping Tsai (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-18164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062281#comment-16062281
 ] 

Chia-Ping Tsai commented on HBASE-18164:


{noformat}
@Ignore // This is broken since new RegionServers does proper average of regions
// and because Master is treated as a regionserver though it hosts two regions 
only.
@Category({FlakeyTests.class, LargeTests.class})
@RunWith(value = Parameterized.class)
public class TestRegionRebalancing {
{noformat}
Oh...TestRegionRebalancing had been annotated with *Ignore* so that the UT is 
skipped in master.

> Much faster locality cost function and candidate generator
> --
>
> Key: HBASE-18164
> URL: https://issues.apache.org/jira/browse/HBASE-18164
> Project: HBase
>  Issue Type: Improvement
>  Components: Balancer
>Reporter: Kahlil Oppenheimer
>Assignee: Kahlil Oppenheimer
>Priority: Critical
> Fix For: 3.0.0, 1.4.0, 2.0.0-alpha-2
>
> Attachments: HBASE-18164-00.patch, HBASE-18164-01.patch, 
> HBASE-18164-02.patch, HBASE-18164-04.patch, HBASE-18164-05.patch
>
>
> We noticed that during the stochastic load balancer was not scaling well with 
> cluster size. That is to say that on our smaller clusters (~17 tables, ~12 
> region servers, ~5k regions), the balancer considers ~100,000 cluster 
> configurations in 60s per balancer run, but only ~5,000 per 60s on our bigger 
> clusters (~82 tables, ~160 region servers, ~13k regions) .
> Because of this, our bigger clusters are not able to converge on balance as 
> quickly for things like table skew, region load, etc. because the balancer 
> does not have enough time to "think".
> We have re-written the locality cost function to be incremental, meaning it 
> only recomputes cost based on the most recent region move proposed by the 
> balancer, rather than recomputing the cost across all regions/servers every 
> iteration.
> Further, we also cache the locality of every region on every server at the 
> beginning of the balancer's execution for both the LocalityBasedCostFunction 
> and the LocalityCandidateGenerator to reference. This way, they need not 
> collect all HDFS blocks of every region at each iteration of the balancer.
> The changes have been running in all 6 of our production clusters and all 4 
> QA clusters without issue. The speed improvements we noticed are massive. Our 
> big clusters now consider 20x more cluster configurations.
> One design decision I made is to consider locality cost as the difference 
> between the best locality that is possible given the current cluster state, 
> and the currently measured locality. The old locality computation would 
> measure the locality cost as the difference from the current locality and 
> 100% locality, but this new computation instead takes the difference between 
> the current locality for a given region and the best locality for that region 
> in the cluster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)