[jira] [Commented] (HBASE-20193) Basic Replication Web UI - Regionserver

[Jingyun Tian (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423870#comment-16423870
 ] 

Jingyun Tian commented on HBASE-20193:
--

[~stack] can you help checko this patch out? The compiling warning is not 
introduced by my patch.

> Basic Replication Web UI - Regionserver 
> 
>
> Key: HBASE-20193
> URL: https://issues.apache.org/jira/browse/HBASE-20193
> Project: HBase
>  Issue Type: Sub-task
>  Components: Replication, Usability
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Critical
> Fix For: 2.1.0
>
> Attachments: HBASE-20193.master.001.patch, 
> HBASE-20193.master.002.patch, HBASE-20193.master.003.patch, 
> HBASE-20193.master.004.patch
>
>
> subtask of HBASE-15809. Implementation of replication UI on Regionserver web 
> page.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20194) Basic Replication WebUI - Master

[Jingyun Tian (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423872#comment-16423872
 ] 

Jingyun Tian commented on HBASE-20194:
--

[~Apache9] Can you check this patch? Add a UT for it.

> Basic Replication WebUI - Master
> 
>
> Key: HBASE-20194
> URL: https://issues.apache.org/jira/browse/HBASE-20194
> Project: HBase
>  Issue Type: Sub-task
>  Components: Replication, Usability
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Critical
> Fix For: 2.1.0
>
> Attachments: HBASE-20194.master.001.patch, 
> HBASE-20194.master.002.patch, HBASE-20194.master.003.patch, 
> HBASE-20194.master.004.patch, HBASE-20194.master.005.patch
>
>
> subtask of HBASE-15809. Implementation of Replication WebUI on Master webpage.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19890) Canary usage should document hbase.canary.sink.class config

[Peter Somogyi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Somogyi updated HBASE-19890:
--
Attachment: HBASE-19890.master.001.patch

> Canary usage should document hbase.canary.sink.class config
> ---
>
> Key: HBASE-19890
> URL: https://issues.apache.org/jira/browse/HBASE-19890
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Peter Somogyi
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-19890.master.001.patch
>
>
> Canary#main uses config hbase.canary.sink.class to instantiate Sink class.
> The Sink instance affects creation of Monitor.
> In the refguide for Canary, hbase.canary.sink.class was not mentioned.
>  We should document this config.
> Additionally, we need to document that using the default sink is not 
> compatible with table parameters as input so the user must change it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19890) Canary usage should document hbase.canary.sink.class config

[Peter Somogyi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Somogyi updated HBASE-19890:
--
Status: Patch Available  (was: Open)

> Canary usage should document hbase.canary.sink.class config
> ---
>
> Key: HBASE-19890
> URL: https://issues.apache.org/jira/browse/HBASE-19890
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Peter Somogyi
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-19890.master.001.patch
>
>
> Canary#main uses config hbase.canary.sink.class to instantiate Sink class.
> The Sink instance affects creation of Monitor.
> In the refguide for Canary, hbase.canary.sink.class was not mentioned.
>  We should document this config.
> Additionally, we need to document that using the default sink is not 
> compatible with table parameters as input so the user must change it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HBASE-20193) Basic Replication Web UI - Regionserver

[Jingyun Tian (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423870#comment-16423870
 ] 

Jingyun Tian edited comment on HBASE-20193 at 4/3/18 11:24 AM:
---

[~stack]  [~Apache9] can you help check this patch out? The compiling warning 
is not introduced by my patch.


was (Author: tianjingyun):
[~stack] can you help checko this patch out? The compiling warning is not 
introduced by my patch.

> Basic Replication Web UI - Regionserver 
> 
>
> Key: HBASE-20193
> URL: https://issues.apache.org/jira/browse/HBASE-20193
> Project: HBase
>  Issue Type: Sub-task
>  Components: Replication, Usability
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Critical
> Fix For: 2.1.0
>
> Attachments: HBASE-20193.master.001.patch, 
> HBASE-20193.master.002.patch, HBASE-20193.master.003.patch, 
> HBASE-20193.master.004.patch
>
>
> subtask of HBASE-15809. Implementation of replication UI on Regionserver web 
> page.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20328) Fix local backup master start command in documentation

[Yuki Tawara (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423738#comment-16423738
 ] 

Yuki Tawara commented on HBASE-20328:
-

Thanks for your review, [~uagashe]!

> Fix local backup master start command in documentation
> --
>
> Key: HBASE-20328
> URL: https://issues.apache.org/jira/browse/HBASE-20328
> Project: HBase
>  Issue Type: Bug
>  Components: documentation
>Reporter: Yuki Tawara
>Assignee: Yuki Tawara
>Priority: Minor
> Attachments: HBASE-20328.master.001.patch
>
>
> In "2.3. Pseudo-Distributed Local Install" section of documentation, a 
> command for starting backup masters lacks "start" argument.
> {noformat}
> $ ./bin/local-master-backup.sh 2 3 5
> {noformat}
> should
> {noformat}
> $ ./bin/local-master-backup.sh start 2 3 5
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20193) Basic Replication Web UI - Regionserver

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423847#comment-16423847
 ] 

Hadoop QA commented on HBASE-20193:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  6m 
 3s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
33s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  1m 45s{color} 
| {color:red} hbase-server generated 1 new + 187 unchanged - 1 fixed = 188 
total (was 188) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  5m 
 4s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
20m 23s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}122m 
25s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
23s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}169m  8s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f |
| JIRA Issue | HBASE-20193 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12917323/HBASE-20193.master.004.patch
 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux 00d77ca7074f 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 219625233c |
| maven | version: Apache Maven 3.5.3 
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC3 |
| javac | 
https://builds.apache.org/job/PreCommit-HBASE-Build/12271/artifact/patchprocess/diff-compile-javac-hbase-server.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/12271/testReport/ |
| Max. process+thread count | 4176 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output | 

[jira] [Commented] (HBASE-15291) FileSystem not closed in secure bulkLoad

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423946#comment-16423946
 ] 

Ashish Singhi commented on HBASE-15291:
---

Test failure is not related.

> FileSystem not closed in secure bulkLoad
> 
>
> Key: HBASE-15291
> URL: https://issues.apache.org/jira/browse/HBASE-15291
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.0.2, 0.98.16.1
>Reporter: Yong Zhang
>Assignee: Ashish Singhi
>Priority: Major
> Attachments: HBASE-15291-revert-master.patch, HBASE-15291.001.patch, 
> HBASE-15291.002.patch, HBASE-15291.003.patch, HBASE-15291.004.patch, 
> HBASE-15291.addendum, HBASE-15291.patch, HBASE-15291.v1.patch, 
> HBASE-15291.v2.patch, HBASE-15291.v2.patch, patch
>
>
> FileSystem not closed in secure bulkLoad after bulkLoad  finish, it will 
> cause memory used more and more if too many bulkLoad .



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20259) Doc configs for in-memory-compaction and add detail to in-memory-compaction logging

[Peter Somogyi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423944#comment-16423944
 ] 

Peter Somogyi commented on HBASE-20259:
---

[~stack], your push to branch-2.0 caused a merge conflict.

[https://github.com/apache/hbase/blob/branch-2.0/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java#L343]
 

> Doc configs for in-memory-compaction and add detail to in-memory-compaction 
> logging
> ---
>
> Key: HBASE-20259
> URL: https://issues.apache.org/jira/browse/HBASE-20259
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-20259.master.001.patch, 
> HBASE-20259.master.002.patch, HBASE-20259.master.003.patch
>
>
> I set {{hbase.systemtables.compacting.memstore.type}} to NONE but it seems 
> like in-memory is still on. My table looks like this:
> {code}
> Table ycsb is ENABLED
> ycsb
> COLUMN FAMILIES DESCRIPTION
> {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', 
> NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', 
> CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 
> 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER =
> > 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', 
> > CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', 
> > COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}
> {code}
> Looks like table doesn't have it on either (IN_MEMORY_COMPACTION doesn't show 
> in the above).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20259) Doc configs for in-memory-compaction and add detail to in-memory-compaction logging

[Peter Somogyi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423950#comment-16423950
 ] 

Peter Somogyi commented on HBASE-20259:
---

Attached a simple addendum.

> Doc configs for in-memory-compaction and add detail to in-memory-compaction 
> logging
> ---
>
> Key: HBASE-20259
> URL: https://issues.apache.org/jira/browse/HBASE-20259
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-20259.branch-2.0.ADDENDUM.patch, 
> HBASE-20259.master.001.patch, HBASE-20259.master.002.patch, 
> HBASE-20259.master.003.patch
>
>
> I set {{hbase.systemtables.compacting.memstore.type}} to NONE but it seems 
> like in-memory is still on. My table looks like this:
> {code}
> Table ycsb is ENABLED
> ycsb
> COLUMN FAMILIES DESCRIPTION
> {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', 
> NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', 
> CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 
> 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER =
> > 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', 
> > CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', 
> > COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}
> {code}
> Looks like table doesn't have it on either (IN_MEMORY_COMPACTION doesn't show 
> in the above).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20259) Doc configs for in-memory-compaction and add detail to in-memory-compaction logging

[Peter Somogyi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Somogyi updated HBASE-20259:
--
Attachment: HBASE-20259.branch-2.0.ADDENDUM.patch

> Doc configs for in-memory-compaction and add detail to in-memory-compaction 
> logging
> ---
>
> Key: HBASE-20259
> URL: https://issues.apache.org/jira/browse/HBASE-20259
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-20259.branch-2.0.ADDENDUM.patch, 
> HBASE-20259.master.001.patch, HBASE-20259.master.002.patch, 
> HBASE-20259.master.003.patch
>
>
> I set {{hbase.systemtables.compacting.memstore.type}} to NONE but it seems 
> like in-memory is still on. My table looks like this:
> {code}
> Table ycsb is ENABLED
> ycsb
> COLUMN FAMILIES DESCRIPTION
> {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', 
> NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', 
> CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 
> 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER =
> > 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', 
> > CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', 
> > COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}
> {code}
> Looks like table doesn't have it on either (IN_MEMORY_COMPACTION doesn't show 
> in the above).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20236) [locking] Write-time worst offenders

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423678#comment-16423678
 ] 

Hadoop QA commented on HBASE-20236:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
1s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
 9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  5m 
 7s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
 8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  1m 35s{color} 
| {color:red} hbase-server generated 6 new + 182 unchanged - 6 fixed = 188 
total (was 188) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  1m 
19s{color} | {color:red} hbase-server: The patch generated 2 new + 282 
unchanged - 0 fixed = 284 total (was 282) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
34s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
17m 55s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}159m 36s{color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
22s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}200m 50s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestPerColumnFamilyFlush |
|   | hadoop.hbase.regionserver.TestDefaultMemStore |
|   | hadoop.hbase.regionserver.throttle.TestFlushWithThroughputController |
|   | hadoop.hbase.regionserver.TestCompactingToCellFlatMapMemStore |
|   | hadoop.hbase.regionserver.TestCompactingMemStore |
|   | hadoop.hbase.regionserver.TestHStore |
|   | hadoop.hbase.regionserver.TestWalAndCompactingMemStoreFlush |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f |
| JIRA Issue | HBASE-20236 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12917304/HBASE-20236.master.001.patch
 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux cdfd0933ee08 4.4.0-104-generic #127-Ubuntu SMP Mon Dec 11 
12:16:42 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 219625233c |

[jira] [Reopened] (HBASE-20259) Doc configs for in-memory-compaction and add detail to in-memory-compaction logging

[Peter Somogyi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Somogyi reopened HBASE-20259:
---

Reopening to fix merge conflict on branch-2.0.

> Doc configs for in-memory-compaction and add detail to in-memory-compaction 
> logging
> ---
>
> Key: HBASE-20259
> URL: https://issues.apache.org/jira/browse/HBASE-20259
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-20259.master.001.patch, 
> HBASE-20259.master.002.patch, HBASE-20259.master.003.patch
>
>
> I set {{hbase.systemtables.compacting.memstore.type}} to NONE but it seems 
> like in-memory is still on. My table looks like this:
> {code}
> Table ycsb is ENABLED
> ycsb
> COLUMN FAMILIES DESCRIPTION
> {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', 
> NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', 
> CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 
> 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER =
> > 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', 
> > CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', 
> > COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}
> {code}
> Looks like table doesn't have it on either (IN_MEMORY_COMPACTION doesn't show 
> in the above).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20298) Doc change in read/write/total accounting metrics

[Yu Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423718#comment-16423718
 ] 

Yu Li commented on HBASE-20298:
---

+1 on the doc changes, thanks for taking care of this sir. [~stack]

>From checkstyle output, we need to remove unused imports (caused by change in 
>this patch) from {{HRegionServer}}

> Doc change in read/write/total accounting metrics
> -
>
> Key: HBASE-20298
> URL: https://issues.apache.org/jira/browse/HBASE-20298
> Project: HBase
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-20298.master.001.patch, 
> HBASE-20298.master.002.patch
>
>
> Doc the change wrought by the parent issue. Get it up into the refguide as 
> part of the difference between old hbases and hbase2.
> The change confused me and took me a while to untangle.
> The read count is for reads that return a non-empty result now. In old 
> hbase1, we'd increment the read-count even if an empty result. This makes 
> reads look bad in YCSB runs when compared to hbase1 (see how 
> totalRequestCount in hbase2 can be way above the sum of reads+writes; it is 
> because it increments even if the row is not found).
> Let me get this into refguide otherwise poor old operators will be baffled. 
> The release note on the parent is great; it just needs to be in our guide.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20194) Basic Replication WebUI - Master

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423857#comment-16423857
 ] 

Hadoop QA commented on HBASE-20194:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  6m 
10s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
30s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
55s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
20m 20s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}123m 
26s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
23s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}170m 52s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f |
| JIRA Issue | HBASE-20194 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12917324/HBASE-20194.master.005.patch
 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux 60588cd137e3 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
 |
| git revision | master / 219625233c |
| maven | version: Apache Maven 3.5.3 
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC3 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/12272/testReport/ |
| Max. process+thread count | 4233 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/12272/console |
| Powered by | Apache Yetus 0.7.0   http://yetus.apache.org |


This message was automatically generated.



> Basic Replication WebUI 

[jira] [Updated] (HBASE-16499) slow replication for small HBase clusters

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-16499:
--
Attachment: HBASE-16499.patch

> slow replication for small HBase clusters
> -
>
> Key: HBASE-16499
> URL: https://issues.apache.org/jira/browse/HBASE-16499
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: Vikas Vishwakarma
>Assignee: Ashish Singhi
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-16499.patch, HBASE-16499.patch
>
>
> For small clusters 10-20 nodes we recently observed that replication is 
> progressing very slowly when we do bulk writes and there is lot of lag 
> accumulation on AgeOfLastShipped / SizeOfLogQueue. From the logs we observed 
> that the number of threads used for shipping wal edits in parallel comes from 
> the following equation in HBaseInterClusterReplicationEndpoint
> int n = Math.min(Math.min(this.maxThreads, entries.size()/100+1),
>   replicationSinkMgr.getSinks().size());
> ... 
>   for (int i=0; i entryLists.add(new ArrayList(entries.size()/n+1)); <-- 
> batch size
>   }
> ...
> for (int i=0; i  .
> // RuntimeExceptions encountered here bubble up and are handled 
> in ReplicationSource
> pool.submit(createReplicator(entryLists.get(i), i));  <-- 
> concurrency 
> futures++;
>   }
> }
> maxThreads is fixed & configurable and since we are taking min of the three 
> values n gets decided based replicationSinkMgr.getSinks().size() when we have 
> enough edits to replicate
> replicationSinkMgr.getSinks().size() is decided based on 
> int numSinks = (int) Math.ceil(slaveAddresses.size() * ratio);
> where ratio is this.ratio = conf.getFloat("replication.source.ratio", 
> DEFAULT_REPLICATION_SOURCE_RATIO);
> Currently DEFAULT_REPLICATION_SOURCE_RATIO is set to 10% so for small 
> clusters of size 10-20 RegionServers  the value we get for numSinks and hence 
> n is very small like 1 or 2. This substantially reduces the pool concurrency 
> used for shipping wal edits in parallel effectively slowing down replication 
> for small clusters and causing lot of lag accumulation in AgeOfLastShipped. 
> Sometimes it takes tens of hours to clear off the entire replication queue 
> even after the client has finished writing on the source side. 
> We are running tests by varying replication.source.ratio and have seen 
> multi-fold improvement in total replication time (will update the results 
> here). I wanted to propose here that we should increase the default value for 
> replication.source.ratio also so that we have sufficient concurrency even for 
> small clusters. We figured it out after lot of iterations and debugging so 
> probably slightly higher default will save the trouble. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-16499) slow replication for small HBase clusters

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423947#comment-16423947
 ] 

Ashish Singhi commented on HBASE-16499:
---

May be same RS in peer was getting selected for replication every time!

> slow replication for small HBase clusters
> -
>
> Key: HBASE-16499
> URL: https://issues.apache.org/jira/browse/HBASE-16499
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: Vikas Vishwakarma
>Assignee: Ashish Singhi
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-16499.patch, HBASE-16499.patch
>
>
> For small clusters 10-20 nodes we recently observed that replication is 
> progressing very slowly when we do bulk writes and there is lot of lag 
> accumulation on AgeOfLastShipped / SizeOfLogQueue. From the logs we observed 
> that the number of threads used for shipping wal edits in parallel comes from 
> the following equation in HBaseInterClusterReplicationEndpoint
> int n = Math.min(Math.min(this.maxThreads, entries.size()/100+1),
>   replicationSinkMgr.getSinks().size());
> ... 
>   for (int i=0; i entryLists.add(new ArrayList(entries.size()/n+1)); <-- 
> batch size
>   }
> ...
> for (int i=0; i  .
> // RuntimeExceptions encountered here bubble up and are handled 
> in ReplicationSource
> pool.submit(createReplicator(entryLists.get(i), i));  <-- 
> concurrency 
> futures++;
>   }
> }
> maxThreads is fixed & configurable and since we are taking min of the three 
> values n gets decided based replicationSinkMgr.getSinks().size() when we have 
> enough edits to replicate
> replicationSinkMgr.getSinks().size() is decided based on 
> int numSinks = (int) Math.ceil(slaveAddresses.size() * ratio);
> where ratio is this.ratio = conf.getFloat("replication.source.ratio", 
> DEFAULT_REPLICATION_SOURCE_RATIO);
> Currently DEFAULT_REPLICATION_SOURCE_RATIO is set to 10% so for small 
> clusters of size 10-20 RegionServers  the value we get for numSinks and hence 
> n is very small like 1 or 2. This substantially reduces the pool concurrency 
> used for shipping wal edits in parallel effectively slowing down replication 
> for small clusters and causing lot of lag accumulation in AgeOfLastShipped. 
> Sometimes it takes tens of hours to clear off the entire replication queue 
> even after the client has finished writing on the source side. 
> We are running tests by varying replication.source.ratio and have seen 
> multi-fold improvement in total replication time (will update the results 
> here). I wanted to propose here that we should increase the default value for 
> replication.source.ratio also so that we have sufficient concurrency even for 
> small clusters. We figured it out after lot of iterations and debugging so 
> probably slightly higher default will save the trouble. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19488) Remove Unused Code from CollectionUtils

[BELUGA BEHR (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

BELUGA BEHR updated HBASE-19488:

Status: Patch Available  (was: Open)

> Remove Unused Code from CollectionUtils
> ---
>
> Key: HBASE-19488
> URL: https://issues.apache.org/jira/browse/HBASE-19488
> Project: HBase
>  Issue Type: Improvement
>  Components: hbase
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HBASE-19488.1.patch, HBASE-19488.2.patch, 
> HBASE-19488.3.patch, HBASE-19488.4.patch
>
>
> A bunch of unused code in CollectionUtils or code that can be found in Apache 
> Commons libraries.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19488) Remove Unused Code from CollectionUtils

[BELUGA BEHR (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

BELUGA BEHR updated HBASE-19488:

Attachment: HBASE-19488.4.patch

> Remove Unused Code from CollectionUtils
> ---
>
> Key: HBASE-19488
> URL: https://issues.apache.org/jira/browse/HBASE-19488
> Project: HBase
>  Issue Type: Improvement
>  Components: hbase
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HBASE-19488.1.patch, HBASE-19488.2.patch, 
> HBASE-19488.3.patch, HBASE-19488.4.patch
>
>
> A bunch of unused code in CollectionUtils or code that can be found in Apache 
> Commons libraries.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19488) Remove Unused Code from CollectionUtils

[BELUGA BEHR (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

BELUGA BEHR updated HBASE-19488:

Status: Open  (was: Patch Available)

> Remove Unused Code from CollectionUtils
> ---
>
> Key: HBASE-19488
> URL: https://issues.apache.org/jira/browse/HBASE-19488
> Project: HBase
>  Issue Type: Improvement
>  Components: hbase
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HBASE-19488.1.patch, HBASE-19488.2.patch, 
> HBASE-19488.3.patch, HBASE-19488.4.patch
>
>
> A bunch of unused code in CollectionUtils or code that can be found in Apache 
> Commons libraries.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19849) MetaEntry in HBaseFsck has inconsistent equals and hashCode

[Peter Somogyi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Somogyi updated HBASE-19849:
--
Affects Version/s: (was: 2.0.0-beta-1)
   1.4.3

> MetaEntry in HBaseFsck has inconsistent equals and hashCode
> ---
>
> Key: HBASE-19849
> URL: https://issues.apache.org/jira/browse/HBASE-19849
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.4.3
>Reporter: Peter Somogyi
>Priority: Major
>
> [~balazs.meszaros] noticed that MetaEntry uses different fields in equals and 
> hashCode. The used fields should be unified.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20188) [TESTING] Performance

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423788#comment-16423788
 ] 

ramkrishna.s.vasudevan commented on HBASE-20188:


The previous results were not apple to apple because offheap bucket cache does 
an onheap copy for hbase-1.2. So tried out the same experiment for a longer 
duration with L1 block cache and the avg was 

hbase -1.2 vs hbase -2 was 32393 ops/sec vs 28898 ops/sec. 

So we have a 4k difference still. In Stack's case it is much higher. Will try 
to figure out if there are any obvious bottlenecks that can be resolved to make 
up for this difference.

> [TESTING] Performance
> -
>
> Key: HBASE-20188
> URL: https://issues.apache.org/jira/browse/HBASE-20188
> Project: HBase
>  Issue Type: Umbrella
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: CAM-CONFIG-V01.patch, ITBLL2.5B_1.2.7vs2.0.0_cpu.png, 
> ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, 
> ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, 
> ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, 
> ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, 
> YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, 
> YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, 
> flamegraph-1072.1.svg, flamegraph-1072.2.svg, tree.txt
>
>
> How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor 
> that it is much slower, that the problem is the asyncwal writing. Does 
> in-memory compaction slow us down or speed us up? What happens when you 
> enable offheaping?
> Keep notes here in this umbrella issue. Need to be able to say something 
> about perf when 2.0.0 ships.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19488) Remove Unused Code from CollectionUtils

[BELUGA BEHR (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423980#comment-16423980
 ] 

BELUGA BEHR commented on HBASE-19488:
-

[~appy] I just threw a new patch up.

> Remove Unused Code from CollectionUtils
> ---
>
> Key: HBASE-19488
> URL: https://issues.apache.org/jira/browse/HBASE-19488
> Project: HBase
>  Issue Type: Improvement
>  Components: hbase
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HBASE-19488.1.patch, HBASE-19488.2.patch, 
> HBASE-19488.3.patch, HBASE-19488.4.patch
>
>
> A bunch of unused code in CollectionUtils or code that can be found in Apache 
> Commons libraries.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20188) [TESTING] Performance

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423581#comment-16423581
 ] 

ramkrishna.s.vasudevan commented on HBASE-20188:


bq.None really. Tried with data cached thinking it i/o that was responsible for 
the difference, but while it brings us closer, we are still down from 
hbase1.2.7 (52221.77 ops/second vs 33839.57).

This is really surprising. Even after data cache you see hbase-2 to be so much 
slower? All are defaults in both cases? What about bloom filters? What is the 
cache size here? And you sure all data is served only from cache?

> [TESTING] Performance
> -
>
> Key: HBASE-20188
> URL: https://issues.apache.org/jira/browse/HBASE-20188
> Project: HBase
>  Issue Type: Umbrella
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: CAM-CONFIG-V01.patch, ITBLL2.5B_1.2.7vs2.0.0_cpu.png, 
> ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, 
> ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, 
> ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, 
> ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, 
> YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, 
> YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, 
> flamegraph-1072.1.svg, flamegraph-1072.2.svg, tree.txt
>
>
> How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor 
> that it is much slower, that the problem is the asyncwal writing. Does 
> in-memory compaction slow us down or speed us up? What happens when you 
> enable offheaping?
> Keep notes here in this umbrella issue. Need to be able to say something 
> about perf when 2.0.0 ships.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20188) [TESTING] Performance

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423585#comment-16423585
 ] 

ramkrishna.s.vasudevan commented on HBASE-20188:


I just reran the experiment with loading around 15G of data and loading 
everything to cache and doing pure read workloads. 

I used the hbase-1.2 client only with 100 threads (all single node)

hbase-1.2.6 vs hbase-2 (31099 ops/sec vs 30838 ops/sec). (Almost same). 

I have 15 regions in my table and there are no splits that happen. All 15G of 
data is served from bucket cache (offheap mode). The results are not so 
different even if serve from L1 block cache also. 

 

> [TESTING] Performance
> -
>
> Key: HBASE-20188
> URL: https://issues.apache.org/jira/browse/HBASE-20188
> Project: HBase
>  Issue Type: Umbrella
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: CAM-CONFIG-V01.patch, ITBLL2.5B_1.2.7vs2.0.0_cpu.png, 
> ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, 
> ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, 
> ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, 
> ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, 
> YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, 
> YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, 
> flamegraph-1072.1.svg, flamegraph-1072.2.svg, tree.txt
>
>
> How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor 
> that it is much slower, that the problem is the asyncwal writing. Does 
> in-memory compaction slow us down or speed us up? What happens when you 
> enable offheaping?
> Keep notes here in this umbrella issue. Need to be able to say something 
> about perf when 2.0.0 ships.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore

[Chance Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423591#comment-16423591
 ] 

Chance Li commented on HBASE-20312:
---

Thanks. [~carp84],[~stack]
It's my fault. I will fix it on next patch. now, to use CompactingMemsotre, We 
must set 
"hbase.regionserver.memstore.class"=org.apache.hadoop.hbase.regionserver.DefaultMemStore.
 That's terrible.

> CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
> --
>
> Key: HBASE-20312
> URL: https://issues.apache.org/jira/browse/HBASE-20312
> Project: HBase
>  Issue Type: New Feature
>  Components: regionserver
>Reporter: Xiang Wang
>Assignee: Chance Li
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20312-1.3.2.patch, HBASE-20312-master.v1.patch, 
> HBASE-20312-master.v2.patch, HBASE-20312-master.v3.patch, 
> ccsmap-branch-1.1.patch, jira1.png, jira2.png, jira3.png
>
>
> Now hbase use ConcurrentSkipListMap as memstore's data structure.
> Although MemSLAB reduces memory fragment brought by key-value pairs.
> Hundred of millions key-value pairs still make young generation 
> garbage-collection(gc) stop time long.
>  
> These are 2 gc problems of ConcurrentSkipListMap:
> 1. HBase needs 3 objects to store one key-value on expectation. One 
> Index(skiplist's average node height is 1), one Node, and one KeyValue. Too 
> many objects are created for memstore.
> 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned 
> on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc 
> algorithm) will change frequently on high writing throughput, which makes YGC 
> slow.
>  
> We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap 
> for short),
> which provides similary features like ConcurrentSkipListMap but get rid of 
> Objects for every key-value pairs.
> CCSMap's memory structure is like this picture:
> !jira1.png!
>  
> One CCSMap consists of a certain number of Chunks. One Chunk consists of a 
> certain number of nodes. One node is corresspding one element. This element's 
> all information and its key-value is encoded on a continuous memory segment 
> without any objects.
> Features:
> 1. all insert,update,delete operations is lock-free on CCSMap.
> 2. Consume less memory, it brings 40% memory saving for 50Byte key-value.
> 3. Faster on small key-value because of better cacheline usage. 20~30% better 
> read/write troughput than ConcurrentSkipListMap for 50Byte key-value.
> CCSMap do not support recyle space when deleting element. But it doesn't 
> matter for hbase because of region flush.
> CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts 
> down YGC time significantly. here are 2 graph of before and after.
> !jira2.png!
> !jira3.png!
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20329) Add note for operators to refguide on AsyncFSWAL

[Yu Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423521#comment-16423521
 ] 

Yu Li commented on HBASE-20329:
---

+1, the doc looks great to me, thanks for writing this up boss.

Minor: WALs edits are written concurrentl -> WAL's edits are written 
concurrently

> Add note for operators to refguide on AsyncFSWAL
> 
>
> Key: HBASE-20329
> URL: https://issues.apache.org/jira/browse/HBASE-20329
> Project: HBase
>  Issue Type: Sub-task
>  Components: documentation, wal
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-20329.master.001.patch
>
>
> Need a few notes in refguide on this new facility.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20329) Add note for operators to refguide on AsyncFSWAL

[Peter Somogyi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423522#comment-16423522
 ] 

Peter Somogyi commented on HBASE-20329:
---

Good summary [~stack]!

nit: chained pipeline as the *default* *default* client does

> Add note for operators to refguide on AsyncFSWAL
> 
>
> Key: HBASE-20329
> URL: https://issues.apache.org/jira/browse/HBASE-20329
> Project: HBase
>  Issue Type: Sub-task
>  Components: documentation, wal
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-20329.master.001.patch
>
>
> Need a few notes in refguide on this new facility.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore

[Chance Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423528#comment-16423528
 ] 

Chance Li commented on HBASE-20312:
---

stack

{quote}regionserver.HStore: Memstore class name is 
org.apache.hadoop.hbase.regionserver.DefaultMemStore ;{quote} 

This log shows the Meta is DefaultMemstore. Which patch did you apply? is it 
HBASE-20312-master.v3.patch?

> CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
> --
>
> Key: HBASE-20312
> URL: https://issues.apache.org/jira/browse/HBASE-20312
> Project: HBase
>  Issue Type: New Feature
>  Components: regionserver
>Reporter: Xiang Wang
>Assignee: Chance Li
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20312-1.3.2.patch, HBASE-20312-master.v1.patch, 
> HBASE-20312-master.v2.patch, HBASE-20312-master.v3.patch, 
> ccsmap-branch-1.1.patch, jira1.png, jira2.png, jira3.png
>
>
> Now hbase use ConcurrentSkipListMap as memstore's data structure.
> Although MemSLAB reduces memory fragment brought by key-value pairs.
> Hundred of millions key-value pairs still make young generation 
> garbage-collection(gc) stop time long.
>  
> These are 2 gc problems of ConcurrentSkipListMap:
> 1. HBase needs 3 objects to store one key-value on expectation. One 
> Index(skiplist's average node height is 1), one Node, and one KeyValue. Too 
> many objects are created for memstore.
> 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned 
> on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc 
> algorithm) will change frequently on high writing throughput, which makes YGC 
> slow.
>  
> We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap 
> for short),
> which provides similary features like ConcurrentSkipListMap but get rid of 
> Objects for every key-value pairs.
> CCSMap's memory structure is like this picture:
> !jira1.png!
>  
> One CCSMap consists of a certain number of Chunks. One Chunk consists of a 
> certain number of nodes. One node is corresspding one element. This element's 
> all information and its key-value is encoded on a continuous memory segment 
> without any objects.
> Features:
> 1. all insert,update,delete operations is lock-free on CCSMap.
> 2. Consume less memory, it brings 40% memory saving for 50Byte key-value.
> 3. Faster on small key-value because of better cacheline usage. 20~30% better 
> read/write troughput than ConcurrentSkipListMap for 50Byte key-value.
> CCSMap do not support recyle space when deleting element. But it doesn't 
> matter for hbase because of region flush.
> CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts 
> down YGC time significantly. here are 2 graph of before and after.
> !jira2.png!
> !jira3.png!
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20256) Document commands that do not work with 1.2 shell

[Peter Somogyi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423541#comment-16423541
 ] 

Peter Somogyi commented on HBASE-20256:
---

[~busbey], are you good with patch 2?

> Document commands that do not work with 1.2 shell
> -
>
> Key: HBASE-20256
> URL: https://issues.apache.org/jira/browse/HBASE-20256
> Project: HBase
>  Issue Type: Sub-task
>  Components: documentation, shell
>Affects Versions: 2.0.0
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Major
> Attachments: HBASE-20256.master.001.patch, 
> HBASE-20256.master.002.patch
>
>
> Some commands do not work from 1.2 shell when running against 2.0 server. Add 
> a section to the reference guide mentioning the incompatibilities.
> Some of these are collected in this document: 
> https://docs.google.com/document/d/1l2ad5G_GUk0WQ02xKeKO6mTbpdyZjU1NGuPPI42Wbf4/edit



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore

[Yu Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423551#comment-16423551
 ] 

Yu Li commented on HBASE-20312:
---

I guess you didn't turn off CompactingMemsotre (set 
"hbase.hregion.compacting.memstore.type" to NONE) in your testing boss, could 
you check and retry if not? Thanks [~stack]

[~chancelq] We need to make it work when 
{{hbase.hregion.compacting.memstore.type}} is set to BASIC or some other not 
NONE type.

In current patch as long as {{hbase.regionserver.memstore.class}} is not set we 
will only initiate CCSMapChunkPool and MSLAB chunk creator won't get 
initialized, which will cause the NPE problem as stack saw in his testing. A 
naive fix might be adding check on {{hbase.hregion.compacting.memstore.type}} 
in {{CCSMapMemStore#isEnabled}} and only returns true if it's "NONE"

> CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
> --
>
> Key: HBASE-20312
> URL: https://issues.apache.org/jira/browse/HBASE-20312
> Project: HBase
>  Issue Type: New Feature
>  Components: regionserver
>Reporter: Xiang Wang
>Assignee: Chance Li
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20312-1.3.2.patch, HBASE-20312-master.v1.patch, 
> HBASE-20312-master.v2.patch, HBASE-20312-master.v3.patch, 
> ccsmap-branch-1.1.patch, jira1.png, jira2.png, jira3.png
>
>
> Now hbase use ConcurrentSkipListMap as memstore's data structure.
> Although MemSLAB reduces memory fragment brought by key-value pairs.
> Hundred of millions key-value pairs still make young generation 
> garbage-collection(gc) stop time long.
>  
> These are 2 gc problems of ConcurrentSkipListMap:
> 1. HBase needs 3 objects to store one key-value on expectation. One 
> Index(skiplist's average node height is 1), one Node, and one KeyValue. Too 
> many objects are created for memstore.
> 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned 
> on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc 
> algorithm) will change frequently on high writing throughput, which makes YGC 
> slow.
>  
> We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap 
> for short),
> which provides similary features like ConcurrentSkipListMap but get rid of 
> Objects for every key-value pairs.
> CCSMap's memory structure is like this picture:
> !jira1.png!
>  
> One CCSMap consists of a certain number of Chunks. One Chunk consists of a 
> certain number of nodes. One node is corresspding one element. This element's 
> all information and its key-value is encoded on a continuous memory segment 
> without any objects.
> Features:
> 1. all insert,update,delete operations is lock-free on CCSMap.
> 2. Consume less memory, it brings 40% memory saving for 50Byte key-value.
> 3. Faster on small key-value because of better cacheline usage. 20~30% better 
> read/write troughput than ConcurrentSkipListMap for 50Byte key-value.
> CCSMap do not support recyle space when deleting element. But it doesn't 
> matter for hbase because of region flush.
> CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts 
> down YGC time significantly. here are 2 graph of before and after.
> !jira2.png!
> !jira3.png!
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20244) NoSuchMethodException when retrieving private method decryptEncryptedDataEncryptionKey from DFSClient

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-20244:
---
Attachment: 20244.v1.txt

> NoSuchMethodException when retrieving private method 
> decryptEncryptedDataEncryptionKey from DFSClient
> -
>
> Key: HBASE-20244
> URL: https://issues.apache.org/jira/browse/HBASE-20244
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20244.v1.txt, 20244.v1.txt, 20244.v1.txt
>
>
> I was running unit test against hadoop 3.0.1 RC and saw the following in test 
> output:
> {code}
> ERROR [RS-EventLoopGroup-3-3] 
> asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(267): Couldn't properly 
> initialize access to HDFS internals. Please update  your WAL Provider to not 
> make use of the 'asyncfs' provider. See HBASE-16110 for more information.
> java.lang.NoSuchMethodException: 
> org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo)
>   at java.lang.Class.getDeclaredMethod(Class.java:2130)
>   at 
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:232)
>   at 
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:262)
>   at 
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:661)
>   at 
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:118)
>   at 
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:720)
>   at 
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:715)
>   at 
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
>   at 
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)
>   at 
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479)
>   at 
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
>   at 
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
>   at 
> org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
>   at 
> org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:306)
>   at 
> org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:341)
>   at 
> org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633)
>   at 
> org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
> {code}
> The private method was moved by HDFS-12574 to HdfsKMSUtil with different 
> signature.
> To accommodate the above method movement, it seems we need to call the 
> following method of DFSClient :
> {code}
>   public KeyProvider getKeyProvider() throws IOException {
> {code}
> Since the new decryptEncryptedDataEncryptionKey method has this signature:
> {code}
>   static KeyVersion decryptEncryptedDataEncryptionKey(FileEncryptionInfo
> feInfo, KeyProvider keyProvider) throws IOException {
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-20333) break up shaded client into one with no Hadoop and one that's standalone

[Sean Busbey (JIRA)]

Sean Busbey created HBASE-20333:
---

 Summary: break up shaded client into one with no Hadoop and one 
that's standalone
 Key: HBASE-20333
 URL: https://issues.apache.org/jira/browse/HBASE-20333
 Project: HBase
  Issue Type: Sub-task
  Components: shading
Affects Versions: 2.0.0
Reporter: Sean Busbey
Assignee: Sean Busbey
 Fix For: 2.0.0


there are contexts where we want to stay out of our downstream users way wrt 
dependencies, but they need more Hadoop classes than we provide. i.e. any 
downstream client that wants to use both HBase and HDFS in their application, 
or any non-MR YARN application.

Now that Hadoop also has shaded client artifacts for Hadoop 3, we're also 
providing less incremental benefit by including our own rewritten Hadoop 
classes to avoid downstream needing to pull in all of Hadoop's transitive 
dependencies.

right now those users need to ensure that any jars from the Hadoop project are 
loaded in the classpath prior to our shaded client jar. This is brittle and 
prone to weird debugging trouble.

instead, we should have two artifacts: one that just lists Hadoop as a 
prerequisite and one that still includes the rewritten-but-not-relocated Hadoop 
classes.

We can then use docs to emphasize when each of these is appropriate to use.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20332) shaded mapreduce module shouldn't include hadoop

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424156#comment-16424156
 ] 

Sean Busbey commented on HBASE-20332:
-

I think the appropriate scope for the hadoop bits we need on this artifact is 
{{provided}}, given #1 and #2 above

> shaded mapreduce module shouldn't include hadoop
> 
>
> Key: HBASE-20332
> URL: https://issues.apache.org/jira/browse/HBASE-20332
> Project: HBase
>  Issue Type: Sub-task
>  Components: mapreduce, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Fix For: 2.0.0
>
>
> AFAICT, we should just entirely skip including hadoop in our shaded mapreduce 
> module
> 1) Folks expect to run yarn / mr apps via {{hadoop jar}} / {{yarn jar}}
> 2) those commands include all the needed Hadoop jars in your classpath by 
> default (both client side and in the containers)
> 3) If you try to use "user classpath first" for your job as a workaround 
> (e.g. for some library your application needs that hadoop provides) then our 
> inclusion of *some but not all* hadoop classes then causes everything to fall 
> over because of mixing rewritten and non-rewritten hadoop classes
> 4) if you don't use "user classpath first" then all of our 
> non-relocated-but-still-shaded hadoop classes are ignored anyways so we're 
> just wasting space



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HBASE-20332) shaded mapreduce module shouldn't include hadoop

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey reassigned HBASE-20332:
---

Assignee: Sean Busbey

> shaded mapreduce module shouldn't include hadoop
> 
>
> Key: HBASE-20332
> URL: https://issues.apache.org/jira/browse/HBASE-20332
> Project: HBase
>  Issue Type: Sub-task
>  Components: mapreduce, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Fix For: 2.0.0
>
>
> AFAICT, we should just entirely skip including hadoop in our shaded mapreduce 
> module
> 1) Folks expect to run yarn / mr apps via {{hadoop jar}} / {{yarn jar}}
> 2) those commands include all the needed Hadoop jars in your classpath by 
> default (both client side and in the containers)
> 3) If you try to use "user classpath first" for your job as a workaround 
> (e.g. for some library your application needs that hadoop provides) then our 
> inclusion of *some but not all* hadoop classes then causes everything to fall 
> over because of mixing rewritten and non-rewritten hadoop classes
> 4) if you don't use "user classpath first" then all of our 
> non-relocated-but-still-shaded hadoop classes are ignored anyways so we're 
> just wasting space



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20305) Add option to SyncTable that skip deletes on target cluster

[Wellington Chevreuil (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424058#comment-16424058
 ] 

Wellington Chevreuil commented on HBASE-20305:
--

Thanks for the review and suggestions [~davelatham]! Indeed, *doDeletes* sounds 
more intuitive. I'm gonna work on this and the additional *doPuts,* together 
with reviews for checkstyle violations.

> Add option to SyncTable that skip deletes on target cluster
> ---
>
> Key: HBASE-20305
> URL: https://issues.apache.org/jira/browse/HBASE-20305
> Project: HBase
>  Issue Type: Improvement
>  Components: mapreduce
>Affects Versions: 2.0.0-alpha-4
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Attachments: 0001-HBASE-20305.master.001.patch
>
>
> We had a situation where two clusters with active-active replication got out 
> of sync, but both had data that should be kept. The tables in question never 
> have data deleted, but ingestion had happened on the two different clusters, 
> some rows had been even updated.
> In this scenario, a cell that is present in one of the table clusters should 
> not be deleted, but replayed on the other. Also, for cells with same 
> identifier but different values, the most recent value should be kept. 
> Current version of SyncTable would not be applicable here, because it would 
> simply copy the whole state from source to target, then losing any additional 
> rows that might be only in target, as well as cell values that got most 
> recent update. This could be solved by adding an option to skip deletes for 
> SyncTable. This way, the additional cells not present on source would still 
> be kept. For cells with same identifier but different values, it would just 
> perform a Put for the cell version from source, but client scans would still 
> fetch the most recent timestamp.
> I'm attaching a patch with this additional option shortly. Please share your 
> thoughts.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20258) Shell hangs when scanning a disabled table

[Balazs Meszaros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Balazs Meszaros updated HBASE-20258:

Attachment: (was: HBASE-20258.master.002.patch)

> Shell hangs when scanning a disabled table
> --
>
> Key: HBASE-20258
> URL: https://issues.apache.org/jira/browse/HBASE-20258
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Balazs Meszaros
>Assignee: Balazs Meszaros
>Priority: Major
> Fix For: 3.0.0, 2.0.0
>
> Attachments: HBASE-20258.branch-2.001.patch, 
> HBASE-20258.master.001.patch
>
>
> I executed the following commands against a 2.0 server:
> {code}
> disable 't'
> scan 't'
> {code}
> From client-1.2: it throws an error because the table is disabled -> OK.
> From client-2.0: the shell hangs -> NOT OK.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20258) Shell hangs when scanning a disabled table

[Balazs Meszaros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Balazs Meszaros updated HBASE-20258:

Attachment: (was: HBASE-20258.branch-2.002.patch)

> Shell hangs when scanning a disabled table
> --
>
> Key: HBASE-20258
> URL: https://issues.apache.org/jira/browse/HBASE-20258
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Balazs Meszaros
>Assignee: Balazs Meszaros
>Priority: Major
> Fix For: 3.0.0, 2.0.0
>
> Attachments: HBASE-20258.branch-2.001.patch, 
> HBASE-20258.master.001.patch
>
>
> I executed the following commands against a 2.0 server:
> {code}
> disable 't'
> scan 't'
> {code}
> From client-1.2: it throws an error because the table is disabled -> OK.
> From client-2.0: the shell hangs -> NOT OK.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20258) Shell hangs when scanning a disabled table

[Balazs Meszaros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Balazs Meszaros updated HBASE-20258:

Attachment: HBASE-20258.master.002.patch
HBASE-20258.branch-2.002.patch

> Shell hangs when scanning a disabled table
> --
>
> Key: HBASE-20258
> URL: https://issues.apache.org/jira/browse/HBASE-20258
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Balazs Meszaros
>Assignee: Balazs Meszaros
>Priority: Major
> Fix For: 3.0.0, 2.0.0
>
> Attachments: HBASE-20258.branch-2.001.patch, 
> HBASE-20258.branch-2.002.patch, HBASE-20258.master.001.patch, 
> HBASE-20258.master.002.patch
>
>
> I executed the following commands against a 2.0 server:
> {code}
> disable 't'
> scan 't'
> {code}
> From client-1.2: it throws an error because the table is disabled -> OK.
> From client-2.0: the shell hangs -> NOT OK.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19852) HBase Thrift 1 server SPNEGO Improvements

[Kevin Risden (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Risden updated HBASE-19852:
-
Attachment: HBASE-19852.master.008.patch

> HBase Thrift 1 server SPNEGO Improvements
> -
>
> Key: HBASE-19852
> URL: https://issues.apache.org/jira/browse/HBASE-19852
> Project: HBase
>  Issue Type: Improvement
>  Components: Thrift
>Reporter: Kevin Risden
>Assignee: Kevin Risden
>Priority: Major
> Attachments: HBASE-19852.master.001.patch, 
> HBASE-19852.master.002.patch, HBASE-19852.master.003.patch, 
> HBASE-19852.master.004.patch, HBASE-19852.master.006.patch, 
> HBASE-19852.master.007.patch.txt, HBASE-19852.master.008.patch
>
>
> HBase Thrift1 server has some issues when trying to use SPNEGO.
> From mailing list:
> http://mail-archives.apache.org/mod_mbox/hbase-user/201801.mbox/%3CCAJU9nmh5YtZ%2BmAQSLo91yKm8pRVzAPNLBU9vdVMCcxHRtRqgoA%40mail.gmail.com%3E
> {quote}While setting up the HBase Thrift server with HTTP, there were a
> significant amount of 401 errors where the HBase Thrift wasn't able to
> handle the incoming Kerberos request. Documentation online is sparse when
> it comes to setting up the principal/keytab for HTTP Kerberos.
> I noticed that the HBase Thrift HTTP implementation was missing SPNEGO
> principal/keytab like other Thrift based servers (HiveServer2). It looks
> like HiveServer2 Thrift implementation and HBase Thrift v1 implementation
> were very close to the same at one point. I made the following changes to
> HBase Thrift v1 server implementation to make it work:
> * add SPNEGO principal/keytab if in HTTP mode
> * return 401 immediately if no authorization header instead of waiting for
> try/catch down in program flow{quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20253) Error message is missing for restore_snapshot

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-20253:

Status: In Progress  (was: Patch Available)

I believe we still need to re-throw the exception even with the improved 
message. Otherwise, the shell will claim that the operation succeeded because 
the {{puts}} succeeds and is the last operation.

example prior to this patch:

{code}
$ hbase shell --noninteractive < create 'test:some_table', 'family'
> snapshot 'test:some_table', 'example_snapshot'
> restore_snapshot 'example_snapshot'
> EOF
Created table test:some_table
Took 1.1634 seconds 
 
Hbase::Table - test:some_table
Took 0.5606 seconds 
 

Took 0.1246 seconds 
 java exception
ERROR Java::OrgApacheHadoopHbase::TableNotDisabledException: test:some_table
[root@hbase19158-1 ~]# echo $?
1
{code}

example with this patch:

{code}
$ hbase shell --noninteractive < create 'test:some_table', 'family'
> snapshot 'test:some_table', 'example_snapshot'
> restore_snapshot 'example_snapshot'
> EOF
Created table test:some_table
Took 1.1208 seconds 
 
Hbase::Table - test:some_table
Took 0.4879 seconds 
 

Table 'test:some_table' should be disabled to restore snapshot.
Took 0.1080 seconds 
 

[root@hbase19158-1 ~]# echo $?
0
{code}

It'd be good to add a test that checks this negative example while we're here.

> Error message is missing for restore_snapshot
> -
>
> Key: HBASE-20253
> URL: https://issues.apache.org/jira/browse/HBASE-20253
> Project: HBase
>  Issue Type: Sub-task
>  Components: shell
>Affects Versions: 2.0.0
>Reporter: Peter Somogyi
>Assignee: Gabor Bota
>Priority: Minor
> Attachments: HBASE-20253.master.001.patch, 
> HBASE-20253.master.002.patch
>
>
> When the table is not disabled and restore_snapshot executed the error 
> message is useless, only displays the table name.
> hbase(main):007:0> restore_snapshot 'tsnap'
> ERROR: t
> Restore a specified snapshot.
> The restore will replace the content of the original table,
> bringing back the content to the snapshot state.
> The table must be disabled.
> Examples:
>   hbase> restore_snapshot 'snapshotName'
> Following command will restore all acl from snapshot table into the table.
>   hbase> restore_snapshot 'snapshotName', \{RESTORE_ACL=>true}
> Took 0.1044 seconds



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20276) [shell] confirm shell REPL change and document

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424068#comment-16424068
 ] 

Sean Busbey commented on HBASE-20276:
-

[I put up a DISCUSS thread pointing folks over 
here|https://lists.apache.org/thread.html/23cf8f132b0b87d34f14c5c7dee2a8458ce89af69e3bf0e9fda9e61a@%3Cdev.hbase.apache.org%3E].

I think we already have enough consensus to get a patch up for 1.5+ I'm happy 
to either get the patch together or review. I'll do the former after I wrap up 
some other stuff, or the latter if someone else has time to do it sooner.

FWIW, I think we should keep the --return-values flag because it's been in a 
release. we should just have it print a message about how the change to not 
return values has been removed.

> [shell] confirm shell REPL change and document
> --
>
> Key: HBASE-20276
> URL: https://issues.apache.org/jira/browse/HBASE-20276
> Project: HBase
>  Issue Type: Sub-task
>  Components: documentation, shell
>Affects Versions: 1.4.0, 2.0.0
>Reporter: Sean Busbey
>Priority: Blocker
> Fix For: 1.4.4, 2.0.0
>
>
> Feedback from [~mdrob] on HBASE-19158:
> {quote}
> Shell:
> HBASE-19770. There was another issue opened where this was identified as a 
> problem so maybe the shape will change further, but I can't find it now.
> {quote}
> New commentary from [~busbey]:
> This was a follow on to HBASE-15965. That change effectively makes it so none 
> of our ruby wrappers can be used to build expressions in an interactive REPL. 
> This is a pretty severe change (most of my tips on HBASE-15611 will break, I 
> think).
> I think we should
> a) Have a DISCUSS thread, spanning dev@ and user@
> b) based on the outcome of that thread, either default to the new behavior or 
> the old behavior
> c) if we keep the HBASE-15965 behavior as  the default, flag it as 
> incompatible, call it out in the hbase 2.0 upgrade section, and update docs 
> (two examples: the output in the shell_exercises sections would be wrong, and 
> the _table_variables section won't work)
> d) In either case document the new flag in the ref guide



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-20332) shaded mapreduce module shouldn't include hadoop

[Sean Busbey (JIRA)]

Sean Busbey created HBASE-20332:
---

 Summary: shaded mapreduce module shouldn't include hadoop
 Key: HBASE-20332
 URL: https://issues.apache.org/jira/browse/HBASE-20332
 Project: HBase
  Issue Type: Sub-task
  Components: mapreduce, shading
Affects Versions: 2.0.0
Reporter: Sean Busbey
 Fix For: 2.0.0


AFAICT, we should just entirely skip including hadoop in our shaded mapreduce 
module

1) Folks expect to run yarn / mr apps via {{hadoop jar}} / {{yarn jar}}
2) those commands include all the needed Hadoop jars in your classpath by 
default (both client side and in the containers)
3) If you try to use "user classpath first" for your job as a workaround (e.g. 
for some library your application needs that hadoop provides) then our 
inclusion of *some but not all* hadoop classes then causes everything to fall 
over because of mixing rewritten and non-rewritten hadoop classes
4) if you don't use "user classpath first" then all of our 
non-relocated-but-still-shaded hadoop classes are ignored anyways so we're just 
wasting space



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20258) Shell hangs when scanning a disabled table

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424066#comment-16424066
 ] 

Hadoop QA commented on HBASE-20258:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
39s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} rubocop {color} | {color:red}  0m  
5s{color} | {color:red} The patch generated 4 new + 25 unchanged - 4 fixed = 29 
total (was 29) {color} |
| {color:red}-1{color} | {color:red} ruby-lint {color} | {color:red}  0m  
1s{color} | {color:red} The patch generated 1 new + 14 unchanged - 0 fixed = 15 
total (was 14) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  7m 
24s{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
 9s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 38s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f |
| JIRA Issue | HBASE-20258 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12917368/HBASE-20258.master.002.patch
 |
| Optional Tests |  asflicense  javac  javadoc  unit  rubocop  ruby_lint  |
| uname | Linux 68b9ded4dae5 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 219625233c |
| maven | version: Apache Maven 3.5.3 
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_162 |
| rubocop | v0.54.0 |
| rubocop | 
https://builds.apache.org/job/PreCommit-HBASE-Build/12276/artifact/patchprocess/diff-patch-rubocop.txt
 |
| ruby-lint | v2.3.1 |
| ruby-lint | 
https://builds.apache.org/job/PreCommit-HBASE-Build/12276/artifact/patchprocess/diff-patch-ruby-lint.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/12276/testReport/ |
| Max. process+thread count | 2137 (vs. ulimit of 1) |
| modules | C: hbase-shell U: hbase-shell |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/12276/console |
| Powered by | Apache Yetus 0.7.0   http://yetus.apache.org |


This message was automatically generated.



> Shell hangs when scanning a disabled table
> --
>
> Key: HBASE-20258
> URL: https://issues.apache.org/jira/browse/HBASE-20258
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Balazs Meszaros
>Assignee: Balazs Meszaros
>Priority: Major
> Fix For: 3.0.0, 2.0.0
>
> Attachments: HBASE-20258.branch-2.001.patch, 
> HBASE-20258.branch-2.002.patch, HBASE-20258.master.001.patch, 
> HBASE-20258.master.002.patch
>
>
> I executed the following commands against a 2.0 server:
> {code}
> disable 't'
> scan 't'
> {code}
> From client-1.2: it throws an error because the table is disabled -> OK.
> From client-2.0: the shell hangs -> NOT OK.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19852) HBase Thrift 1 server SPNEGO Improvements

[Kevin Risden (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424071#comment-16424071
 ] 

Kevin Risden commented on HBASE-19852:
--

Looks like the precommit test failures were due to something else. Re-uploaded 
patch with correct naming convention.

Ping [~elserj] and [~carp84] - Would appreciate another review.

> HBase Thrift 1 server SPNEGO Improvements
> -
>
> Key: HBASE-19852
> URL: https://issues.apache.org/jira/browse/HBASE-19852
> Project: HBase
>  Issue Type: Improvement
>  Components: Thrift
>Reporter: Kevin Risden
>Assignee: Kevin Risden
>Priority: Major
> Attachments: HBASE-19852.master.001.patch, 
> HBASE-19852.master.002.patch, HBASE-19852.master.003.patch, 
> HBASE-19852.master.004.patch, HBASE-19852.master.006.patch, 
> HBASE-19852.master.007.patch.txt, HBASE-19852.master.008.patch
>
>
> HBase Thrift1 server has some issues when trying to use SPNEGO.
> From mailing list:
> http://mail-archives.apache.org/mod_mbox/hbase-user/201801.mbox/%3CCAJU9nmh5YtZ%2BmAQSLo91yKm8pRVzAPNLBU9vdVMCcxHRtRqgoA%40mail.gmail.com%3E
> {quote}While setting up the HBase Thrift server with HTTP, there were a
> significant amount of 401 errors where the HBase Thrift wasn't able to
> handle the incoming Kerberos request. Documentation online is sparse when
> it comes to setting up the principal/keytab for HTTP Kerberos.
> I noticed that the HBase Thrift HTTP implementation was missing SPNEGO
> principal/keytab like other Thrift based servers (HiveServer2). It looks
> like HiveServer2 Thrift implementation and HBase Thrift v1 implementation
> were very close to the same at one point. I made the following changes to
> HBase Thrift v1 server implementation to make it work:
> * add SPNEGO principal/keytab if in HTTP mode
> * return 401 immediately if no authorization header instead of waiting for
> try/catch down in program flow{quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20095) Redesign single instance pool in CleanerChore

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424133#comment-16424133
 ] 

Ted Yu commented on HBASE-20095:


Ping [~mdrob]

Please take a look at the addendum at your convenience.

> Redesign single instance pool in CleanerChore
> -
>
> Key: HBASE-20095
> URL: https://issues.apache.org/jira/browse/HBASE-20095
> Project: HBase
>  Issue Type: Improvement
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Critical
> Fix For: 3.0.0, 2.1.0
>
> Attachments: 20095.addendum, 20095.addendum-2.txt, 
> HBASE-20095.master.001.patch, HBASE-20095.master.002.patch, 
> HBASE-20095.master.003.patch, HBASE-20095.master.004.patch, 
> HBASE-20095.master.005.patch, HBASE-20095.master.006.patch, 
> HBASE-20095.master.007.patch, HBASE-20095.master.008.patch, 
> HBASE-20095.master.009.patch, HBASE-20095.master.010.patch, 
> HBASE-20095.master.011.patch, HBASE-20095.master.012.patch, 
> HBASE-20095.master.013.patch, HBASE-20095.master.014.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-20331) clean up shaded packaging for 2.0

[Sean Busbey (JIRA)]

Sean Busbey created HBASE-20331:
---

 Summary: clean up shaded packaging for 2.0
 Key: HBASE-20331
 URL: https://issues.apache.org/jira/browse/HBASE-20331
 Project: HBase
  Issue Type: Umbrella
  Components: Client, mapreduce, shading
Affects Versions: 2.0.0
Reporter: Sean Busbey
Assignee: Sean Busbey
 Fix For: 2.0.0


polishing pass on shaded modules for 2.0 based on trying to use them in more 
contexts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19890) Canary usage should document hbase.canary.sink.class config

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424119#comment-16424119
 ] 

Ted Yu commented on HBASE-19890:


lgtm

> Canary usage should document hbase.canary.sink.class config
> ---
>
> Key: HBASE-19890
> URL: https://issues.apache.org/jira/browse/HBASE-19890
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Peter Somogyi
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-19890.master.001.patch
>
>
> Canary#main uses config hbase.canary.sink.class to instantiate Sink class.
> The Sink instance affects creation of Monitor.
> In the refguide for Canary, hbase.canary.sink.class was not mentioned.
>  We should document this config.
> Additionally, we need to document that using the default sink is not 
> compatible with table parameters as input so the user must change it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19852) HBase Thrift 1 server SPNEGO Improvements

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424123#comment-16424123
 ] 

Hadoop QA commented on HBASE-19852:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  5m 
16s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
48s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
19m 22s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
59s{color} | {color:green} hbase-thrift in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
 8s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m  8s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f |
| JIRA Issue | HBASE-19852 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12917369/HBASE-19852.master.008.patch
 |
| Optional Tests |  asflicense  javac  javadoc  unit  shadedjars  hadoopcheck  
xml  compile  findbugs  hbaseanti  checkstyle  |
| uname | Linux ecdcf200bb7c 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 219625233c |
| maven | version: Apache Maven 3.5.3 
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC3 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/12277/testReport/ |
| Max. process+thread count | 1784 (vs. ulimit of 1) |
| modules | C: hbase-thrift U: hbase-thrift |
| Console output | 

[jira] [Commented] (HBASE-20258) Shell hangs when scanning a disabled table

[Balazs Meszaros (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424028#comment-16424028
 ] 

Balazs Meszaros commented on HBASE-20258:
-

.002 correct the failures, but they weren't checked.

> Shell hangs when scanning a disabled table
> --
>
> Key: HBASE-20258
> URL: https://issues.apache.org/jira/browse/HBASE-20258
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Balazs Meszaros
>Assignee: Balazs Meszaros
>Priority: Major
> Fix For: 3.0.0, 2.0.0
>
> Attachments: HBASE-20258.branch-2.001.patch, 
> HBASE-20258.master.001.patch
>
>
> I executed the following commands against a 2.0 server:
> {code}
> disable 't'
> scan 't'
> {code}
> From client-1.2: it throws an error because the table is disabled -> OK.
> From client-2.0: the shell hangs -> NOT OK.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20332) shaded mapreduce module shouldn't include hadoop

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424152#comment-16424152
 ] 

Sean Busbey commented on HBASE-20332:
-

re #1, you currently can't use our shaded mapreduce jar to submit MR 
applications directly without the use of the {{hadoop jar}} / {{yarn jar}} 
commands, because we need some Hadoop common bits for dealing with the local 
filesystem. so if we want a standalone jar for e.g. submitting MR jobs from a 
node that has no Hadoop installation, that's a different pile of work (and I'd 
argue a less common case that we should handle after making sure we have a 
simple "easy" path)

> shaded mapreduce module shouldn't include hadoop
> 
>
> Key: HBASE-20332
> URL: https://issues.apache.org/jira/browse/HBASE-20332
> Project: HBase
>  Issue Type: Sub-task
>  Components: mapreduce, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Priority: Critical
> Fix For: 2.0.0
>
>
> AFAICT, we should just entirely skip including hadoop in our shaded mapreduce 
> module
> 1) Folks expect to run yarn / mr apps via {{hadoop jar}} / {{yarn jar}}
> 2) those commands include all the needed Hadoop jars in your classpath by 
> default (both client side and in the containers)
> 3) If you try to use "user classpath first" for your job as a workaround 
> (e.g. for some library your application needs that hadoop provides) then our 
> inclusion of *some but not all* hadoop classes then causes everything to fall 
> over because of mixing rewritten and non-rewritten hadoop classes
> 4) if you don't use "user classpath first" then all of our 
> non-relocated-but-still-shaded hadoop classes are ignored anyways so we're 
> just wasting space



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20298) Doc change in read/write/total accounting metrics

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-20298:
--
Attachment: HBASE-20298.master.003.patch

> Doc change in read/write/total accounting metrics
> -
>
> Key: HBASE-20298
> URL: https://issues.apache.org/jira/browse/HBASE-20298
> Project: HBase
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-20298.master.001.patch, 
> HBASE-20298.master.002.patch, HBASE-20298.master.003.patch
>
>
> Doc the change wrought by the parent issue. Get it up into the refguide as 
> part of the difference between old hbases and hbase2.
> The change confused me and took me a while to untangle.
> The read count is for reads that return a non-empty result now. In old 
> hbase1, we'd increment the read-count even if an empty result. This makes 
> reads look bad in YCSB runs when compared to hbase1 (see how 
> totalRequestCount in hbase2 can be way above the sum of reads+writes; it is 
> because it increments even if the row is not found).
> Let me get this into refguide otherwise poor old operators will be baffled. 
> The release note on the parent is great; it just needs to be in our guide.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20335) nightly jobs no longer contain machine information

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-20335:

Status: Patch Available  (was: In Progress)

wild guess based on looking at some console output

-0
  - make sure any step that uses inline bash will fail if something fails, e.g. 
the output dir creation
  - make sure machine stat gathering script will complain loudly if its 
directory isn't writable

> nightly jobs no longer contain machine information
> --
>
> Key: HBASE-20335
> URL: https://issues.apache.org/jira/browse/HBASE-20335
> Project: HBase
>  Issue Type: Bug
>  Components: test
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Attachments: HBASE-20335.0.patch
>
>
> something is up with nightly jobs. they no longer have the machine 
> information from HBASE-19228.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20335) nightly jobs no longer contain machine information

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424318#comment-16424318
 ] 

Hadoop QA commented on HBASE-20335:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  2m 
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue}  0m  
3s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green}  0m 
 1s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}  2m 51s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f |
| JIRA Issue | HBASE-20335 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12917403/HBASE-20335.0.patch |
| Optional Tests |  asflicense  shellcheck  shelldocs  |
| uname | Linux dfe1e009cfa2 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 
13:48:03 UTC 2016 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
 |
| git revision | master / bf29a1fee9 |
| maven | version: Apache Maven 3.5.3 
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| shellcheck | v0.4.4 |
| Max. process+thread count | 48 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/12281/console |
| Powered by | Apache Yetus 0.7.0   http://yetus.apache.org |


This message was automatically generated.



> nightly jobs no longer contain machine information
> --
>
> Key: HBASE-20335
> URL: https://issues.apache.org/jira/browse/HBASE-20335
> Project: HBase
>  Issue Type: Bug
>  Components: test
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Attachments: HBASE-20335.0.patch
>
>
> something is up with nightly jobs. they no longer have the machine 
> information from HBASE-19228.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20329) Add note for operators to refguide on AsyncFSWAL

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-20329:
--
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: (was: 2.0.0)
   3.0.0
   Status: Resolved  (was: Patch Available)

Pushed to master branch. I'll pull it back to branch-2 and branch-2.0 when I 
copy the refguide from master to the release branch. Thanks for reviews.

> Add note for operators to refguide on AsyncFSWAL
> 
>
> Key: HBASE-20329
> URL: https://issues.apache.org/jira/browse/HBASE-20329
> Project: HBase
>  Issue Type: Sub-task
>  Components: documentation, wal
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HBASE-20329.master.001.patch
>
>
> Need a few notes in refguide on this new facility.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20193) Basic Replication Web UI - Regionserver

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424228#comment-16424228
 ] 

Sean Busbey commented on HBASE-20193:
-

thanks for the patch [~tianjingyun]! I'll take a look at the combination of 
this and HBASE-20194 later this week. Before I can do that I have to clear out 
some blockers on the upcoming 2.0 release.

Anything you can do to review 2.0 blockers or to clean up the javac warning 
will make reviewing faster / easier. (I get that your patch may not have 
introduced the javac warning, but I'll still have to chase down who did and/or 
fix it before committing.)

> Basic Replication Web UI - Regionserver 
> 
>
> Key: HBASE-20193
> URL: https://issues.apache.org/jira/browse/HBASE-20193
> Project: HBase
>  Issue Type: Sub-task
>  Components: Replication, Usability
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Critical
> Fix For: 2.1.0
>
> Attachments: HBASE-20193.master.001.patch, 
> HBASE-20193.master.002.patch, HBASE-20193.master.003.patch, 
> HBASE-20193.master.004.patch
>
>
> subtask of HBASE-15809. Implementation of replication UI on Regionserver web 
> page.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-16499) slow replication for small HBase clusters

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424167#comment-16424167
 ] 

Hadoop QA commented on HBASE-16499:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  6m 
 2s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
55s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
20m 20s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}117m 
51s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
24s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}163m 54s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f |
| JIRA Issue | HBASE-16499 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12917354/HBASE-16499.patch |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux 8b3fa7e43440 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 219625233c |
| maven | version: Apache Maven 3.5.3 
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC3 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/12274/testReport/ |
| Max. process+thread count | 4181 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/12274/console |
| Powered by | Apache Yetus 0.7.0   http://yetus.apache.org |


This message was automatically generated.



> slow replication for small HBase 

[jira] [Commented] (HBASE-20329) Add note for operators to refguide on AsyncFSWAL

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424195#comment-16424195
 ] 

stack commented on HBASE-20329:
---

Thanks boys. Let me add in your fixes on push.

> Add note for operators to refguide on AsyncFSWAL
> 
>
> Key: HBASE-20329
> URL: https://issues.apache.org/jira/browse/HBASE-20329
> Project: HBase
>  Issue Type: Sub-task
>  Components: documentation, wal
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-20329.master.001.patch
>
>
> Need a few notes in refguide on this new facility.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19488) Remove Unused Code from CollectionUtils

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424223#comment-16424223
 ] 

Hadoop QA commented on HBASE-19488:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  3m 
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  7m 
19s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m  
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  3m  
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  3m  
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
27s{color} | {color:green} hbase-common: The patch generated 0 new + 89 
unchanged - 1 fixed = 89 total (was 90) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
30s{color} | {color:green} The patch hbase-client passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
11s{color} | {color:green} The patch hbase-replication passed checkstyle 
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
17s{color} | {color:green} hbase-server: The patch generated 0 new + 294 
unchanged - 1 fixed = 294 total (was 295) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
57s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
20m 57s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
36s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
26s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
26s{color} | {color:green} hbase-replication in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}117m 
43s{color} | {color:green} hbase-server in the patch 

[jira] [Commented] (HBASE-20256) Document commands that do not work with 1.2 shell

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424230#comment-16424230
 ] 

Sean Busbey commented on HBASE-20256:
-

+1

> Document commands that do not work with 1.2 shell
> -
>
> Key: HBASE-20256
> URL: https://issues.apache.org/jira/browse/HBASE-20256
> Project: HBase
>  Issue Type: Sub-task
>  Components: documentation, shell
>Affects Versions: 2.0.0
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Major
> Attachments: HBASE-20256.master.001.patch, 
> HBASE-20256.master.002.patch
>
>
> Some commands do not work from 1.2 shell when running against 2.0 server. Add 
> a section to the reference guide mentioning the incompatibilities.
> Some of these are collected in this document: 
> https://docs.google.com/document/d/1l2ad5G_GUk0WQ02xKeKO6mTbpdyZjU1NGuPPI42Wbf4/edit



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20188) [TESTING] Performance

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424243#comment-16424243
 ] 

stack commented on HBASE-20188:
---

Thanks [~ram_krish] It helps to have independent observation. I'm doing default 
configs. and basic YCSB loadings. When all is cached, the two come closer but 
still a significant difference; not as near as you see (15%). What happens if 
reads are not all from cache in your setup? Currently looking at shortcircuit 
reading. We don't seem to be doing it my hbase2 runs compared to hbase1 runs. 
Will be back.

> [TESTING] Performance
> -
>
> Key: HBASE-20188
> URL: https://issues.apache.org/jira/browse/HBASE-20188
> Project: HBase
>  Issue Type: Umbrella
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: CAM-CONFIG-V01.patch, ITBLL2.5B_1.2.7vs2.0.0_cpu.png, 
> ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, 
> ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, 
> ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, 
> ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, 
> YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, 
> YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, 
> flamegraph-1072.1.svg, flamegraph-1072.2.svg, tree.txt
>
>
> How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor 
> that it is much slower, that the problem is the asyncwal writing. Does 
> in-memory compaction slow us down or speed us up? What happens when you 
> enable offheaping?
> Keep notes here in this umbrella issue. Need to be able to say something 
> about perf when 2.0.0 ships.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20298) Doc change in read/write/total accounting metrics

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-20298:
--
Attachment: HBASE-20298.master.004.patch

> Doc change in read/write/total accounting metrics
> -
>
> Key: HBASE-20298
> URL: https://issues.apache.org/jira/browse/HBASE-20298
> Project: HBase
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-20298.master.001.patch, 
> HBASE-20298.master.002.patch, HBASE-20298.master.003.patch, 
> HBASE-20298.master.004.patch
>
>
> Doc the change wrought by the parent issue. Get it up into the refguide as 
> part of the difference between old hbases and hbase2.
> The change confused me and took me a while to untangle.
> The read count is for reads that return a non-empty result now. In old 
> hbase1, we'd increment the read-count even if an empty result. This makes 
> reads look bad in YCSB runs when compared to hbase1 (see how 
> totalRequestCount in hbase2 can be way above the sum of reads+writes; it is 
> because it increments even if the row is not found).
> Let me get this into refguide otherwise poor old operators will be baffled. 
> The release note on the parent is great; it just needs to be in our guide.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-20336) org.apache.hadoop.hbase.spark.IntegrationTestSparkBulkLoad fails

[Sean Busbey (JIRA)]

Sean Busbey created HBASE-20336:
---

 Summary: 
org.apache.hadoop.hbase.spark.IntegrationTestSparkBulkLoad fails
 Key: HBASE-20336
 URL: https://issues.apache.org/jira/browse/HBASE-20336
 Project: HBase
  Issue Type: Bug
  Components: spark
Affects Versions: 3.0.0
Reporter: Sean Busbey
 Fix For: 3.0.0


running {{mvn verify}} for the spark integration tests against current master 
fails

{code}
$ mvn -DskipTests install
11:37:26,815 [INFO] Scanning for projects...
...
11:43:36,711 [INFO] 

11:43:36,711 [INFO] Reactor Summary:
11:43:36,711 [INFO] 
11:43:36,712 [INFO] Apache HBase ... 
SUCCESS [  6.294 s]
11:43:36,712 [INFO] Apache HBase - Checkstyle .. 
SUCCESS [  1.742 s]
11:43:36,712 [INFO] Apache HBase - Build Support ... 
SUCCESS [  0.070 s]
11:43:36,712 [INFO] Apache HBase - Error Prone Rules ... 
SUCCESS [  2.206 s]
11:43:36,712 [INFO] Apache HBase - Annotations . 
SUCCESS [  1.413 s]
11:43:36,712 [INFO] Apache HBase - Build Configuration . 
SUCCESS [  0.254 s]
11:43:36,712 [INFO] Apache HBase - Shaded Protocol . 
SUCCESS [ 37.870 s]
11:43:36,712 [INFO] Apache HBase - Common .. 
SUCCESS [ 12.526 s]
11:43:36,712 [INFO] Apache HBase - Metrics API . 
SUCCESS [  2.412 s]
11:43:36,712 [INFO] Apache HBase - Hadoop Compatibility  
SUCCESS [  3.260 s]
11:43:36,712 [INFO] Apache HBase - Metrics Implementation .. 
SUCCESS [  2.756 s]
11:43:36,713 [INFO] Apache HBase - Hadoop Two Compatibility  
SUCCESS [  3.959 s]
11:43:36,713 [INFO] Apache HBase - Protocol  
SUCCESS [ 11.295 s]
11:43:36,713 [INFO] Apache HBase - Client .. 
SUCCESS [ 15.360 s]
11:43:36,713 [INFO] Apache HBase - Zookeeper ... 
SUCCESS [  4.389 s]
11:43:36,713 [INFO] Apache HBase - Replication . 
SUCCESS [  4.202 s]
11:43:36,713 [INFO] Apache HBase - Resource Bundle . 
SUCCESS [  0.206 s]
11:43:36,713 [INFO] Apache HBase - HTTP  
SUCCESS [  8.530 s]
11:43:36,713 [INFO] Apache HBase - Procedure ... 
SUCCESS [  4.196 s]
11:43:36,713 [INFO] Apache HBase - Server .. 
SUCCESS [ 44.604 s]
11:43:36,713 [INFO] Apache HBase - MapReduce ... 
SUCCESS [ 11.122 s]
11:43:36,713 [INFO] Apache HBase - Testing Util  
SUCCESS [  6.633 s]
11:43:36,713 [INFO] Apache HBase - Thrift .. 
SUCCESS [  9.771 s]
11:43:36,713 [INFO] Apache HBase - RSGroup . 
SUCCESS [  6.703 s]
11:43:36,713 [INFO] Apache HBase - Shell ... 
SUCCESS [  7.094 s]
11:43:36,713 [INFO] Apache HBase - Coprocessor Endpoint  
SUCCESS [  7.542 s]
11:43:36,713 [INFO] Apache HBase - Backup .. 
SUCCESS [  6.246 s]
11:43:36,713 [INFO] Apache HBase - Integration Tests ... 
SUCCESS [  7.461 s]
11:43:36,713 [INFO] Apache HBase - Examples  
SUCCESS [  9.054 s]
11:43:36,713 [INFO] Apache HBase - Rest  
SUCCESS [  8.972 s]
11:43:36,713 [INFO] Apache HBase - External Block Cache  
SUCCESS [  5.180 s]
11:43:36,713 [INFO] Apache HBase - Spark ... 
SUCCESS [01:10 min]
11:43:36,713 [INFO] Apache HBase - Spark Integration Tests . 
SUCCESS [  7.774 s]
11:43:36,713 [INFO] Apache HBase - Assembly  
SUCCESS [ 15.075 s]
11:43:36,713 [INFO] Apache HBase - Shaded .. 
SUCCESS [  0.323 s]
11:43:36,713 [INFO] Apache HBase - Shaded - Client . 
SUCCESS [  3.004 s]
11:43:36,713 [INFO] Apache HBase - Shaded - MapReduce .. 
SUCCESS [  3.542 s]
11:43:36,714 [INFO] Apache HBase Shaded Packaging Invariants ... 
SUCCESS [  3.105 s]
11:43:36,714 [INFO] Apache HBase - Archetypes .. 
SUCCESS [  0.063 s]
11:43:36,714 [INFO] Apache HBase - Exemplar for hbase-client archetype . 
SUCCESS [  4.843 s]
11:43:36,714 [INFO] Apache HBase - Exemplar for hbase-shaded-client archetype 
SUCCESS [  4.523 s]
11:43:36,714 [INFO] Apache HBase - Archetype builder ... 
SUCCESS [  0.739 s]
11:43:36,714 [INFO] 

11:43:36,714 [INFO] BUILD SUCCESS
11:43:36,714 [INFO] 

11:43:36,714 [INFO] Total time: 06:09 min
11:43:36,714 [INFO] 

[jira] [Updated] (HBASE-20256) Document commands that do not work with 1.2 shell

[Peter Somogyi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Somogyi updated HBASE-20256:
--
Status: Patch Available  (was: In Progress)

> Document commands that do not work with 1.2 shell
> -
>
> Key: HBASE-20256
> URL: https://issues.apache.org/jira/browse/HBASE-20256
> Project: HBase
>  Issue Type: Sub-task
>  Components: documentation, shell
>Affects Versions: 2.0.0
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Major
> Attachments: HBASE-20256.master.001.patch, 
> HBASE-20256.master.002.patch, HBASE-20256.master.002.patch
>
>
> Some commands do not work from 1.2 shell when running against 2.0 server. Add 
> a section to the reference guide mentioning the incompatibilities.
> Some of these are collected in this document: 
> https://docs.google.com/document/d/1l2ad5G_GUk0WQ02xKeKO6mTbpdyZjU1NGuPPI42Wbf4/edit



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20256) Document commands that do not work with 1.2 shell

[Peter Somogyi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Somogyi updated HBASE-20256:
--
Status: In Progress  (was: Patch Available)

> Document commands that do not work with 1.2 shell
> -
>
> Key: HBASE-20256
> URL: https://issues.apache.org/jira/browse/HBASE-20256
> Project: HBase
>  Issue Type: Sub-task
>  Components: documentation, shell
>Affects Versions: 2.0.0
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Major
> Attachments: HBASE-20256.master.001.patch, 
> HBASE-20256.master.002.patch, HBASE-20256.master.002.patch
>
>
> Some commands do not work from 1.2 shell when running against 2.0 server. Add 
> a section to the reference guide mentioning the incompatibilities.
> Some of these are collected in this document: 
> https://docs.google.com/document/d/1l2ad5G_GUk0WQ02xKeKO6mTbpdyZjU1NGuPPI42Wbf4/edit



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20224) Web UI is broken in standalone mode

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-20224:
---
Attachment: 20224.addendum.6

> Web UI is broken in standalone mode
> ---
>
> Key: HBASE-20224
> URL: https://issues.apache.org/jira/browse/HBASE-20224
> Project: HBase
>  Issue Type: Bug
>  Components: UI, Usability
>Affects Versions: 2.0.0-beta-2
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: 
> 0001-HBASE-20224-Web-UI-is-broken-in-standalone-mode-ADDE.ADDENDUM.patch, 
> 20224-addendum.3.txt, 20224.addendum.4, 20224.addendum.5, 20224.addendum.6, 
> hbase-20224.master.001.patch, hbase-20224.master.002.patch, 
> hbase-20224.master.003.patch, hbase-20224.master.addendum.patch
>
>
> Web UI doesn't show up in standalone mode on default port. This can be seen 
> on master and branch-2.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20256) Document commands that do not work with 1.2 shell

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424231#comment-16424231
 ] 

Sean Busbey commented on HBASE-20256:
-

any idea why qabot hasn't shown up?

> Document commands that do not work with 1.2 shell
> -
>
> Key: HBASE-20256
> URL: https://issues.apache.org/jira/browse/HBASE-20256
> Project: HBase
>  Issue Type: Sub-task
>  Components: documentation, shell
>Affects Versions: 2.0.0
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Major
> Attachments: HBASE-20256.master.001.patch, 
> HBASE-20256.master.002.patch
>
>
> Some commands do not work from 1.2 shell when running against 2.0 server. Add 
> a section to the reference guide mentioning the incompatibilities.
> Some of these are collected in this document: 
> https://docs.google.com/document/d/1l2ad5G_GUk0WQ02xKeKO6mTbpdyZjU1NGuPPI42Wbf4/edit



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20256) Document commands that do not work with 1.2 shell

[Peter Somogyi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424239#comment-16424239
 ] 

Peter Somogyi commented on HBASE-20256:
---

No, I’ll retrigger it later.

> Document commands that do not work with 1.2 shell
> -
>
> Key: HBASE-20256
> URL: https://issues.apache.org/jira/browse/HBASE-20256
> Project: HBase
>  Issue Type: Sub-task
>  Components: documentation, shell
>Affects Versions: 2.0.0
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Major
> Attachments: HBASE-20256.master.001.patch, 
> HBASE-20256.master.002.patch
>
>
> Some commands do not work from 1.2 shell when running against 2.0 server. Add 
> a section to the reference guide mentioning the incompatibilities.
> Some of these are collected in this document: 
> https://docs.google.com/document/d/1l2ad5G_GUk0WQ02xKeKO6mTbpdyZjU1NGuPPI42Wbf4/edit



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20335) nightly jobs no longer contain machine information

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424257#comment-16424257
 ] 

Sean Busbey commented on HBASE-20335:
-

ugh, precommit job doesn't rely on the script for gathering machine stats. :(

> nightly jobs no longer contain machine information
> --
>
> Key: HBASE-20335
> URL: https://issues.apache.org/jira/browse/HBASE-20335
> Project: HBase
>  Issue Type: Bug
>  Components: test
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
>
> something is up with nightly jobs. they no longer have the machine 
> information from HBASE-19228.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20257) hbase-spark should not depend on com.google.code.findbugs.jsr305

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424284#comment-16424284
 ] 

Sean Busbey commented on HBASE-20257:
-

blast it. {{hbase-spark-it}} is busted. it fails before your patch as well, so 
I don't think it's something that was done here. but I don't want to commit 
without verifying that things work end-to-end with this in place.

filed HBASE-20336, haven't had a chance to triage it yet. probably will get to 
it last this week / early next week if no one else gets there first.

> hbase-spark should not depend on com.google.code.findbugs.jsr305
> 
>
> Key: HBASE-20257
> URL: https://issues.apache.org/jira/browse/HBASE-20257
> Project: HBase
>  Issue Type: Task
>  Components: build
>Affects Versions: 3.0.0
>Reporter: Ted Yu
>Assignee: Artem Ervits
>Priority: Minor
>  Labels: beginner
> Attachments: HBASE-20257.v01.patch, HBASE-20257.v02.patch, 
> HBASE-20257.v03.patch, HBASE-20257.v04.patch
>
>
> The following can be observed in the build output of master branch:
> {code}
> [WARNING] Rule 0: org.apache.maven.plugins.enforcer.BannedDependencies failed 
> with message:
> We don't allow the JSR305 jar from the Findbugs project, see HBASE-16321.
> Found Banned Dependency: com.google.code.findbugs:jsr305:jar:1.3.9
> Use 'mvn dependency:tree' to locate the source of the banned dependencies.
> {code}
> Here is related snippet from hbase-spark/pom.xml:
> {code}
> 
>   com.google.code.findbugs
>   jsr305
> {code}
> Dependency on jsr305 should be dropped.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-19890) Canary usage should document hbase.canary.sink.class config

[Peter Somogyi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Somogyi updated HBASE-19890:
--
Attachment: HBASE-19890.master.001.patch

> Canary usage should document hbase.canary.sink.class config
> ---
>
> Key: HBASE-19890
> URL: https://issues.apache.org/jira/browse/HBASE-19890
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Peter Somogyi
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-19890.master.001.patch, 
> HBASE-19890.master.001.patch
>
>
> Canary#main uses config hbase.canary.sink.class to instantiate Sink class.
> The Sink instance affects creation of Monitor.
> In the refguide for Canary, hbase.canary.sink.class was not mentioned.
>  We should document this config.
> Additionally, we need to document that using the default sink is not 
> compatible with table parameters as input so the user must change it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20333) break up shaded client into one with no Hadoop and one that's standalone

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424161#comment-16424161
 ] 

Sean Busbey commented on HBASE-20333:
-

artifact naming is difficult here. which is the more common use case: "just 
using HBase" or "using hbase around hadoop"? whichever should be the one that 
gets the less complicated name. I suspect "just using HBase" is the more common 
case, but names like "hbase-shaded-client-without-hadoop" are awkward.

> break up shaded client into one with no Hadoop and one that's standalone
> 
>
> Key: HBASE-20333
> URL: https://issues.apache.org/jira/browse/HBASE-20333
> Project: HBase
>  Issue Type: Sub-task
>  Components: shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Fix For: 2.0.0
>
>
> there are contexts where we want to stay out of our downstream users way wrt 
> dependencies, but they need more Hadoop classes than we provide. i.e. any 
> downstream client that wants to use both HBase and HDFS in their application, 
> or any non-MR YARN application.
> Now that Hadoop also has shaded client artifacts for Hadoop 3, we're also 
> providing less incremental benefit by including our own rewritten Hadoop 
> classes to avoid downstream needing to pull in all of Hadoop's transitive 
> dependencies.
> right now those users need to ensure that any jars from the Hadoop project 
> are loaded in the classpath prior to our shaded client jar. This is brittle 
> and prone to weird debugging trouble.
> instead, we should have two artifacts: one that just lists Hadoop as a 
> prerequisite and one that still includes the rewritten-but-not-relocated 
> Hadoop classes.
> We can then use docs to emphasize when each of these is appropriate to use.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-20334) add a test that expressly uses both our shaded client and the one from hadoop 3

[Sean Busbey (JIRA)]

Sean Busbey created HBASE-20334:
---

 Summary: add a test that expressly uses both our shaded client and 
the one from hadoop 3
 Key: HBASE-20334
 URL: https://issues.apache.org/jira/browse/HBASE-20334
 Project: HBase
  Issue Type: Sub-task
  Components: hadoop3, shading
Affects Versions: 2.0.0
Reporter: Sean Busbey


Since we're making a shaded client that bleed out of our namespace and into 
Hadoop's, we should ensure that we can show our clients coexisting. Even if 
it's just an IT that successfully talks to both us and HDFS via our respective 
shaded clients, that'd be a big help in keeping us proactive.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-20335) nightly jobs no longer contain machine information

[Sean Busbey (JIRA)]

Sean Busbey created HBASE-20335:
---

 Summary: nightly jobs no longer contain machine information
 Key: HBASE-20335
 URL: https://issues.apache.org/jira/browse/HBASE-20335
 Project: HBase
  Issue Type: Bug
  Components: test
Reporter: Sean Busbey


something is up with nightly jobs. they no longer have the machine information 
from HBASE-19228.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work started] (HBASE-20335) nightly jobs no longer contain machine information

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HBASE-20335 started by Sean Busbey.
---
> nightly jobs no longer contain machine information
> --
>
> Key: HBASE-20335
> URL: https://issues.apache.org/jira/browse/HBASE-20335
> Project: HBase
>  Issue Type: Bug
>  Components: test
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
>
> something is up with nightly jobs. they no longer have the machine 
> information from HBASE-19228.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HBASE-20335) nightly jobs no longer contain machine information

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey reassigned HBASE-20335:
---

Assignee: Sean Busbey

> nightly jobs no longer contain machine information
> --
>
> Key: HBASE-20335
> URL: https://issues.apache.org/jira/browse/HBASE-20335
> Project: HBase
>  Issue Type: Bug
>  Components: test
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
>
> something is up with nightly jobs. they no longer have the machine 
> information from HBASE-19228.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20256) Document commands that do not work with 1.2 shell

[Peter Somogyi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Somogyi updated HBASE-20256:
--
Attachment: HBASE-20256.master.002.patch

> Document commands that do not work with 1.2 shell
> -
>
> Key: HBASE-20256
> URL: https://issues.apache.org/jira/browse/HBASE-20256
> Project: HBase
>  Issue Type: Sub-task
>  Components: documentation, shell
>Affects Versions: 2.0.0
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Major
> Attachments: HBASE-20256.master.001.patch, 
> HBASE-20256.master.002.patch, HBASE-20256.master.002.patch
>
>
> Some commands do not work from 1.2 shell when running against 2.0 server. Add 
> a section to the reference guide mentioning the incompatibilities.
> Some of these are collected in this document: 
> https://docs.google.com/document/d/1l2ad5G_GUk0WQ02xKeKO6mTbpdyZjU1NGuPPI42Wbf4/edit



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20335) nightly jobs no longer contain machine information

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-20335:

Attachment: HBASE-20335.0.patch

> nightly jobs no longer contain machine information
> --
>
> Key: HBASE-20335
> URL: https://issues.apache.org/jira/browse/HBASE-20335
> Project: HBase
>  Issue Type: Bug
>  Components: test
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Attachments: HBASE-20335.0.patch
>
>
> something is up with nightly jobs. they no longer have the machine 
> information from HBASE-19228.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20224) Web UI is broken in standalone mode

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424313#comment-16424313
 ] 

Ted Yu commented on HBASE-20224:


[~uagashe]:
Looks like hbase-spark module needs similar addition to pom.xml

Can you take a look at addendum v6 ?

thanks

> Web UI is broken in standalone mode
> ---
>
> Key: HBASE-20224
> URL: https://issues.apache.org/jira/browse/HBASE-20224
> Project: HBase
>  Issue Type: Bug
>  Components: UI, Usability
>Affects Versions: 2.0.0-beta-2
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: 
> 0001-HBASE-20224-Web-UI-is-broken-in-standalone-mode-ADDE.ADDENDUM.patch, 
> 20224-addendum.3.txt, 20224.addendum.4, 20224.addendum.5, 20224.addendum.6, 
> hbase-20224.master.001.patch, hbase-20224.master.002.patch, 
> hbase-20224.master.003.patch, hbase-20224.master.addendum.patch
>
>
> Web UI doesn't show up in standalone mode on default port. This can be seen 
> on master and branch-2.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20244) NoSuchMethodException when retrieving private method decryptEncryptedDataEncryptionKey from DFSClient

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424327#comment-16424327
 ] 

Hadoop QA commented on HBASE-20244:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
8s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue}  0m  
1s{color} | {color:blue} The patch file was not named according to hbase's 
naming conventions. Please see 
https://yetus.apache.org/documentation/0.7.0/precommit-patchnames for 
instructions. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  5m 
59s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
46s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
19m 14s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}103m 31s{color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
20s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}148m  3s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.replication.TestReplicationDroppedTables |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f |
| JIRA Issue | HBASE-20244 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12917379/20244.v1.txt |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux 78fc7f81b071 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 219625233c |
| maven | version: Apache Maven 3.5.3 
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 

[jira] [Updated] (HBASE-20327) When qualifier is not specified, append and incr operation do not work (shell)

[Nihal Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nihal Jain updated HBASE-20327:
---
Attachment: HBASE-20327.master.001.patch

> When qualifier is not specified, append and incr operation do not work (shell)
> --
>
> Key: HBASE-20327
> URL: https://issues.apache.org/jira/browse/HBASE-20327
> Project: HBase
>  Issue Type: Bug
>  Components: shell
>Affects Versions: 1.0.0, 3.0.0, 2.0.0
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Minor
> Attachments: HBASE-20327.master.001.patch
>
>
> Running the example commands specified in shell docs for "append" and "incr" 
> throw following error:
> {code:java}
> ERROR: Failed to provide both column family and column qualifier for 
> append{code}
> {code:java}
> ERROR: Failed to provide both column family and column qualifier for 
> incr{code}
> While running the same command via java does not require the user to provide 
> both column and qualifier and works smoothly.
>  
> Steps to reproduce:
> 1) APPEND
> {code:java}
> hbase(main):002:0> create 't1', 'c1', 'c2'
> Created table t1
> Took 0.8151 seconds   
>      
> hbase(main):003:0> append 't1', 'r1', 'c1', 'value'
> ERROR: Failed to provide both column family and column qualifier for append
> Appends a cell 'value' at specified table/row/column coordinates.
>   hbase> append 't1', 'r1', 'c1', 'value', ATTRIBUTES=>{'mykey'=>'myvalue'}
>   hbase> append 't1', 'r1', 'c1', 'value', {VISIBILITY=>'PRIVATE|SECRET'}
> The same commands also can be run on a table reference. Suppose you had a 
> reference
> t to table 't1', the corresponding command would be:
>   hbase> t.append 'r1', 'c1', 'value', ATTRIBUTES=>{'mykey'=>'myvalue'}
>   hbase> t.append 'r1', 'c1', 'value', {VISIBILITY=>'PRIVATE|SECRET'}
> Took 0.0326 seconds  
> hbase(main):004:0> scan 't1'
> ROW  COLUMN+CELL  
>      
> 0 row(s)
> Took 0.1273 seconds   
>   {code}
> While the same command would run if we run the following java code:
> {code:java}
>     try (Connection connection = ConnectionFactory.createConnection(config);
>     Admin admin = connection.getAdmin();) {
>   Table table = connection.getTable(TableName.valueOf("t1"));
>   Append a = new Append(Bytes.toBytes("r1"));
>   a.addColumn(Bytes.toBytes("c1"), null, Bytes.toBytes("value"));
>   table.append(a);
>     }{code}
> Scan result after executing java code:
> {code:java}
> hbase(main):005:0> scan 't1'
> ROW  COLUMN+CELL  
>      
>  r1  column=c1:, timestamp=1522649623090, 
> value=value  
> 1 row(s)
> Took 0.0188 seconds
> {code}
>  
> 2) INCREMENT:
> Similarly in case of increment, we get the following error (shell):
> {code:java}
> hbase(main):006:0> incr 't1', 'r2', 'c1', 111
> ERROR: Failed to provide both column family and column qualifier for incr
> Increments a cell 'value' at specified table/row/column coordinates.
> To increment a cell value in table 'ns1:t1' or 't1' at row 'r1' under column
> 'c1' by 1 (can be omitted) or 10 do:
>   hbase> incr 'ns1:t1', 'r1', 'c1'
>   hbase> incr 't1', 'r1', 'c1'
>   hbase> incr 't1', 'r1', 'c1', 1
>   hbase> incr 't1', 'r1', 'c1', 10
>   hbase> incr 't1', 'r1', 'c1', 10, {ATTRIBUTES=>{'mykey'=>'myvalue'}}
>   hbase> incr 't1', 'r1', 'c1', {ATTRIBUTES=>{'mykey'=>'myvalue'}}
>   hbase> incr 't1', 'r1', 'c1', 10, {VISIBILITY=>'PRIVATE|SECRET'}
> The same commands also can be run on a table reference. Suppose you had a 
> reference
> t to table 't1', the corresponding command would be:
>   hbase> t.incr 'r1', 'c1'
>   hbase> t.incr 'r1', 'c1', 1
>   hbase> t.incr 'r1', 'c1', 10, {ATTRIBUTES=>{'mykey'=>'myvalue'}}
>   hbase> t.incr 'r1', 'c1', 10, {VISIBILITY=>'PRIVATE|SECRET'}
> Took 0.0103 seconds   
>      
> hbase(main):007:0> scan 't1'
> ROW  COLUMN+CELL  
>      
>  r1  column=c1:, timestamp=1522649623090, 
> value=value  
> 1 row(s)
> Took 0.0062 seconds

[jira] [Commented] (HBASE-20224) Web UI is broken in standalone mode

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424365#comment-16424365
 ] 

Ted Yu commented on HBASE-20224:


[~busbey]:
Addendum 6 fixes the test error you mentioned in HBASE-20336

> Web UI is broken in standalone mode
> ---
>
> Key: HBASE-20224
> URL: https://issues.apache.org/jira/browse/HBASE-20224
> Project: HBase
>  Issue Type: Bug
>  Components: UI, Usability
>Affects Versions: 2.0.0-beta-2
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: 
> 0001-HBASE-20224-Web-UI-is-broken-in-standalone-mode-ADDE.ADDENDUM.patch, 
> 20224-addendum.3.txt, 20224.addendum.4, 20224.addendum.5, 20224.addendum.6, 
> hbase-20224.master.001.patch, hbase-20224.master.002.patch, 
> hbase-20224.master.003.patch, hbase-20224.master.addendum.patch
>
>
> Web UI doesn't show up in standalone mode on default port. This can be seen 
> on master and branch-2.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HBASE-20336) org.apache.hadoop.hbase.spark.IntegrationTestSparkBulkLoad fails

[Artem Ervits (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Artem Ervits reassigned HBASE-20336:


Assignee: Artem Ervits

> org.apache.hadoop.hbase.spark.IntegrationTestSparkBulkLoad fails
> 
>
> Key: HBASE-20336
> URL: https://issues.apache.org/jira/browse/HBASE-20336
> Project: HBase
>  Issue Type: Bug
>  Components: spark
>Affects Versions: 3.0.0
>Reporter: Sean Busbey
>Assignee: Artem Ervits
>Priority: Blocker
> Fix For: 3.0.0
>
>
> running {{mvn verify}} for the spark integration tests against current master 
> fails
> {code}
> $ mvn -DskipTests install
> 11:37:26,815 [INFO] Scanning for projects...
> ...
> 11:43:36,711 [INFO] 
> 
> 11:43:36,711 [INFO] Reactor Summary:
> 11:43:36,711 [INFO] 
> 11:43:36,712 [INFO] Apache HBase ... 
> SUCCESS [  6.294 s]
> 11:43:36,712 [INFO] Apache HBase - Checkstyle .. 
> SUCCESS [  1.742 s]
> 11:43:36,712 [INFO] Apache HBase - Build Support ... 
> SUCCESS [  0.070 s]
> 11:43:36,712 [INFO] Apache HBase - Error Prone Rules ... 
> SUCCESS [  2.206 s]
> 11:43:36,712 [INFO] Apache HBase - Annotations . 
> SUCCESS [  1.413 s]
> 11:43:36,712 [INFO] Apache HBase - Build Configuration . 
> SUCCESS [  0.254 s]
> 11:43:36,712 [INFO] Apache HBase - Shaded Protocol . 
> SUCCESS [ 37.870 s]
> 11:43:36,712 [INFO] Apache HBase - Common .. 
> SUCCESS [ 12.526 s]
> 11:43:36,712 [INFO] Apache HBase - Metrics API . 
> SUCCESS [  2.412 s]
> 11:43:36,712 [INFO] Apache HBase - Hadoop Compatibility  
> SUCCESS [  3.260 s]
> 11:43:36,712 [INFO] Apache HBase - Metrics Implementation .. 
> SUCCESS [  2.756 s]
> 11:43:36,713 [INFO] Apache HBase - Hadoop Two Compatibility  
> SUCCESS [  3.959 s]
> 11:43:36,713 [INFO] Apache HBase - Protocol  
> SUCCESS [ 11.295 s]
> 11:43:36,713 [INFO] Apache HBase - Client .. 
> SUCCESS [ 15.360 s]
> 11:43:36,713 [INFO] Apache HBase - Zookeeper ... 
> SUCCESS [  4.389 s]
> 11:43:36,713 [INFO] Apache HBase - Replication . 
> SUCCESS [  4.202 s]
> 11:43:36,713 [INFO] Apache HBase - Resource Bundle . 
> SUCCESS [  0.206 s]
> 11:43:36,713 [INFO] Apache HBase - HTTP  
> SUCCESS [  8.530 s]
> 11:43:36,713 [INFO] Apache HBase - Procedure ... 
> SUCCESS [  4.196 s]
> 11:43:36,713 [INFO] Apache HBase - Server .. 
> SUCCESS [ 44.604 s]
> 11:43:36,713 [INFO] Apache HBase - MapReduce ... 
> SUCCESS [ 11.122 s]
> 11:43:36,713 [INFO] Apache HBase - Testing Util  
> SUCCESS [  6.633 s]
> 11:43:36,713 [INFO] Apache HBase - Thrift .. 
> SUCCESS [  9.771 s]
> 11:43:36,713 [INFO] Apache HBase - RSGroup . 
> SUCCESS [  6.703 s]
> 11:43:36,713 [INFO] Apache HBase - Shell ... 
> SUCCESS [  7.094 s]
> 11:43:36,713 [INFO] Apache HBase - Coprocessor Endpoint  
> SUCCESS [  7.542 s]
> 11:43:36,713 [INFO] Apache HBase - Backup .. 
> SUCCESS [  6.246 s]
> 11:43:36,713 [INFO] Apache HBase - Integration Tests ... 
> SUCCESS [  7.461 s]
> 11:43:36,713 [INFO] Apache HBase - Examples  
> SUCCESS [  9.054 s]
> 11:43:36,713 [INFO] Apache HBase - Rest  
> SUCCESS [  8.972 s]
> 11:43:36,713 [INFO] Apache HBase - External Block Cache  
> SUCCESS [  5.180 s]
> 11:43:36,713 [INFO] Apache HBase - Spark ... 
> SUCCESS [01:10 min]
> 11:43:36,713 [INFO] Apache HBase - Spark Integration Tests . 
> SUCCESS [  7.774 s]
> 11:43:36,713 [INFO] Apache HBase - Assembly  
> SUCCESS [ 15.075 s]
> 11:43:36,713 [INFO] Apache HBase - Shaded .. 
> SUCCESS [  0.323 s]
> 11:43:36,713 [INFO] Apache HBase - Shaded - Client . 
> SUCCESS [  3.004 s]
> 11:43:36,713 [INFO] Apache HBase - Shaded - MapReduce .. 
> SUCCESS [  3.542 s]
> 11:43:36,714 [INFO] Apache HBase Shaded Packaging Invariants ... 
> SUCCESS [  3.105 s]
> 11:43:36,714 [INFO] Apache HBase - Archetypes .. 
> SUCCESS [  0.063 s]
> 11:43:36,714 [INFO] Apache HBase - Exemplar for hbase-client archetype . 
> SUCCESS [  4.843 s]
> 11:43:36,714 

[jira] [Assigned] (HBASE-20336) org.apache.hadoop.hbase.spark.IntegrationTestSparkBulkLoad fails

[Artem Ervits (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Artem Ervits reassigned HBASE-20336:


Assignee: (was: Artem Ervits)

> org.apache.hadoop.hbase.spark.IntegrationTestSparkBulkLoad fails
> 
>
> Key: HBASE-20336
> URL: https://issues.apache.org/jira/browse/HBASE-20336
> Project: HBase
>  Issue Type: Bug
>  Components: spark
>Affects Versions: 3.0.0
>Reporter: Sean Busbey
>Priority: Blocker
> Fix For: 3.0.0
>
>
> running {{mvn verify}} for the spark integration tests against current master 
> fails
> {code}
> $ mvn -DskipTests install
> 11:37:26,815 [INFO] Scanning for projects...
> ...
> 11:43:36,711 [INFO] 
> 
> 11:43:36,711 [INFO] Reactor Summary:
> 11:43:36,711 [INFO] 
> 11:43:36,712 [INFO] Apache HBase ... 
> SUCCESS [  6.294 s]
> 11:43:36,712 [INFO] Apache HBase - Checkstyle .. 
> SUCCESS [  1.742 s]
> 11:43:36,712 [INFO] Apache HBase - Build Support ... 
> SUCCESS [  0.070 s]
> 11:43:36,712 [INFO] Apache HBase - Error Prone Rules ... 
> SUCCESS [  2.206 s]
> 11:43:36,712 [INFO] Apache HBase - Annotations . 
> SUCCESS [  1.413 s]
> 11:43:36,712 [INFO] Apache HBase - Build Configuration . 
> SUCCESS [  0.254 s]
> 11:43:36,712 [INFO] Apache HBase - Shaded Protocol . 
> SUCCESS [ 37.870 s]
> 11:43:36,712 [INFO] Apache HBase - Common .. 
> SUCCESS [ 12.526 s]
> 11:43:36,712 [INFO] Apache HBase - Metrics API . 
> SUCCESS [  2.412 s]
> 11:43:36,712 [INFO] Apache HBase - Hadoop Compatibility  
> SUCCESS [  3.260 s]
> 11:43:36,712 [INFO] Apache HBase - Metrics Implementation .. 
> SUCCESS [  2.756 s]
> 11:43:36,713 [INFO] Apache HBase - Hadoop Two Compatibility  
> SUCCESS [  3.959 s]
> 11:43:36,713 [INFO] Apache HBase - Protocol  
> SUCCESS [ 11.295 s]
> 11:43:36,713 [INFO] Apache HBase - Client .. 
> SUCCESS [ 15.360 s]
> 11:43:36,713 [INFO] Apache HBase - Zookeeper ... 
> SUCCESS [  4.389 s]
> 11:43:36,713 [INFO] Apache HBase - Replication . 
> SUCCESS [  4.202 s]
> 11:43:36,713 [INFO] Apache HBase - Resource Bundle . 
> SUCCESS [  0.206 s]
> 11:43:36,713 [INFO] Apache HBase - HTTP  
> SUCCESS [  8.530 s]
> 11:43:36,713 [INFO] Apache HBase - Procedure ... 
> SUCCESS [  4.196 s]
> 11:43:36,713 [INFO] Apache HBase - Server .. 
> SUCCESS [ 44.604 s]
> 11:43:36,713 [INFO] Apache HBase - MapReduce ... 
> SUCCESS [ 11.122 s]
> 11:43:36,713 [INFO] Apache HBase - Testing Util  
> SUCCESS [  6.633 s]
> 11:43:36,713 [INFO] Apache HBase - Thrift .. 
> SUCCESS [  9.771 s]
> 11:43:36,713 [INFO] Apache HBase - RSGroup . 
> SUCCESS [  6.703 s]
> 11:43:36,713 [INFO] Apache HBase - Shell ... 
> SUCCESS [  7.094 s]
> 11:43:36,713 [INFO] Apache HBase - Coprocessor Endpoint  
> SUCCESS [  7.542 s]
> 11:43:36,713 [INFO] Apache HBase - Backup .. 
> SUCCESS [  6.246 s]
> 11:43:36,713 [INFO] Apache HBase - Integration Tests ... 
> SUCCESS [  7.461 s]
> 11:43:36,713 [INFO] Apache HBase - Examples  
> SUCCESS [  9.054 s]
> 11:43:36,713 [INFO] Apache HBase - Rest  
> SUCCESS [  8.972 s]
> 11:43:36,713 [INFO] Apache HBase - External Block Cache  
> SUCCESS [  5.180 s]
> 11:43:36,713 [INFO] Apache HBase - Spark ... 
> SUCCESS [01:10 min]
> 11:43:36,713 [INFO] Apache HBase - Spark Integration Tests . 
> SUCCESS [  7.774 s]
> 11:43:36,713 [INFO] Apache HBase - Assembly  
> SUCCESS [ 15.075 s]
> 11:43:36,713 [INFO] Apache HBase - Shaded .. 
> SUCCESS [  0.323 s]
> 11:43:36,713 [INFO] Apache HBase - Shaded - Client . 
> SUCCESS [  3.004 s]
> 11:43:36,713 [INFO] Apache HBase - Shaded - MapReduce .. 
> SUCCESS [  3.542 s]
> 11:43:36,714 [INFO] Apache HBase Shaded Packaging Invariants ... 
> SUCCESS [  3.105 s]
> 11:43:36,714 [INFO] Apache HBase - Archetypes .. 
> SUCCESS [  0.063 s]
> 11:43:36,714 [INFO] Apache HBase - Exemplar for hbase-client archetype . 
> SUCCESS [  4.843 s]
> 11:43:36,714 [INFO] Apache HBase - 

[jira] [Commented] (HBASE-20305) Add option to SyncTable that skip deletes on target cluster

[Wellington Chevreuil (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424382#comment-16424382
 ] 

Wellington Chevreuil commented on HBASE-20305:
--

Added new patch with the suggested the changes and additional tests for doPuts. 

> Add option to SyncTable that skip deletes on target cluster
> ---
>
> Key: HBASE-20305
> URL: https://issues.apache.org/jira/browse/HBASE-20305
> Project: HBase
>  Issue Type: Improvement
>  Components: mapreduce
>Affects Versions: 2.0.0-alpha-4
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Attachments: 0001-HBASE-20305.master.001.patch, 
> HBASE-20305.master.002.patch
>
>
> We had a situation where two clusters with active-active replication got out 
> of sync, but both had data that should be kept. The tables in question never 
> have data deleted, but ingestion had happened on the two different clusters, 
> some rows had been even updated.
> In this scenario, a cell that is present in one of the table clusters should 
> not be deleted, but replayed on the other. Also, for cells with same 
> identifier but different values, the most recent value should be kept. 
> Current version of SyncTable would not be applicable here, because it would 
> simply copy the whole state from source to target, then losing any additional 
> rows that might be only in target, as well as cell values that got most 
> recent update. This could be solved by adding an option to skip deletes for 
> SyncTable. This way, the additional cells not present on source would still 
> be kept. For cells with same identifier but different values, it would just 
> perform a Put for the cell version from source, but client scans would still 
> fetch the most recent timestamp.
> I'm attaching a patch with this additional option shortly. Please share your 
> thoughts.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20305) Add option to SyncTable that skip deletes on target cluster

[Wellington Chevreuil (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wellington Chevreuil updated HBASE-20305:
-
Attachment: HBASE-20305.master.002.patch

> Add option to SyncTable that skip deletes on target cluster
> ---
>
> Key: HBASE-20305
> URL: https://issues.apache.org/jira/browse/HBASE-20305
> Project: HBase
>  Issue Type: Improvement
>  Components: mapreduce
>Affects Versions: 2.0.0-alpha-4
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Attachments: 0001-HBASE-20305.master.001.patch, 
> HBASE-20305.master.002.patch
>
>
> We had a situation where two clusters with active-active replication got out 
> of sync, but both had data that should be kept. The tables in question never 
> have data deleted, but ingestion had happened on the two different clusters, 
> some rows had been even updated.
> In this scenario, a cell that is present in one of the table clusters should 
> not be deleted, but replayed on the other. Also, for cells with same 
> identifier but different values, the most recent value should be kept. 
> Current version of SyncTable would not be applicable here, because it would 
> simply copy the whole state from source to target, then losing any additional 
> rows that might be only in target, as well as cell values that got most 
> recent update. This could be solved by adding an option to skip deletes for 
> SyncTable. This way, the additional cells not present on source would still 
> be kept. For cells with same identifier but different values, it would just 
> perform a Put for the cell version from source, but client scans would still 
> fetch the most recent timestamp.
> I'm attaching a patch with this additional option shortly. Please share your 
> thoughts.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (HBASE-10790) make assembly:single as default in pom.xml

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-10790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey resolved HBASE-10790.
-
Resolution: Won't Fix
  Assignee: (was: Liu Shaohui)

now covered in the section "build a tarball" under building.

https://hbase.apache.org/book.html#_build_a_tarball

> make assembly:single as default in pom.xml
> --
>
> Key: HBASE-10790
> URL: https://issues.apache.org/jira/browse/HBASE-10790
> Project: HBase
>  Issue Type: Improvement
>Reporter: Liu Shaohui
>Priority: Minor
> Attachments: HBASE-10790-trunk-v1.diff
>
>
> Now to compile a HBase tar release package, we should use
> the cmd: 
> {code}
>  mvn clean package assembly:single
> {code}, which is not convenient. We can make assembly:single as a default 
> option and run the assembly plugin in maven package phase. Then we can just 
> use the cmd {code} mvn clean package {code} to get a release package.
> Other suggestions are welcomed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20322) CME in StoreScanner causes region server crash

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424449#comment-16424449
 ] 

Hudson commented on HBASE-20322:


SUCCESS: Integrated in Jenkins build HBase-1.3-IT #387 (See 
[https://builds.apache.org/job/HBase-1.3-IT/387/])
HBASE-20322 CME in StoreScanner causes region server crash (apurtell: rev 
9eaafe1ee91bc12d8933e17ad6cab7af8251c0f5)
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java


> CME in StoreScanner causes region server crash
> --
>
> Key: HBASE-20322
> URL: https://issues.apache.org/jira/browse/HBASE-20322
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.2
>Reporter: Thiruvel Thirumoolan
>Assignee: Thiruvel Thirumoolan
>Priority: Major
> Fix For: 1.5.0, 1.3.3, 1.4.4
>
> Attachments: HBASE-20322.branch-1.3.001.patch, 
> HBASE-20322.branch-1.3.002.patch, HBASE-20322.branch-1.4.001.patch
>
>
> RS crashed with ConcurrentModificationException on our 1.3 cluster, stack 
> trace below. [~toffer] and I checked and there is a race condition between 
> flush and scanner close. When StoreScanner.updateReaders() is updating the 
> scanners after a newly flushed file (in this trace below a region close 
> during a split), the client's scanner could be closing thus causing CME.
> Its rare, but since it crashes the region server, needs to be fixed.
> FATAL regionserver.HRegionServer [regionserver/] : ABORTING region server 
> : Replay of WAL required. Forcing server shutdown
> org.apache.hadoop.hbase.DroppedSnapshotException: region: 
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2579)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2255)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2217)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2207)
> at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1501)
> at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1420)
> at 
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.stepsBeforePONR(SplitTransactionImpl.java:398)
> at 
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.createDaughters(SplitTransactionImpl.java:278)
> at 
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.execute(SplitTransactionImpl.java:566)
> at 
> org.apache.hadoop.hbase.regionserver.SplitRequest.doSplitting(SplitRequest.java:82)
> at 
> org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:154)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.ConcurrentModificationException
> at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
> at java.util.ArrayList$Itr.next(ArrayList.java:851)
> at 
> org.apache.hadoop.hbase.regionserver.StoreScanner.clearAndClose(StoreScanner.java:797)
> at 
> org.apache.hadoop.hbase.regionserver.StoreScanner.updateReaders(StoreScanner.java:825)
> at 
> org.apache.hadoop.hbase.regionserver.HStore.notifyChangedReadersObservers(HStore.java:1155)
> PS: ignore the line no in the above stack trace, method calls should help 
> understand whats happening.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-18309) Support multi threads in CleanerChore

[Zach York (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424471#comment-16424471
 ] 

Zach York commented on HBASE-18309:
---

[~reidchan] can you put the branch-1 part in review board as well? Thanks

> Support multi threads in CleanerChore
> -
>
> Key: HBASE-18309
> URL: https://issues.apache.org/jira/browse/HBASE-18309
> Project: HBase
>  Issue Type: Improvement
>Reporter: binlijin
>Assignee: Reid Chan
>Priority: Major
> Fix For: 2.0.0-beta-1, 2.0.0
>
> Attachments: HBASE-18309.addendum.patch, 
> HBASE-18309.branch-1.001.patch, HBASE-18309.branch-1.002.patch, 
> HBASE-18309.branch-1.003.patch, HBASE-18309.branch-1.004.patch, 
> HBASE-18309.branch-1.005.patch, HBASE-18309.branch-1.006.patch, 
> HBASE-18309.master.001.patch, HBASE-18309.master.002.patch, 
> HBASE-18309.master.004.patch, HBASE-18309.master.005.patch, 
> HBASE-18309.master.006.patch, HBASE-18309.master.007.patch, 
> HBASE-18309.master.008.patch, HBASE-18309.master.009.patch, 
> HBASE-18309.master.010.patch, HBASE-18309.master.011.patch, 
> HBASE-18309.master.012.patch, space_consumption_in_archive.png
>
>
> There is only one thread in LogCleaner to clean oldWALs and in our big 
> cluster we find this is not enough. The number of files under oldWALs reach 
> the max-directory-items limit of HDFS and cause region server crash, so we 
> use multi threads for LogCleaner and the crash not happened any more.
> What's more, currently there's only one thread iterating the archive 
> directory, and we could use multiple threads cleaning sub directories in 
> parallel to speed it up.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20256) Document commands that do not work with 1.2 shell

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424485#comment-16424485
 ] 

Hadoop QA commented on HBASE-20256:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  2m 
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
 6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}119m 41s{color} 
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
25s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}136m  2s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.replication.TestReplicationDroppedTables |
|   | 
hadoop.hbase.replication.multiwal.TestReplicationKillMasterRSCompressedWithMultipleAsyncWAL
 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f |
| JIRA Issue | HBASE-20256 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12917400/HBASE-20256.master.002.patch
 |
| Optional Tests |  asflicense  javac  javadoc  unit  |
| uname | Linux dacd499b0cab 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 
13:48:03 UTC 2016 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / bf29a1fee9 |
| maven | version: Apache Maven 3.5.3 
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_162 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/12280/artifact/patchprocess/patch-unit-root.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/12280/testReport/ |
| Max. process+thread count | 5165 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/12280/console |
| Powered by | Apache Yetus 0.7.0   http://yetus.apache.org |


This message was automatically generated.



> Document commands that do not work with 1.2 shell
> -
>
> Key: HBASE-20256
> URL: https://issues.apache.org/jira/browse/HBASE-20256
> Project: HBase
>  Issue Type: Sub-task
>  Components: documentation, shell
>Affects Versions: 2.0.0
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Major
> Attachments: HBASE-20256.master.001.patch, 
> HBASE-20256.master.002.patch, HBASE-20256.master.002.patch
>
>
> Some commands do not work from 1.2 shell when running against 2.0 server. Add 
> a section to the reference guide mentioning the incompatibilities.
> Some of these are collected in this document: 
> https://docs.google.com/document/d/1l2ad5G_GUk0WQ02xKeKO6mTbpdyZjU1NGuPPI42Wbf4/edit



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20337) Update the doc on how to setup shortcircuit reads; its stale

[Wei-Chiu Chuang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424484#comment-16424484
 ] 

Wei-Chiu Chuang commented on HBASE-20337:
-

LGTM (non-binding)

> Update the doc on how to setup shortcircuit reads; its stale
> 
>
> Key: HBASE-20337
> URL: https://issues.apache.org/jira/browse/HBASE-20337
> Project: HBase
>  Issue Type: Sub-task
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-20337.master.001.patch, 
> HBASE-20337.master.002.patch
>
>
> The doc is from another era. Update it. Short-circuit reads can make a big 
> difference.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-4436) Remove methods deprecated in 0.90 from TRUNK and 0.92

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424488#comment-16424488
 ] 

stack commented on HBASE-4436:
--

HBASE-4624 was committed with this JIRA's number. Leaving it as is since long 
ago.

> Remove methods deprecated in 0.90 from TRUNK and 0.92
> -
>
> Key: HBASE-4436
> URL: https://issues.apache.org/jira/browse/HBASE-4436
> Project: HBase
>  Issue Type: Task
>Reporter: stack
>Assignee: Jonathan Hsieh
>Priority: Major
>
> Remove methods deprecated in 0.90 from codebase.  i took a quick look.  The 
> messy bit is thrift referring to old stuff; will take a little work to do the 
> convertion over to the new methods.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20141) Fix TooManyFiles exception when RefreshingChannels in FileIOEngine

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424495#comment-16424495
 ] 

stack commented on HBASE-20141:
---

Mighty [~zyork]. For the future This issue was committed like this:

kalashnikov:hbase.git.clean stack$ git show 10a7b5ea15
commit 10a7b5ea1591e93fff39a0c9fc35ee2c8805d49f
Author: Zach York 
Date:   Wed Feb 28 10:40:38 2018 -0800

[HBASE-20141] Fix TooManyFiles exception when RefreshingChannels

HBASE-19435 implements a fix for reopening file channels when they are 
unnexpected closed
to avoid disabling the BucketCache. However, it was missed that the the 
channels might not
actually be completely closed (the write or read channel might still be open
(see 
https://docs.oracle.com/javase/7/docs/api/java/nio/channels/ClosedChannelException.html)
This commit closes any open channels before creating a new channel.

Don't add the square brackets around the JIRA number going forward. Just an FYI 
sir (messes up reporting after the fact...no big deal).

> Fix TooManyFiles exception when RefreshingChannels in FileIOEngine
> --
>
> Key: HBASE-20141
> URL: https://issues.apache.org/jira/browse/HBASE-20141
> Project: HBase
>  Issue Type: Bug
>  Components: BucketCache
>Affects Versions: 1.4.0, 2.0.0-beta-1, 1.4.2
>Reporter: Zach York
>Assignee: Zach York
>Priority: Major
> Fix For: 1.5.0, 1.4.3, 2.0.0
>
> Attachments: HBASE-20141.master.001.patch, 
> HBASE-20141.master.002.patch, HBASE-20141.master.003.patch, 
> HBASE-20141.master.004.patch
>
>
> HBASE-19435 implements a fix for reopening file channels when they are 
> unnexpected closed
> to avoid disabling the BucketCache. However, it was missed that the the 
> channels might not
> actually be completely closed (the write or read channel might still be open
> (see 
> https://docs.oracle.com/javase/7/docs/api/java/nio/channels/ClosedChannelException.html)
> This commit closes any open channels before creating a new channel.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19890) Canary usage should document hbase.canary.sink.class config

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424200#comment-16424200
 ] 

Hadoop QA commented on HBASE-19890:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red}419m 
52s{color} | {color:red} Docker failed to build yetus/hbase:d8b550f. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-19890 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12917326/HBASE-19890.master.001.patch
 |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/12273/console |
| Powered by | Apache Yetus 0.7.0   http://yetus.apache.org |


This message was automatically generated.



> Canary usage should document hbase.canary.sink.class config
> ---
>
> Key: HBASE-19890
> URL: https://issues.apache.org/jira/browse/HBASE-19890
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Peter Somogyi
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-19890.master.001.patch
>
>
> Canary#main uses config hbase.canary.sink.class to instantiate Sink class.
> The Sink instance affects creation of Monitor.
> In the refguide for Canary, hbase.canary.sink.class was not mentioned.
>  We should document this config.
> Additionally, we need to document that using the default sink is not 
> compatible with table parameters as input so the user must change it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20224) Web UI is broken in standalone mode

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-20224:
---
Attachment: 20224.addendum.6

> Web UI is broken in standalone mode
> ---
>
> Key: HBASE-20224
> URL: https://issues.apache.org/jira/browse/HBASE-20224
> Project: HBase
>  Issue Type: Bug
>  Components: UI, Usability
>Affects Versions: 2.0.0-beta-2
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: 
> 0001-HBASE-20224-Web-UI-is-broken-in-standalone-mode-ADDE.ADDENDUM.patch, 
> 20224-addendum.3.txt, 20224.addendum.4, 20224.addendum.5, 20224.addendum.6, 
> hbase-20224.master.001.patch, hbase-20224.master.002.patch, 
> hbase-20224.master.003.patch, hbase-20224.master.addendum.patch
>
>
> Web UI doesn't show up in standalone mode on default port. This can be seen 
> on master and branch-2.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20224) Web UI is broken in standalone mode

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-20224:
---
Attachment: (was: 20224.addendum.6)

> Web UI is broken in standalone mode
> ---
>
> Key: HBASE-20224
> URL: https://issues.apache.org/jira/browse/HBASE-20224
> Project: HBase
>  Issue Type: Bug
>  Components: UI, Usability
>Affects Versions: 2.0.0-beta-2
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: 
> 0001-HBASE-20224-Web-UI-is-broken-in-standalone-mode-ADDE.ADDENDUM.patch, 
> 20224-addendum.3.txt, 20224.addendum.4, 20224.addendum.5, 20224.addendum.6, 
> hbase-20224.master.001.patch, hbase-20224.master.002.patch, 
> hbase-20224.master.003.patch, hbase-20224.master.addendum.patch
>
>
> Web UI doesn't show up in standalone mode on default port. This can be seen 
> on master and branch-2.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-20337) Update the doc on how to setup shortcircuit reads; its stale

[stack (JIRA)]

stack created HBASE-20337:
-

 Summary: Update the doc on how to setup shortcircuit reads; its 
stale
 Key: HBASE-20337
 URL: https://issues.apache.org/jira/browse/HBASE-20337
 Project: HBase
  Issue Type: Sub-task
Reporter: stack
Assignee: stack
 Fix For: 2.0.0


The doc is from another era. Update it. Short-circuit reads can make a big 
difference.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20337) Update the doc on how to setup shortcircuit reads; its stale

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-20337:
--
Attachment: HBASE-20337.master.001.patch

> Update the doc on how to setup shortcircuit reads; its stale
> 
>
> Key: HBASE-20337
> URL: https://issues.apache.org/jira/browse/HBASE-20337
> Project: HBase
>  Issue Type: Sub-task
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-20337.master.001.patch
>
>
> The doc is from another era. Update it. Short-circuit reads can make a big 
> difference.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20337) Update the doc on how to setup shortcircuit reads; its stale

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-20337:
--
Attachment: HBASE-20337.master.002.patch

> Update the doc on how to setup shortcircuit reads; its stale
> 
>
> Key: HBASE-20337
> URL: https://issues.apache.org/jira/browse/HBASE-20337
> Project: HBase
>  Issue Type: Sub-task
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-20337.master.001.patch, 
> HBASE-20337.master.002.patch
>
>
> The doc is from another era. Update it. Short-circuit reads can make a big 
> difference.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20337) Update the doc on how to setup shortcircuit reads; its stale

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424377#comment-16424377
 ] 

stack commented on HBASE-20337:
---

Thanks for the review [~jojochuang] .. I was looking for how to do 'stable' ... 
I'd forgotten. Thanks for the pointer.

Yeah, no metrics and nothing really in logs when all working nicely. Its a nice 
feature. Can make a big difference when lots of random reading. I could file a 
few issues I suppose

> Update the doc on how to setup shortcircuit reads; its stale
> 
>
> Key: HBASE-20337
> URL: https://issues.apache.org/jira/browse/HBASE-20337
> Project: HBase
>  Issue Type: Sub-task
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-20337.master.001.patch, 
> HBASE-20337.master.002.patch
>
>
> The doc is from another era. Update it. Short-circuit reads can make a big 
> difference.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)