[jira] [Updated] (HBASE-16989) RowProcess#postBatchMutate doesn’t be executed before the mvcc transaction completion

[Anoop Sam John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-16989:
---
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Pushed to master. Thanks for the patch.

> RowProcess#postBatchMutate doesn’t be executed before the mvcc transaction 
> completion
> -
>
> Key: HBASE-16989
> URL: https://issues.apache.org/jira/browse/HBASE-16989
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
> Fix For: 2.0.0
>
> Attachments: HBASE-16989.v0.patch, HBASE-16989.v1.patch, 
> HBASE-16989.v2.patch, HBASE-16989.v3.patch, HBASE-16989.v4.patch
>
>
> After the [HBASE-15158|https://issues.apache.org/jira/browse/HBASE-15158], 
> RowProcess#postBatchMutate will be executed “after” the mvcc transaction 
> completion.
> {code:title=HRegion#processRowsWithLocks}
>   // STEP 8. Complete mvcc.
>   mvcc.completeAndWait(writeEntry);
>   writeEntry = null;
> 
>   // STEP 9. Release region lock
>   if (locked) {
> this.updatesLock.readLock().unlock();
> locked = false;
>   }
> 
>   // STEP 10. Release row lock(s)
>   releaseRowLocks(acquiredRowLocks);
> 
>   // STEP 11. call postBatchMutate hook
>   processor.postBatchMutate(this);
> {code}
> {code:title=RowProcess#postBatchMutate}
>   /**
>* The hook to be executed after the process() and applying the Mutations 
> to region. The
>* difference of this one with {@link #postProcess(HRegion, WALEdit, 
> boolean)} is this hook will
>* be executed before the mvcc transaction completion.
>*/
>   void postBatchMutate(HRegion region) throws IOException;
> {code}
> Do we ought to revamp the comment of RowProcess#postBatchMutate or change the 
> call order?
> I prefer the former, because the HRegion#doMiniBatchMutate() also call 
> postBatchMutate() after the mvcc transaction completion.
> Any comment? Thanks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17088) Refactor RWQueueRpcExecutor/BalancedQueueRpcExecutor/RpcExecutor

[Duo Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang updated HBASE-17088:
--
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 1.4.0
   2.0.0
   Status: Resolved  (was: Patch Available)

Pushed to master and branch-1.

Thanks [~zghaobac] for the patch. Thanks [~mbertozzi] for reviewing.

> Refactor RWQueueRpcExecutor/BalancedQueueRpcExecutor/RpcExecutor
> 
>
> Key: HBASE-17088
> URL: https://issues.apache.org/jira/browse/HBASE-17088
> Project: HBase
>  Issue Type: Improvement
>  Components: rpc
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17088-branch-1.patch, HBASE-17088-v1.patch, 
> HBASE-17088-v2.patch, HBASE-17088-v3.patch, HBASE-17088-v3.patch, 
> HBASE-17088-v4.patch, HBASE-17088-v4.patch
>
>
> 1. The RWQueueRpcExecutor has eight constructor method and the longest one 
> has ten parameters. But It is only used in SimpleRpcScheduler and easy to 
> confused when read the code.
> 2. There are duplicate method implement in RWQueueRpcExecutor and 
> BalancedQueueRpcExecutor. They can be implemented in their parent class 
> RpcExecutor.
> 3. SimpleRpcScheduler read many configs to new RpcExecutor. But the 
> CALL_QUEUE_SCAN_SHARE_CONF_KEY is only needed by RWQueueRpcExecutor. And 
> CALL_QUEUE_CODEL_TARGET_DELAY, CALL_QUEUE_CODEL_INTERVAL and 
> CALL_QUEUE_CODEL_LIFO_THRESHOLD are only needed by AdaptiveLifoCoDelCallQueue.
> So I thought we can refactor it. Suggestions are welcome.
> Review board: https://reviews.apache.org/r/53726/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17123) Add postBulkLoadHFile variant that notifies the final paths for the hfiles

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678583#comment-15678583
 ] 

Hadoop QA commented on HBASE-17123:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 3s 
{color} | {color:blue} The patch file was not named according to hbase's naming 
conventions. Please see 
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for 
instructions. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
31s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
27s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
26m 55s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 100m 31s 
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 25s 
{color} | {color:green} hbase-endpoint in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
26s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 145m 12s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12839667/17123.v5.txt |
| JIRA Issue | HBASE-17123 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 6c81838a555c 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
 |
| git revision | master / be519ca |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4542/testReport/ |
| modules | C: 

[jira] [Commented] (HBASE-17088) Refactor RWQueueRpcExecutor/BalancedQueueRpcExecutor/RpcExecutor

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678557#comment-15678557
 ] 

Hadoop QA commented on HBASE-17088:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
43s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s 
{color} | {color:green} branch-1 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s 
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
57s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
18s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 3s 
{color} | {color:red} hbase-server in branch-1 has 2 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s 
{color} | {color:green} branch-1 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s 
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s 
{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s 
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
15m 40s {color} | {color:green} The patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s 
{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s 
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 95m 49s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
21s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 126m 42s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestFromClientSide |
|   | hadoop.hbase.client.TestFromClientSideWithCoprocessor |
|   | hadoop.hbase.regionserver.TestFailedAppendAndSync |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:e01ee2f |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12839668/HBASE-17088-branch-1.patch
 |
| JIRA Issue | HBASE-17088 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 9a6a219b61de 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | 

[jira] [Commented] (HBASE-17121) Undo the building of xref as part of site build

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678532#comment-15678532
 ] 

Hudson commented on HBASE-17121:


SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #1978 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/1978/])
HBASE-17121 Undo the building of xref as part of site build; Remove (stack: rev 
be519ca1a55827b3c85040cb070e489de5edb11d)
* (edit) pom.xml
* (edit) src/main/asciidoc/_chapters/cp.adoc
* (edit) src/main/asciidoc/_chapters/architecture.adoc
* (edit) src/main/asciidoc/_chapters/appendix_contributing_to_documentation.adoc
* (edit) src/main/site/site.xml


> Undo the building of xref as part of site build
> ---
>
> Key: HBASE-17121
> URL: https://issues.apache.org/jira/browse/HBASE-17121
> Project: HBase
>  Issue Type: Task
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0
>
>
> Remove xref generation as part of site build. It was useful once before 
> grepcode and easy perusal of src via git views: 
> https://git-wip-us.apache.org/repos/asf?p=hbase.git;a=summary
> Replace the xref link with a pointer to apache git.
> DISCUSS thread on dev list: http://osdir.com/ml/general/2016-11/msg22051.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17128) Find Cause of a Write Perf Regression in branch-1.2

[Graham Baecher (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678456#comment-15678456
 ] 

Graham Baecher commented on HBASE-17128:


Thanks, stack.

The YCSB workload that's regressed for us is workload A. We've been setting up 
the YCSB test table with 40 regions, loading the workload once, then running 10 
times with the following configuration and averaging the results:
{quote}recordcount=100
operationcount=100
workload=com.yahoo.ycsb.workloads.CoreWorkload

readallfields=true

readproportion=0.5
updateproportion=0.5
scanproportion=0
insertproportion=0

requestdistribution=zipfian{quote}

We've tested against a couple different HDFS versions, CDH 5.7 and 5.9, and 
gotten similar results. Will next try isolating the performance change to those 
candidate commits you mentioned.

I'm away next week for Thanksgiving so won't be updating with any progress 
during that time.


> Find Cause of a Write Perf Regression in branch-1.2
> ---
>
> Key: HBASE-17128
> URL: https://issues.apache.org/jira/browse/HBASE-17128
> Project: HBase
>  Issue Type: Task
>Reporter: stack
>
> As reported by [~gbaecher] up on the mailing list, there is a regression in 
> 1.2. The regression is in a CDH version of 1.2 actually but the CDH hbase is 
> a near pure 1.2. This is a working issue to figure which of the below changes 
> brought on slower writes (The list comes from doing the following...git log 
> --oneline  
> remotes/origin/cdh5-1.2.0_5.8.0_dev..remotes/origin/cdh5-1.2.0_5.9.0_dev ... 
> I stripped the few CDH specific changes, packaging and tagging only, and then 
> made two groupings; candidates and the unlikelies):
> {code}
>   1 bbc6762 HBASE-16023 Fastpath for the FIFO rpcscheduler Adds an executor 
> that does balanced queue and fast path handing off requests directly to 
> waiting handlers if any present. Idea taken from Apace Kudu (incubating). See 
> https://gerr#
>   2 a260917 HBASE-16288 HFile intermediate block level indexes might recurse 
> forever creating multi TB files
>   3 5633281 HBASE-15811 Batch Get after batch Put does not fetch all Cells We 
> were not waiting on all executors in a batch to complete. The test for 
> no-more-executors was damaged by the 0.99/0.98.4 fix "HBASE-11403 Fix race 
> conditions aro#
>   4 780f720 HBASE-11625 - Verifies data before building HFileBlock. - Adds 
> HFileBlock.Header class which contains information about location of fields. 
> Testing: Adds CorruptedFSReaderImpl to TestChecksum. (Apekshit)
>   5 d735680 HBASE-12133 Add FastLongHistogram for metric computation (Yi Deng)
>   6 c4ee832 HBASE-15222 Use less contended classes for metrics
>   7
>   8 17320a4 HBASE-15683 Min latency in latency histograms are emitted as 
> Long.MAX_VALUE
>   9 283b39f HBASE-15396 Enhance mapreduce.TableSplit to add encoded region 
> name
>  10 39db592 HBASE-16195 Should not add chunk into chunkQueue if not using 
> chunk pool in HeapMemStoreLAB
>  11 5ff28b7 HBASE-16194 Should count in MSLAB chunk allocation into heap size 
> change when adding duplicate cells
>  12 5e3e0d2 HBASE-16318 fail build while rendering velocity template if 
> dependency license isn't in whitelist.
>  13 3ed66e3 HBASE-16318 consistently use the correct name for 'Apache 
> License, Version 2.0'
>  14 351832d HBASE-16340 exclude Xerces iplementation jars from coming in 
> transitively.
>  15 b6aa4be HBASE-16321 ensure no findbugs-jsr305
>  16 4f9dde7 HBASE-16317 revert all ESAPI changes
>  17 71b6a8a HBASE-16284 Unauthorized client can shutdown the cluster (Deokwoo 
> Han)
>  18 523753f HBASE-16450 Shell tool to dump replication queues
>  19 ca5f2ee HBASE-16379 [replication] Minor improvement to 
> replication/copy_tables_desc.rb
>  20 effd105 HBASE-16135 PeerClusterZnode under rs of removed peer may never 
> be deleted
>  21 a5c6610 HBASE-16319 Fix TestCacheOnWrite after HBASE-16288
>  22 1956bb0 HBASE-15808 Reduce potential bulk load intermediate space usage 
> and waste
>  23 031c54e HBASE-16096 Backport. Cleanly remove replication peers from 
> ZooKeeper.
>  24 60a3b12 HBASE-14963 Remove use of Guava Stopwatch from HBase client code 
> (Devaraj Das)
>  25 c7724fc HBASE-16207 can't restore snapshot without "Admin" permission
>  26 8322a0b HBASE-16227 [Shell] Column value formatter not working in scans. 
> Tested : manually using shell.
>  27 8f86658 HBASE-14818 user_permission does not list namespace permissions 
> (li xiang)
>  28 775cd21 HBASE-15465 userPermission returned by getUserPermission() for 
> the selected namespace does not have namespace set (li xiang)
>  29 8d85aff HBASE-16093 Fix splits failed before creating daughter regions 
> leave meta inconsistent
>  30 bc41317 HBASE-16140 bump owasp.esapi from 2.1.0 to 2.1.0.1
>  31 6fc70cd HBASE-16035 Nested AutoCloseables might 

[jira] [Commented] (HBASE-17088) Refactor RWQueueRpcExecutor/BalancedQueueRpcExecutor/RpcExecutor

[Guanghao Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678381#comment-15678381
 ] 

Guanghao Zhang commented on HBASE-17088:


Thanks. Marked the old constructors as deprecated in patch for branch-1.

> Refactor RWQueueRpcExecutor/BalancedQueueRpcExecutor/RpcExecutor
> 
>
> Key: HBASE-17088
> URL: https://issues.apache.org/jira/browse/HBASE-17088
> Project: HBase
>  Issue Type: Improvement
>  Components: rpc
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-17088-branch-1.patch, HBASE-17088-v1.patch, 
> HBASE-17088-v2.patch, HBASE-17088-v3.patch, HBASE-17088-v3.patch, 
> HBASE-17088-v4.patch, HBASE-17088-v4.patch
>
>
> 1. The RWQueueRpcExecutor has eight constructor method and the longest one 
> has ten parameters. But It is only used in SimpleRpcScheduler and easy to 
> confused when read the code.
> 2. There are duplicate method implement in RWQueueRpcExecutor and 
> BalancedQueueRpcExecutor. They can be implemented in their parent class 
> RpcExecutor.
> 3. SimpleRpcScheduler read many configs to new RpcExecutor. But the 
> CALL_QUEUE_SCAN_SHARE_CONF_KEY is only needed by RWQueueRpcExecutor. And 
> CALL_QUEUE_CODEL_TARGET_DELAY, CALL_QUEUE_CODEL_INTERVAL and 
> CALL_QUEUE_CODEL_LIFO_THRESHOLD are only needed by AdaptiveLifoCoDelCallQueue.
> So I thought we can refactor it. Suggestions are welcome.
> Review board: https://reviews.apache.org/r/53726/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17088) Refactor RWQueueRpcExecutor/BalancedQueueRpcExecutor/RpcExecutor

[Guanghao Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang updated HBASE-17088:
---
Attachment: HBASE-17088-branch-1.patch

Attach patch for branch-1.

> Refactor RWQueueRpcExecutor/BalancedQueueRpcExecutor/RpcExecutor
> 
>
> Key: HBASE-17088
> URL: https://issues.apache.org/jira/browse/HBASE-17088
> Project: HBase
>  Issue Type: Improvement
>  Components: rpc
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-17088-branch-1.patch, HBASE-17088-v1.patch, 
> HBASE-17088-v2.patch, HBASE-17088-v3.patch, HBASE-17088-v3.patch, 
> HBASE-17088-v4.patch, HBASE-17088-v4.patch
>
>
> 1. The RWQueueRpcExecutor has eight constructor method and the longest one 
> has ten parameters. But It is only used in SimpleRpcScheduler and easy to 
> confused when read the code.
> 2. There are duplicate method implement in RWQueueRpcExecutor and 
> BalancedQueueRpcExecutor. They can be implemented in their parent class 
> RpcExecutor.
> 3. SimpleRpcScheduler read many configs to new RpcExecutor. But the 
> CALL_QUEUE_SCAN_SHARE_CONF_KEY is only needed by RWQueueRpcExecutor. And 
> CALL_QUEUE_CODEL_TARGET_DELAY, CALL_QUEUE_CODEL_INTERVAL and 
> CALL_QUEUE_CODEL_LIFO_THRESHOLD are only needed by AdaptiveLifoCoDelCallQueue.
> So I thought we can refactor it. Suggestions are welcome.
> Review board: https://reviews.apache.org/r/53726/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17123) Add postBulkLoadHFile variant that notifies the final paths for the hfiles

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678371#comment-15678371
 ] 

Ted Yu commented on HBASE-17123:


Test failure in TestSaslFanOutOneBlockAsyncDFSOutput was not related.

> Add postBulkLoadHFile variant that notifies the final paths for the hfiles
> --
>
> Key: HBASE-17123
> URL: https://issues.apache.org/jira/browse/HBASE-17123
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 17123.v1.txt, 17123.v3.txt, 17123.v4.txt, 17123.v5.txt
>
>
> Currently the postBulkLoadHFile() hook passes the same familyPaths parameter 
> which it receives as method parameter.
> See code in SecureBulkLoadManager :
> {code}
>loaded = region.getCoprocessorHost().postBulkLoadHFile(familyPaths, 
> loaded);
> {code}
> Meaning, the paths are not final, moved paths of the loaded hfiles.
> This issue is to add a variant which notifies the final paths of the loaded 
> hfiles.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17123) Add postBulkLoadHFile variant that notifies the final paths for the hfiles

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-17123:
---
Attachment: 17123.v5.txt

> Add postBulkLoadHFile variant that notifies the final paths for the hfiles
> --
>
> Key: HBASE-17123
> URL: https://issues.apache.org/jira/browse/HBASE-17123
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 17123.v1.txt, 17123.v3.txt, 17123.v4.txt, 17123.v5.txt
>
>
> Currently the postBulkLoadHFile() hook passes the same familyPaths parameter 
> which it receives as method parameter.
> See code in SecureBulkLoadManager :
> {code}
>loaded = region.getCoprocessorHost().postBulkLoadHFile(familyPaths, 
> loaded);
> {code}
> Meaning, the paths are not final, moved paths of the loaded hfiles.
> This issue is to add a variant which notifies the final paths of the loaded 
> hfiles.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17123) Add postBulkLoadHFile variant that notifies the final paths for the hfiles

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678357#comment-15678357
 ] 

Ted Yu commented on HBASE-17123:


bq. deprecate the RegionObserver method in favor of the previous one

You mean deprecate the method without finalPaths parameter, right ?

> Add postBulkLoadHFile variant that notifies the final paths for the hfiles
> --
>
> Key: HBASE-17123
> URL: https://issues.apache.org/jira/browse/HBASE-17123
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 17123.v1.txt, 17123.v3.txt, 17123.v4.txt
>
>
> Currently the postBulkLoadHFile() hook passes the same familyPaths parameter 
> which it receives as method parameter.
> See code in SecureBulkLoadManager :
> {code}
>loaded = region.getCoprocessorHost().postBulkLoadHFile(familyPaths, 
> loaded);
> {code}
> Meaning, the paths are not final, moved paths of the loaded hfiles.
> This issue is to add a variant which notifies the final paths of the loaded 
> hfiles.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17123) Add postBulkLoadHFile variant that notifies the final paths for the hfiles

[Enis Soztutar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678349#comment-15678349
 ] 

Enis Soztutar commented on HBASE-17123:
---

The patch looks good. We want to also deprecate the RegionObserver method in 
favor of the previous one. Can you please rename the {{map}} argument to 
something like {{finalPaths}} or {{storeFiles}}. Maybe also change familyPaths 
to {{stagingFamilyPaths}} to make it explicit that these are paths in the 
staging directory.   



> Add postBulkLoadHFile variant that notifies the final paths for the hfiles
> --
>
> Key: HBASE-17123
> URL: https://issues.apache.org/jira/browse/HBASE-17123
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 17123.v1.txt, 17123.v3.txt, 17123.v4.txt
>
>
> Currently the postBulkLoadHFile() hook passes the same familyPaths parameter 
> which it receives as method parameter.
> See code in SecureBulkLoadManager :
> {code}
>loaded = region.getCoprocessorHost().postBulkLoadHFile(familyPaths, 
> loaded);
> {code}
> Meaning, the paths are not final, moved paths of the loaded hfiles.
> This issue is to add a variant which notifies the final paths of the loaded 
> hfiles.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16894) Create more than 1 split per region, generalize HBASE-12590

[Enis Soztutar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678327#comment-15678327
 ] 

Enis Soztutar commented on HBASE-16894:
---

bq. We have been discussing an interesting edge case when a table has some skew 
due the key design and it would be helpful to write in a single reducer HFiles 
for multiple regions.
For HBase TableOutputFormat where the output record writer directly writes to 
HBase, it might be fine to combine multiple regions in a single reducer since 
the client will do the correct thing of finding relevant locations to write to. 
However, for bulk load and HFileOutputFormat, we should not have more than 1 
region per reduce task. It is "by design" that we should have at least one 
reducer per region since the bulk load process cannot load partial ranges from 
an hfile. Created HFile must fit into the region boundaries otherwise the BL 
client will manually split and rewrite the file, thus causing serial and 
unnecessary work. 

> Create more than 1 split per region, generalize HBASE-12590
> ---
>
> Key: HBASE-16894
> URL: https://issues.apache.org/jira/browse/HBASE-16894
> Project: HBase
>  Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Yi Liang
>  Labels: beginner, beginners
>
> A common request from users is to be able to better control how many map 
> tasks are created per region. Right now, it is always 1 region = 1 input 
> split = 1 map task. Same goes for Spark since it uses the TIF. With region 
> sizes as large as 50 GBs, it is desirable to be able to create more than 1 
> split per region.
> HBASE-12590 adds a config property for MR jobs to be able to handle skew in 
> region sizes. The algorithm is roughly: 
> {code}
> If (region size >= average size*ratio) : cut the region into two MR input 
> splits
> If (average size <= region size < average size*ratio) : one region as one MR 
> input split
> If (sum of several continuous regions size < average size * ratio): combine 
> these regions into one MR input split.
> {code}
> Although we can set data skew ratio to be 0.5 or something to abuse 
> HBASE-12590 into creating more than 1 split task per region, it is not ideal. 
> But there is no way to create more with the patch as it is. For example we 
> cannot create more than 2 tasks per region. 
> If we want to fix this properly, we should extend the approach in 
> HBASE-12590, and make it so that the client can specify the desired num of 
> mappers, or desired split size, and the TIF generates the splits based on the 
> current region sizes very similar to the algorithm in HBASE-12590, but a more 
> generic way. This also would eliminate the hand tuning of data skew ratio.
> We also can think about the guidepost approach that Phoenix has in the stats 
> table which is used for exactly this purpose. Right now, the region can be 
> split into powers of two assuming uniform distribution within the region. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17072) CPU usage starts to climb up to 90-100% when using G1GC

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678305#comment-15678305
 ] 

stack commented on HBASE-17072:
---

HBASE-10676's soln is to cache the header on the reader instance rather than by 
thread. Its a good idea. It removes Handler lottery but it  would be thrown off 
if two threads contending against same file.

The caching of header came in w/ the original fb hfilev2 patch in HBASE-3857 in 
time for 0.92 so it has been a 'problem' for a long time.





> CPU usage starts to climb up to 90-100% when using G1GC
> ---
>
> Key: HBASE-17072
> URL: https://issues.apache.org/jira/browse/HBASE-17072
> Project: HBase
>  Issue Type: Bug
>  Components: Performance, regionserver
>Affects Versions: 1.0.0, 1.2.0
>Reporter: Eiichi Sato
> Attachments: disable-block-header-cache.patch, mat-threadlocals.png, 
> mat-threads.png, metrics.png, slave1.svg, slave2.svg, slave3.svg, slave4.svg
>
>
> h5. Problem
> CPU usage of a region server in our CDH 5.4.5 cluster, at some point, starts 
> to gradually get higher up to nearly 90-100% when using G1GC.  We've also run 
> into this problem on CDH 5.7.3 and CDH 5.8.2.
> In our production cluster, it normally takes a few weeks for this to happen 
> after restarting a RS.  We reproduced this on our test cluster and attached 
> the results.  Please note that, to make it easy to reproduce, we did some 
> "anti-tuning" on a table when running tests.
> In metrics.png, soon after we started running some workloads against a test 
> cluster (CDH 5.8.2) at about 7 p.m. CPU usage of the two RSs started to rise. 
>  Flame Graphs (slave1.svg to slave4.svg) are generated from jstack dumps of 
> each RS process around 10:30 a.m. the next day.
> After investigating heapdumps from another occurrence on a test cluster 
> running CDH 5.7.3, we found that the ThreadLocalMap contain a lot of 
> contiguous entries of {{HFileBlock$PrefetchedHeader}} probably due to primary 
> clustering.  This caused more loops in 
> {{ThreadLocalMap#expungeStaleEntries()}}, consuming a certain amount of CPU 
> time.  What is worse is that the method is called from RPC metrics code, 
> which means even a small amount of per-RPC time soon adds up to a huge amount 
> of CPU time.
> This is very similar to the issue in HBASE-16616, but we have many 
> {{HFileBlock$PrefetchedHeader}} not only {{Counter$IndexHolder}} instances.  
> Here are some OQL counts from Eclipse Memory Analyzer (MAT).  This shows a 
> number of ThreadLocal instances in the ThreadLocalMap of a single handler 
> thread.
> {code}
> SELECT *
> FROM OBJECTS (SELECT AS RETAINED SET OBJECTS value
> FROM OBJECTS 0x4ee380430) obj
> WHERE obj.@clazz.@name = 
> "org.apache.hadoop.hbase.io.hfile.HFileBlock$PrefetchedHeader"
> #=> 10980 instances
> {code}
> {code}
> SELECT *
> FROM OBJECTS (SELECT AS RETAINED SET OBJECTS value
> FROM OBJECTS 0x4ee380430) obj
> WHERE obj.@clazz.@name = "org.apache.hadoop.hbase.util.Counter$IndexHolder"
> #=> 2052 instances
> {code}
> Although as described in HBASE-16616 this somewhat seems to be an issue in 
> G1GC side regarding weakly-reachable objects, we should keep ThreadLocal 
> usage minimal and avoid creating an indefinite number (in this case, a number 
> of HFiles) of ThreadLocal instances.
> HBASE-16146 removes ThreadLocals from the RPC metrics code.  That may solve 
> the issue (I just saw the patch, never tested it at all), but the 
> {{HFileBlock$PrefetchedHeader}} are still there in the ThreadLocalMap, which 
> may cause issues in the future again.
> h5. Our Solution
> We simply removed the whole {{HFileBlock$PrefetchedHeader}} caching and 
> fortunately we didn't notice any performance degradation for our production 
> workloads.
> Because the PrefetchedHeader caching uses ThreadLocal and because RPCs are 
> handled randomly in any of the handlers, small Get or small Scan RPCs do not 
> benefit from the caching (See HBASE-10676 and HBASE-11402 for the details).  
> Probably, we need to see how well reads are saved by the caching for large 
> Scan or Get RPCs and especially for compactions if we really remove the 
> caching. It's probably better if we can remove ThreadLocals without breaking 
> the current caching behavior.
> FWIW, I'm attaching the patch we applied. It's for CDH 5.4.5.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17123) Add postBulkLoadHFile variant that notifies the final paths for the hfiles

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678282#comment-15678282
 ] 

Hadoop QA commented on HBASE-17123:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 2s 
{color} | {color:blue} The patch file was not named according to hbase's naming 
conventions. Please see 
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for 
instructions. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
33s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 6s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 55s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
28m 35s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m 16s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 29s 
{color} | {color:green} hbase-endpoint in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
26s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 137m 21s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.hbase.io.asyncfs.TestSaslFanOutOneBlockAsyncDFSOutput |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12839625/17123.v4.txt |
| JIRA Issue | HBASE-17123 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux b10c30213018 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / be519ca |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit | 

[jira] [Updated] (HBASE-16912) TEST: update HBaseTestingUtility to avoid direct use of filesystem / legacy implementation

[Umesh Agashe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Umesh Agashe updated HBASE-16912:
-
Attachment: HBASE-16912.hbase-14439.002.review

[~appy], changes look good to me. Please get review comments from [~busbey] as 
well before committing the changes. I've a few minor comments. Please see the 
attached .review files for my comments.

> TEST: update HBaseTestingUtility to avoid direct use of filesystem / legacy 
> implementation
> --
>
> Key: HBASE-16912
> URL: https://issues.apache.org/jira/browse/HBASE-16912
> Project: HBase
>  Issue Type: Sub-task
>  Components: Filesystem Integration
>Reporter: Sean Busbey
> Attachments: HBASE-16912.hbase-14439.001.patch, 
> HBASE-16912.hbase-14439.002.patch, HBASE-16912.hbase-14439.002.review, 
> fs-redo-HBTU.patch
>
>
> Update the HBaseTestingUtility to ensure it relies on MasterStorage / 
> RegionStorage APIs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17117) Reversed scan returns deleted versions and breaks RegionLocator

[Ashu Pachauri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashu Pachauri updated HBASE-17117:
--
Attachment: reverseScanMetaDeleteFamily.txt

reverseScanMetaDeleteFamily.txt : Reattaching with txt extension (Jira does not 
open the file in browser otherwise! )

> Reversed scan returns deleted versions and breaks RegionLocator
> ---
>
> Key: HBASE-17117
> URL: https://issues.apache.org/jira/browse/HBASE-17117
> Project: HBase
>  Issue Type: Bug
>  Components: Scanners
>Affects Versions: 1.3.0
>Reporter: Ashu Pachauri
> Fix For: 1.3.0
>
> Attachments: reverseScanMetaDeleteFamily.txt
>
>
> We started seeing clients persistently throwing errors as they were trying to 
> talk to a region that was non existent (split a few days ago). We verified 
> that the region was deleted from meta when the split happened.
> On performing a raw scan on meta, the deleted version for the split region 
> appears, which also does on performing a normal reversed scan. Since 
> MetaScanner uses a reversed scan, this explains why clients see non existent 
> regions.
> We also verified that there was no in-memory corrupt state by failing over 
> the master. When we trigger major compaction on meta, the problem goes away 
> further confirming the fact that we were seeing deleted versions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HBASE-16912) TEST: update HBaseTestingUtility to avoid direct use of filesystem / legacy implementation

[Umesh Agashe (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678256#comment-15678256
 ] 

Umesh Agashe edited comment on HBASE-16912 at 11/19/16 12:52 AM:
-

[~appy], changes look good to me. Please get review comments from [~busbey] as 
well before committing the changes. I've a few minor comments. Please search 
for 'uagashe>>>' string in the attached .review files for my comments.


was (Author: uagashe):
[~appy], changes look good to me. Please get review comments from [~busbey] as 
well before committing the changes. I've a few minor comments. Please see the 
attached .review files for my comments.

> TEST: update HBaseTestingUtility to avoid direct use of filesystem / legacy 
> implementation
> --
>
> Key: HBASE-16912
> URL: https://issues.apache.org/jira/browse/HBASE-16912
> Project: HBase
>  Issue Type: Sub-task
>  Components: Filesystem Integration
>Reporter: Sean Busbey
> Attachments: HBASE-16912.hbase-14439.001.patch, 
> HBASE-16912.hbase-14439.002.patch, HBASE-16912.hbase-14439.002.review, 
> fs-redo-HBTU.patch
>
>
> Update the HBaseTestingUtility to ensure it relies on MasterStorage / 
> RegionStorage APIs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17117) Reversed scan returns deleted versions and breaks RegionLocator

[Ashu Pachauri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashu Pachauri updated HBASE-17117:
--
Attachment: (was: reverseScanMetaDeleteFamily)

> Reversed scan returns deleted versions and breaks RegionLocator
> ---
>
> Key: HBASE-17117
> URL: https://issues.apache.org/jira/browse/HBASE-17117
> Project: HBase
>  Issue Type: Bug
>  Components: Scanners
>Affects Versions: 1.3.0
>Reporter: Ashu Pachauri
> Fix For: 1.3.0
>
>
> We started seeing clients persistently throwing errors as they were trying to 
> talk to a region that was non existent (split a few days ago). We verified 
> that the region was deleted from meta when the split happened.
> On performing a raw scan on meta, the deleted version for the split region 
> appears, which also does on performing a normal reversed scan. Since 
> MetaScanner uses a reversed scan, this explains why clients see non existent 
> regions.
> We also verified that there was no in-memory corrupt state by failing over 
> the master. When we trigger major compaction on meta, the problem goes away 
> further confirming the fact that we were seeing deleted versions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17117) Reversed scan returns deleted versions and breaks RegionLocator

[Ashu Pachauri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashu Pachauri updated HBASE-17117:
--
Attachment: reverseScanMetaDeleteFamily

> Reversed scan returns deleted versions and breaks RegionLocator
> ---
>
> Key: HBASE-17117
> URL: https://issues.apache.org/jira/browse/HBASE-17117
> Project: HBase
>  Issue Type: Bug
>  Components: Scanners
>Affects Versions: 1.3.0
>Reporter: Ashu Pachauri
> Fix For: 1.3.0
>
> Attachments: reverseScanMetaDeleteFamily
>
>
> We started seeing clients persistently throwing errors as they were trying to 
> talk to a region that was non existent (split a few days ago). We verified 
> that the region was deleted from meta when the split happened.
> On performing a raw scan on meta, the deleted version for the split region 
> appears, which also does on performing a normal reversed scan. Since 
> MetaScanner uses a reversed scan, this explains why clients see non existent 
> regions.
> We also verified that there was no in-memory corrupt state by failing over 
> the master. When we trigger major compaction on meta, the problem goes away 
> further confirming the fact that we were seeing deleted versions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17117) Reversed scan returns deleted versions and breaks RegionLocator

[Ashu Pachauri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashu Pachauri updated HBASE-17117:
--
Attachment: (was: reverseScanMetaDeleteFamily)

> Reversed scan returns deleted versions and breaks RegionLocator
> ---
>
> Key: HBASE-17117
> URL: https://issues.apache.org/jira/browse/HBASE-17117
> Project: HBase
>  Issue Type: Bug
>  Components: Scanners
>Affects Versions: 1.3.0
>Reporter: Ashu Pachauri
> Fix For: 1.3.0
>
>
> We started seeing clients persistently throwing errors as they were trying to 
> talk to a region that was non existent (split a few days ago). We verified 
> that the region was deleted from meta when the split happened.
> On performing a raw scan on meta, the deleted version for the split region 
> appears, which also does on performing a normal reversed scan. Since 
> MetaScanner uses a reversed scan, this explains why clients see non existent 
> regions.
> We also verified that there was no in-memory corrupt state by failing over 
> the master. When we trigger major compaction on meta, the problem goes away 
> further confirming the fact that we were seeing deleted versions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17117) Reversed scan returns deleted versions and breaks RegionLocator

[Ashu Pachauri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashu Pachauri updated HBASE-17117:
--
Attachment: reverseScanMetaDeleteFamily

reverseScanMetaDeleteFamily : Output of raw, forward and reversed scan on meta 
during the issue.
I have replaced the start and end keys with dummy values (but in correct order).

We did not back up the store files for meta, and I am unable to reproduce the 
issue just by creating a delete family marker in meta and doing a reverse scan. 
This is not happening consistently during splits either.  So, I am dropping the 
priority down to major from blocker. I'll try to research edge cases in the 
scanner code and come up with a unit test for reproducing this. Any ideas are 
welcome.

> Reversed scan returns deleted versions and breaks RegionLocator
> ---
>
> Key: HBASE-17117
> URL: https://issues.apache.org/jira/browse/HBASE-17117
> Project: HBase
>  Issue Type: Bug
>  Components: Scanners
>Affects Versions: 1.3.0
>Reporter: Ashu Pachauri
>Priority: Blocker
> Fix For: 1.3.0
>
> Attachments: reverseScanMetaDeleteFamily
>
>
> We started seeing clients persistently throwing errors as they were trying to 
> talk to a region that was non existent (split a few days ago). We verified 
> that the region was deleted from meta when the split happened.
> On performing a raw scan on meta, the deleted version for the split region 
> appears, which also does on performing a normal reversed scan. Since 
> MetaScanner uses a reversed scan, this explains why clients see non existent 
> regions.
> We also verified that there was no in-memory corrupt state by failing over 
> the master. When we trigger major compaction on meta, the problem goes away 
> further confirming the fact that we were seeing deleted versions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17117) Reversed scan returns deleted versions and breaks RegionLocator

[Ashu Pachauri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashu Pachauri updated HBASE-17117:
--
Priority: Major  (was: Blocker)

> Reversed scan returns deleted versions and breaks RegionLocator
> ---
>
> Key: HBASE-17117
> URL: https://issues.apache.org/jira/browse/HBASE-17117
> Project: HBase
>  Issue Type: Bug
>  Components: Scanners
>Affects Versions: 1.3.0
>Reporter: Ashu Pachauri
> Fix For: 1.3.0
>
> Attachments: reverseScanMetaDeleteFamily
>
>
> We started seeing clients persistently throwing errors as they were trying to 
> talk to a region that was non existent (split a few days ago). We verified 
> that the region was deleted from meta when the split happened.
> On performing a raw scan on meta, the deleted version for the split region 
> appears, which also does on performing a normal reversed scan. Since 
> MetaScanner uses a reversed scan, this explains why clients see non existent 
> regions.
> We also verified that there was no in-memory corrupt state by failing over 
> the master. When we trigger major compaction on meta, the problem goes away 
> further confirming the fact that we were seeing deleted versions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17051) libhbase++: implement RPC client and connection management

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678233#comment-15678233
 ] 

Hadoop QA commented on HBASE-17051:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m 16s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m 7s 
{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 1s 
{color} | {color:green} HBASE-14850 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 2s 
{color} | {color:green} HBASE-14850 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 1s 
{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 1s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 1s 
{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 1s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m 
4s {color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
32m 13s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
3s {color} | {color:green} the patch passed {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 51s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:date2016-11-18 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12839645/HBASE-17051-HBASE-14850.001.patch
 |
| JIRA Issue | HBASE-17051 |
| Optional Tests |  cc  compile  shellcheck  shelldocs  |
| uname | Linux ab7fb5a1d8bc 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | nobuild |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
 |
| git revision | HBASE-14850 / 5742267 |
| shellcheck | v0.4.5 |
| modules | C: . U: . |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4540/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> libhbase++: implement RPC client and connection management
> --
>
> Key: HBASE-17051
> URL: https://issues.apache.org/jira/browse/HBASE-17051
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HBASE-17051-HBASE-14850.000.patch, 
> HBASE-17051-HBASE-14850.001.patch
>
>
> This proposes building RPC client and connection management layer, which 
> supports the equivalent functions resides in RpcClient.java and 
> RpcConnection.java. Specifically, handler/pipeline concepts will be used for 
> implementation, similar to NettyRpcClient and NettyRpcConnection in java side.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17051) libhbase++: implement RPC client and connection management

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678165#comment-15678165
 ] 

Hadoop QA commented on HBASE-17051:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m 44s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m 6s 
{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 1s 
{color} | {color:green} HBASE-14850 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 2s 
{color} | {color:green} HBASE-14850 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 1s 
{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 1s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 1s 
{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 1s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m 
4s {color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
27m 44s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
3s {color} | {color:green} the patch passed {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 50s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-11-18 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12839632/HBASE-17051-HBASE-14850.000.patch
 |
| JIRA Issue | HBASE-17051 |
| Optional Tests |  cc  compile  shellcheck  shelldocs  |
| uname | Linux baeeaa4defee 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | nobuild |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | HBASE-14850 / 5742267 |
| shellcheck | v0.4.5 |
| modules | C: . U: . |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4539/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> libhbase++: implement RPC client and connection management
> --
>
> Key: HBASE-17051
> URL: https://issues.apache.org/jira/browse/HBASE-17051
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HBASE-17051-HBASE-14850.000.patch, 
> HBASE-17051-HBASE-14850.001.patch
>
>
> This proposes building RPC client and connection management layer, which 
> supports the equivalent functions resides in RpcClient.java and 
> RpcConnection.java. Specifically, handler/pipeline concepts will be used for 
> implementation, similar to NettyRpcClient and NettyRpcConnection in java side.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17080) rest.TestTableResource fails in master branch

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678147#comment-15678147
 ] 

Hudson commented on HBASE-17080:


SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #1977 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/1977/])
HBASE-17080 rest.TestTableResource fails in master branch (ChiaPing (tedyu: rev 
72438c02237bcaca10939469dc160f0820b5d2aa)
* (edit) 
hbase-rest/src/test/java/org/apache/hadoop/hbase/rest/TestTableResource.java


> rest.TestTableResource fails in master branch
> -
>
> Key: HBASE-17080
> URL: https://issues.apache.org/jira/browse/HBASE-17080
> Project: HBase
>  Issue Type: Test
>Affects Versions: 2.0.0
>Reporter: Ted Yu
>Assignee: ChiaPing Tsai
> Fix For: 2.0.0
>
> Attachments: HBASE-17080.v0.patch
>
>
> The test fails consistently.
> {code}
> testTableInfoPB(org.apache.hadoop.hbase.rest.TestTableResource)  Time 
> elapsed: 0.118 sec  <<< FAILURE!
> org.junit.ComparisonFailure: expected:<192.168.0.14:5055[8]> but 
> was:<192.168.0.14:5055[5]>
>   at org.junit.Assert.assertEquals(Assert.java:115)
>   at org.junit.Assert.assertEquals(Assert.java:144)
>   at 
> org.apache.hadoop.hbase.rest.TestTableResource.checkTableInfo(TestTableResource.java:191)
>   at 
> org.apache.hadoop.hbase.rest.TestTableResource.testTableInfoPB(TestTableResource.java:272)
> testTableInfoXML(org.apache.hadoop.hbase.rest.TestTableResource)  Time 
> elapsed: 0.084 sec  <<< FAILURE!
> org.junit.ComparisonFailure: expected:<192.168.0.14:5055[8]> but 
> was:<192.168.0.14:5055[5]>
>   at org.junit.Assert.assertEquals(Assert.java:115)
>   at org.junit.Assert.assertEquals(Assert.java:144)
>   at 
> org.apache.hadoop.hbase.rest.TestTableResource.checkTableInfo(TestTableResource.java:191)
>   at 
> org.apache.hadoop.hbase.rest.TestTableResource.testTableInfoXML(TestTableResource.java:255)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16708) Expose endpoint Coprocessor name in "responseTooSlow" log messages

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678146#comment-15678146
 ] 

Hudson commented on HBASE-16708:


SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #1977 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/1977/])
HBASE-16708 Expose endpoint Coprocessor name in responseTooSlow log (jerryjch: 
rev a8ee83c092c5ff22923ac359f2969b6601a01ca3)
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java


> Expose endpoint Coprocessor name in "responseTooSlow" log messages
> --
>
> Key: HBASE-16708
> URL: https://issues.apache.org/jira/browse/HBASE-16708
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 1.0.0
>Reporter: Nick Dimiduk
>Assignee: Yi Liang
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16708-Branch1.patch, HBASE-16708-V1.patch, 
> HBASE-16708-V2.patch, HBASE-16708-V3.patch, HBASE-ShortName.patch
>
>
> Operational diagnostics of a Phoenix install would be easier if we included 
> which endpoint coprocessor was being called in this responseTooSlow WARN 
> message.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17051) libhbase++: implement RPC client and connection management

[Xiaobing Zhou (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678139#comment-15678139
 ] 

Xiaobing Zhou commented on HBASE-17051:
---

Posted another patch v001. It 
# implements RpcClient::close
# makes RpcConnection::close and RpcClient::close protected to avoid being 
called outside since they are invoked by destructor automatically.

> libhbase++: implement RPC client and connection management
> --
>
> Key: HBASE-17051
> URL: https://issues.apache.org/jira/browse/HBASE-17051
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HBASE-17051-HBASE-14850.000.patch, 
> HBASE-17051-HBASE-14850.001.patch
>
>
> This proposes building RPC client and connection management layer, which 
> supports the equivalent functions resides in RpcClient.java and 
> RpcConnection.java. Specifically, handler/pipeline concepts will be used for 
> implementation, similar to NettyRpcClient and NettyRpcConnection in java side.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17051) libhbase++: implement RPC client and connection management

[Xiaobing Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HBASE-17051:
--
Attachment: HBASE-17051-HBASE-14850.001.patch

> libhbase++: implement RPC client and connection management
> --
>
> Key: HBASE-17051
> URL: https://issues.apache.org/jira/browse/HBASE-17051
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HBASE-17051-HBASE-14850.000.patch, 
> HBASE-17051-HBASE-14850.001.patch
>
>
> This proposes building RPC client and connection management layer, which 
> supports the equivalent functions resides in RpcClient.java and 
> RpcConnection.java. Specifically, handler/pipeline concepts will be used for 
> implementation, similar to NettyRpcClient and NettyRpcConnection in java side.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13395) Remove HTableInterface

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-13395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678121#comment-15678121
 ] 

Hadoop QA commented on HBASE-13395:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
9s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
28m 31s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 1s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 58s 
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
6s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 43s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12839628/HBASE-13395.master.001.patch
 |
| JIRA Issue | HBASE-13395 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux aa5e493a5b10 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
 |
| git revision | master / be519ca |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4538/testReport/ |
| modules | C: hbase-client U: hbase-client |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4538/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> Remove HTableInterface
> --
>
> Key: HBASE-13395
> URL: https://issues.apache.org/jira/browse/HBASE-13395
> Project: HBase
>  Issue Type: Sub-task
>  Components: API
>Affects Versions: 2.0.0
>Reporter: Mikhail Antonov
>Assignee: Jan Hentschel
>  

[jira] [Commented] (HBASE-16894) Create more than 1 split per region, generalize HBASE-12590

[Esteban Gutierrez (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678115#comment-15678115
 ] 

Esteban Gutierrez commented on HBASE-16894:
---

[~enis] I've created another JIRA so we can discuss the WRITE case there: 
HBASE-17130. The expectation for a user would be to specify a fixed number of 
reducer. We have been discussing an interesting edge case when a table has some 
skew due the key design and it would be helpful to write in a single reducer 
HFiles for multiple regions.

> Create more than 1 split per region, generalize HBASE-12590
> ---
>
> Key: HBASE-16894
> URL: https://issues.apache.org/jira/browse/HBASE-16894
> Project: HBase
>  Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Yi Liang
>  Labels: beginner, beginners
>
> A common request from users is to be able to better control how many map 
> tasks are created per region. Right now, it is always 1 region = 1 input 
> split = 1 map task. Same goes for Spark since it uses the TIF. With region 
> sizes as large as 50 GBs, it is desirable to be able to create more than 1 
> split per region.
> HBASE-12590 adds a config property for MR jobs to be able to handle skew in 
> region sizes. The algorithm is roughly: 
> {code}
> If (region size >= average size*ratio) : cut the region into two MR input 
> splits
> If (average size <= region size < average size*ratio) : one region as one MR 
> input split
> If (sum of several continuous regions size < average size * ratio): combine 
> these regions into one MR input split.
> {code}
> Although we can set data skew ratio to be 0.5 or something to abuse 
> HBASE-12590 into creating more than 1 split task per region, it is not ideal. 
> But there is no way to create more with the patch as it is. For example we 
> cannot create more than 2 tasks per region. 
> If we want to fix this properly, we should extend the approach in 
> HBASE-12590, and make it so that the client can specify the desired num of 
> mappers, or desired split size, and the TIF generates the splits based on the 
> current region sizes very similar to the algorithm in HBASE-12590, but a more 
> generic way. This also would eliminate the hand tuning of data skew ratio.
> We also can think about the guidepost approach that Phoenix has in the stats 
> table which is used for exactly this purpose. Right now, the region can be 
> split into powers of two assuming uniform distribution within the region. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-17130) Add support to specify an arbitrary number of reducers when writing HFiles for bulk load

[Esteban Gutierrez (JIRA)]

Esteban Gutierrez created HBASE-17130:
-

 Summary: Add support to specify an arbitrary number of reducers 
when writing HFiles for bulk load
 Key: HBASE-17130
 URL: https://issues.apache.org/jira/browse/HBASE-17130
 Project: HBase
  Issue Type: Improvement
  Components: mapreduce
Reporter: Esteban Gutierrez


>From the discussion from HBASE-16894 there is a set of use cases where writing 
>to multiple regions in a single reducer can be helpful to reduce the overhead 
>of MR jobs when a large number of regions exist in an HBase cluster and some 
>regions can present a data skew, e.g. 100s or 1000s of regions with a very 
>small number of rows vs. regions with 10s or millions or rows as part of the 
>same job. And merging regions is not an option for the use case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17122) Change in behavior when creating a scanner for a disabled table

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678104#comment-15678104
 ] 

Andrew Purtell commented on HBASE-17122:


We will have another, perhaps final 0.98 release in December. Does this affect 
other branches? If so let's fix for sure. If not, what's the opinion on fix or 
not?

> Change in behavior when creating a scanner for a disabled table
> ---
>
> Key: HBASE-17122
> URL: https://issues.apache.org/jira/browse/HBASE-17122
> Project: HBase
>  Issue Type: Bug
>Reporter: Samarth Jain
>
> {code}
> @Test
> public void testQueryingDisabledTable() throws Exception {
> try (Connection conn = DriverManager.getConnection(getUrl())) {
> String tableName = generateUniqueName();
> conn.createStatement().execute(
> "CREATE TABLE " + tableName
> + " (k1 VARCHAR NOT NULL, k2 VARCHAR, CONSTRAINT PK 
> PRIMARY KEY(K1,K2)) ");
> try (HBaseAdmin admin = 
> conn.unwrap(PhoenixConnection.class).getQueryServices().getAdmin()) {
> admin.disableTable(Bytes.toBytes(tableName));
> }
> String query = "SELECT * FROM " + tableName + " WHERE 1=1";
> try (Connection conn2 = DriverManager.getConnection(getUrl())) {
> try (ResultSet rs = 
> conn2.createStatement().executeQuery(query)) {
> assertFalse(rs.next());
> }
> }
> }
> }
> {code}
> This is a phoenix specific test case. I will try an come up with something 
> using the HBase API. But the gist is that with HBase 0.98.21 and beyond, we 
> are seeing that creating a scanner is throwing a NotServingRegionException. 
> Stacktrace for NotServingRegionException
> {code}
> org.apache.phoenix.exception.PhoenixIOException: 
> org.apache.phoenix.exception.PhoenixIOException: callTimeout=120, 
> callDuration=9000104: row '' on table 'T01' at 
> region=T01,,1479429739864.643dde31cc19b549192576eea7791a6f., 
> hostname=localhost,60022,1479429692090, seqNum=1
>   at 
> org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:113)
>   at 
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:752)
>   at 
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:696)
>   at 
> org.apache.phoenix.iterate.ConcatResultIterator.getIterators(ConcatResultIterator.java:50)
>   at 
> org.apache.phoenix.iterate.ConcatResultIterator.currentIterator(ConcatResultIterator.java:97)
>   at 
> org.apache.phoenix.iterate.ConcatResultIterator.next(ConcatResultIterator.java:117)
>   at 
> org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:778)
>   at 
> org.apache.phoenix.end2end.PhoenixRuntimeIT.testQueryingDisabledTable(PhoenixRuntimeIT.java:167)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at 
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
>   at 
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
>   at 
> 

[jira] [Commented] (HBASE-16002) Many DataType constructors are not public

[Nick Dimiduk (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678102#comment-15678102
 ] 

Nick Dimiduk commented on HBASE-16002:
--

Maybe this is invalid actually. Notice most of these have {{public static 
ASCENDING, DESCENDING}} instances.

> Many DataType constructors are not public
> -
>
> Key: HBASE-16002
> URL: https://issues.apache.org/jira/browse/HBASE-16002
> Project: HBase
>  Issue Type: Bug
>Reporter: Nick Dimiduk
>Assignee: Jan Hentschel
> Attachments: HBASE-16002.master.001.patch
>
>
> Using the {{Struct}} class to build a rowkey, I noticed than many of the 
> provided {{DataType}} implementations' constructors are protected. They 
> should be public in most (all?) cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17051) libhbase++: implement RPC client and connection management

[Xiaobing Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HBASE-17051:
--
Status: Patch Available  (was: Open)

> libhbase++: implement RPC client and connection management
> --
>
> Key: HBASE-17051
> URL: https://issues.apache.org/jira/browse/HBASE-17051
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HBASE-17051-HBASE-14850.000.patch
>
>
> This proposes building RPC client and connection management layer, which 
> supports the equivalent functions resides in RpcClient.java and 
> RpcConnection.java. Specifically, handler/pipeline concepts will be used for 
> implementation, similar to NettyRpcClient and NettyRpcConnection in java side.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HBASE-17051) libhbase++: implement RPC client and connection management

[Xiaobing Zhou (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678067#comment-15678067
 ] 

Xiaobing Zhou edited comment on HBASE-17051 at 11/18/16 11:14 PM:
--

I posted the initial patch v000, and will add tests in next patch.
# built User class with user name only inside for simplicity.
# built ConnectionId class that packs user and server info(i.e. 
hbase::pb::ServerName), which is used to identify connection uniquely.
# built RpcConnection class that packs ConnectionId and HBaseService (i.e. 
wangle::Service). HBaseService represents 
various remote RPC APIs HBase implemented. RpcConnection will forward requests 
to HBaseService.
# built RpcClient that handles lifecycle of RpcConnections and offers 
sync/async call. 


was (Author: xiaobingo):
I posted the initial patch v000, and will add tests in next patch.
#built User class with user name only inside for simplicity.
#built ConnectionId class that packs user and server info(i.e. 
hbase::pb::ServerName), which is used to identify connection uniquely.
#built RpcConnection class that packs ConnectionId and HBaseService (i.e. 
wangle::Service). HBaseService represents 
various remote RPC APIs HBase implemented. RpcConnection will forward requests 
to HBaseService.
#built RpcClient that handles lifecycle of RpcConnections and offers sync/async 
call. 

> libhbase++: implement RPC client and connection management
> --
>
> Key: HBASE-17051
> URL: https://issues.apache.org/jira/browse/HBASE-17051
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HBASE-17051-HBASE-14850.000.patch
>
>
> This proposes building RPC client and connection management layer, which 
> supports the equivalent functions resides in RpcClient.java and 
> RpcConnection.java. Specifically, handler/pipeline concepts will be used for 
> implementation, similar to NettyRpcClient and NettyRpcConnection in java side.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16894) Create more than 1 split per region, generalize HBASE-12590

[Yi Liang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678066#comment-15678066
 ] 

Yi Liang commented on HBASE-16894:
--

Thanks [~enis] for your reply, it is really helpful :). I will do more research 
and focus on this jira to see if I can come up with better solution.

> Create more than 1 split per region, generalize HBASE-12590
> ---
>
> Key: HBASE-16894
> URL: https://issues.apache.org/jira/browse/HBASE-16894
> Project: HBase
>  Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Yi Liang
>  Labels: beginner, beginners
>
> A common request from users is to be able to better control how many map 
> tasks are created per region. Right now, it is always 1 region = 1 input 
> split = 1 map task. Same goes for Spark since it uses the TIF. With region 
> sizes as large as 50 GBs, it is desirable to be able to create more than 1 
> split per region.
> HBASE-12590 adds a config property for MR jobs to be able to handle skew in 
> region sizes. The algorithm is roughly: 
> {code}
> If (region size >= average size*ratio) : cut the region into two MR input 
> splits
> If (average size <= region size < average size*ratio) : one region as one MR 
> input split
> If (sum of several continuous regions size < average size * ratio): combine 
> these regions into one MR input split.
> {code}
> Although we can set data skew ratio to be 0.5 or something to abuse 
> HBASE-12590 into creating more than 1 split task per region, it is not ideal. 
> But there is no way to create more with the patch as it is. For example we 
> cannot create more than 2 tasks per region. 
> If we want to fix this properly, we should extend the approach in 
> HBASE-12590, and make it so that the client can specify the desired num of 
> mappers, or desired split size, and the TIF generates the splits based on the 
> current region sizes very similar to the algorithm in HBASE-12590, but a more 
> generic way. This also would eliminate the hand tuning of data skew ratio.
> We also can think about the guidepost approach that Phoenix has in the stats 
> table which is used for exactly this purpose. Right now, the region can be 
> split into powers of two assuming uniform distribution within the region. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17051) libhbase++: implement RPC client and connection management

[Xiaobing Zhou (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678067#comment-15678067
 ] 

Xiaobing Zhou commented on HBASE-17051:
---

I posted the initial patch v000, and will add tests in next patch.
#built User class with user name only inside for simplicity.
#built ConnectionId class that packs user and server info(i.e. 
hbase::pb::ServerName), which is used to identify connection uniquely.
#built RpcConnection class that packs ConnectionId and HBaseService (i.e. 
wangle::Service). HBaseService represents 
various remote RPC APIs HBase implemented. RpcConnection will forward requests 
to HBaseService.
#built RpcClient that handles lifecycle of RpcConnections and offers sync/async 
call. 

> libhbase++: implement RPC client and connection management
> --
>
> Key: HBASE-17051
> URL: https://issues.apache.org/jira/browse/HBASE-17051
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HBASE-17051-HBASE-14850.000.patch
>
>
> This proposes building RPC client and connection management layer, which 
> supports the equivalent functions resides in RpcClient.java and 
> RpcConnection.java. Specifically, handler/pipeline concepts will be used for 
> implementation, similar to NettyRpcClient and NettyRpcConnection in java side.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-8997) Restore the disabled test TestHCM.testDeleteForZKConnLeak

[Jan Hentschel (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678056#comment-15678056
 ] 

Jan Hentschel commented on HBASE-8997:
--

It seems that this test was already removed. This ticket can probably be closed.

> Restore the disabled test TestHCM.testDeleteForZKConnLeak
> -
>
> Key: HBASE-8997
> URL: https://issues.apache.org/jira/browse/HBASE-8997
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>
> hbase-8996 disabled the test because it is flakey.  See hbase-8996 for the 
> thread dump when test was hung.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17051) libhbase++: implement RPC client and connection management

[Xiaobing Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HBASE-17051:
--
Attachment: HBASE-17051-HBASE-14850.000.patch

> libhbase++: implement RPC client and connection management
> --
>
> Key: HBASE-17051
> URL: https://issues.apache.org/jira/browse/HBASE-17051
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HBASE-17051-HBASE-14850.000.patch
>
>
> This proposes building RPC client and connection management layer, which 
> supports the equivalent functions resides in RpcClient.java and 
> RpcConnection.java. Specifically, handler/pipeline concepts will be used for 
> implementation, similar to NettyRpcClient and NettyRpcConnection in java side.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13395) Remove HTableInterface

[Enis Soztutar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-13395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678047#comment-15678047
 ] 

Enis Soztutar commented on HBASE-13395:
---

It is deprecated in 1.0 I think. So we can remove it. 

> Remove HTableInterface
> --
>
> Key: HBASE-13395
> URL: https://issues.apache.org/jira/browse/HBASE-13395
> Project: HBase
>  Issue Type: Sub-task
>  Components: API
>Affects Versions: 2.0.0
>Reporter: Mikhail Antonov
>Assignee: Jan Hentschel
> Fix For: 2.0.0
>
> Attachments: HBASE-13395.master.001.patch
>
>
> This class is marked as deprecated, probably can remove it, and if any 
> methods from this specific class are in active use, need to decide what to do 
> on callers' side. Should be able to replace with just Table interface usage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17051) libhbase++: implement RPC client and connection management

[Xiaobing Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HBASE-17051:
--
Description: 
This proposes building RPC client and connection management layer, which 
supports the equivalent functions resides in RpcClient.java and 
RpcConnection.java. Specifically, handler/pipeline concepts will be used for 
implementation, similar to NettyRpcClient and NettyRpcConnection in java side.


  was:
This proposes building RPC connection management layer, which supports the 
equivalent functions resides in RpcConnection.java. Specifically, 
handler/pipeline concepts will be used for implementation, similar to 
NettyRpcConnection in java side.



> libhbase++: implement RPC client and connection management
> --
>
> Key: HBASE-17051
> URL: https://issues.apache.org/jira/browse/HBASE-17051
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
>
> This proposes building RPC client and connection management layer, which 
> supports the equivalent functions resides in RpcClient.java and 
> RpcConnection.java. Specifically, handler/pipeline concepts will be used for 
> implementation, similar to NettyRpcClient and NettyRpcConnection in java side.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-3935) HServerLoad.storefileIndexSizeMB should be changed to storefileIndexSizeKB

[Jan Hentschel (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-3935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678044#comment-15678044
 ] 

Jan Hentschel commented on HBASE-3935:
--

HServerLoad seems to don't exist anymore. This ticket can probably be closed.

> HServerLoad.storefileIndexSizeMB should be changed to storefileIndexSizeKB
> --
>
> Key: HBASE-3935
> URL: https://issues.apache.org/jira/browse/HBASE-3935
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>
> Related to HBASE-3927, Matt proposed changing 
> HServerLoad.storefileIndexSizeMB to storefileIndexSizeKB so that user can see 
> the size of small store file index.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17051) libhbase++: implement RPC client and connection management

[Xiaobing Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HBASE-17051:
--
Summary: libhbase++: implement RPC client and connection management  (was: 
libhbase++: implement RPC connection management)

> libhbase++: implement RPC client and connection management
> --
>
> Key: HBASE-17051
> URL: https://issues.apache.org/jira/browse/HBASE-17051
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
>
> This proposes building RPC connection management layer, which supports the 
> equivalent functions resides in RpcConnection.java. Specifically, 
> handler/pipeline concepts will be used for implementation, similar to 
> NettyRpcConnection in java side.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16894) Create more than 1 split per region, generalize HBASE-12590

[Enis Soztutar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678039#comment-15678039
 ] 

Enis Soztutar commented on HBASE-16894:
---

Thanks [~easyliangjob] for picking up this. My suggestion is to start with at 
least these requirements: 
For jobs that READ from HBase, we would like the user to be able to use any of 
these three methods:  
(1) Users should be able to specify total number of maps desired, and HBase 
comes up with Splits grouping / splitting regions based on sizes and number of 
required maps. Example is that if table has 100 regions, but we want the MR 
jobs to run with 200 splits, or if table has 100 regions but we want to run 
only 50 map tasks. 
(2) Users should be able to specify the input split size, which will then 
determine how many map tasks are created. User will specify the size as let's 
say 2GB, and HBase TableInputFormat will create 1 split per roughly 2 GB of 
data depending on existing region sizes (An 8GB region will be split into 4, 
two 1GB neighbor regions are combined into 2GB regions, etc). 
(3) Users should be able to specify how many splits per region she wants. If 
100 regions, and tasks per region is 3, then we will generate 300 splits. The 
current way HBase works will be using 1 task-per-region as default. 

For Jobs that WRITE to HBase, then we should be able to specify any of these 
two methods: 
(1) Users should be able to specify total number of reducers desired, and HBase 
comes up with partitioning based on region boundaries. Note that especially in 
bulk load, we do not want to group more than 1 region per task (unlike the map 
case). The number of requested reducers will be just a hint, and HBase might 
end up generating more reducers. 
(2) Users should be able to specify how many reducers per region we want. 

Let me know if makes sense or not. There is usually no good way to know the 
exact split points within the region to see how you can divide the region into 
N pieces. The code that HBASE-12590 assumes text or binary keys and uniform 
distribution. We can also look into improving that, by asking the region server 
hosting the region about the split points (which should be calculated from 
hfile indexes, similar to the mid point calculation that we have in region 
splits) or we can look into maintaining equi-width histograms for keys (Phoenix 
guideposts). We can do these suggestions in a follow up issue. 

> Create more than 1 split per region, generalize HBASE-12590
> ---
>
> Key: HBASE-16894
> URL: https://issues.apache.org/jira/browse/HBASE-16894
> Project: HBase
>  Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Yi Liang
>  Labels: beginner, beginners
>
> A common request from users is to be able to better control how many map 
> tasks are created per region. Right now, it is always 1 region = 1 input 
> split = 1 map task. Same goes for Spark since it uses the TIF. With region 
> sizes as large as 50 GBs, it is desirable to be able to create more than 1 
> split per region.
> HBASE-12590 adds a config property for MR jobs to be able to handle skew in 
> region sizes. The algorithm is roughly: 
> {code}
> If (region size >= average size*ratio) : cut the region into two MR input 
> splits
> If (average size <= region size < average size*ratio) : one region as one MR 
> input split
> If (sum of several continuous regions size < average size * ratio): combine 
> these regions into one MR input split.
> {code}
> Although we can set data skew ratio to be 0.5 or something to abuse 
> HBASE-12590 into creating more than 1 split task per region, it is not ideal. 
> But there is no way to create more with the patch as it is. For example we 
> cannot create more than 2 tasks per region. 
> If we want to fix this properly, we should extend the approach in 
> HBASE-12590, and make it so that the client can specify the desired num of 
> mappers, or desired split size, and the TIF generates the splits based on the 
> current region sizes very similar to the algorithm in HBASE-12590, but a more 
> generic way. This also would eliminate the hand tuning of data skew ratio.
> We also can think about the guidepost approach that Phoenix has in the stats 
> table which is used for exactly this purpose. Right now, the region can be 
> split into powers of two assuming uniform distribution within the region. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13395) Remove HTableInterface

[Jan Hentschel (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-13395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678029#comment-15678029
 ] 

Jan Hentschel commented on HBASE-13395:
---

As far as I understand the documentation a major version allows to remove 
deprecated APIs. The section mentioned by Nick also contains a short example 
which matches our case: A user using a newly deprecated API does not need to 
modify application code with HBase API calls until the next major version.

> Remove HTableInterface
> --
>
> Key: HBASE-13395
> URL: https://issues.apache.org/jira/browse/HBASE-13395
> Project: HBase
>  Issue Type: Sub-task
>  Components: API
>Affects Versions: 2.0.0
>Reporter: Mikhail Antonov
>Assignee: Jan Hentschel
> Fix For: 2.0.0
>
> Attachments: HBASE-13395.master.001.patch
>
>
> This class is marked as deprecated, probably can remove it, and if any 
> methods from this specific class are in active use, need to decide what to do 
> on callers' side. Should be able to replace with just Table interface usage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work started] (HBASE-13395) Remove HTableInterface

[Jan Hentschel (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HBASE-13395 started by Jan Hentschel.
-
> Remove HTableInterface
> --
>
> Key: HBASE-13395
> URL: https://issues.apache.org/jira/browse/HBASE-13395
> Project: HBase
>  Issue Type: Sub-task
>  Components: API
>Affects Versions: 2.0.0
>Reporter: Mikhail Antonov
>Assignee: Jan Hentschel
> Fix For: 2.0.0
>
> Attachments: HBASE-13395.master.001.patch
>
>
> This class is marked as deprecated, probably can remove it, and if any 
> methods from this specific class are in active use, need to decide what to do 
> on callers' side. Should be able to replace with just Table interface usage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13395) Remove HTableInterface

[Jan Hentschel (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Hentschel updated HBASE-13395:
--
Status: Patch Available  (was: In Progress)

> Remove HTableInterface
> --
>
> Key: HBASE-13395
> URL: https://issues.apache.org/jira/browse/HBASE-13395
> Project: HBase
>  Issue Type: Sub-task
>  Components: API
>Affects Versions: 2.0.0
>Reporter: Mikhail Antonov
>Assignee: Jan Hentschel
> Fix For: 2.0.0
>
> Attachments: HBASE-13395.master.001.patch
>
>
> This class is marked as deprecated, probably can remove it, and if any 
> methods from this specific class are in active use, need to decide what to do 
> on callers' side. Should be able to replace with just Table interface usage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HBASE-13395) Remove HTableInterface

[Jan Hentschel (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Hentschel reassigned HBASE-13395:
-

Assignee: Jan Hentschel

> Remove HTableInterface
> --
>
> Key: HBASE-13395
> URL: https://issues.apache.org/jira/browse/HBASE-13395
> Project: HBase
>  Issue Type: Sub-task
>  Components: API
>Affects Versions: 2.0.0
>Reporter: Mikhail Antonov
>Assignee: Jan Hentschel
> Fix For: 2.0.0
>
> Attachments: HBASE-13395.master.001.patch
>
>
> This class is marked as deprecated, probably can remove it, and if any 
> methods from this specific class are in active use, need to decide what to do 
> on callers' side. Should be able to replace with just Table interface usage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13395) Remove HTableInterface

[Jan Hentschel (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Hentschel updated HBASE-13395:
--
Attachment: HBASE-13395.master.001.patch

> Remove HTableInterface
> --
>
> Key: HBASE-13395
> URL: https://issues.apache.org/jira/browse/HBASE-13395
> Project: HBase
>  Issue Type: Sub-task
>  Components: API
>Affects Versions: 2.0.0
>Reporter: Mikhail Antonov
> Fix For: 2.0.0
>
> Attachments: HBASE-13395.master.001.patch
>
>
> This class is marked as deprecated, probably can remove it, and if any 
> methods from this specific class are in active use, need to decide what to do 
> on callers' side. Should be able to replace with just Table interface usage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17123) Add postBulkLoadHFile variant that notifies the final paths for the hfiles

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-17123:
---
Attachment: 17123.v4.txt

> Add postBulkLoadHFile variant that notifies the final paths for the hfiles
> --
>
> Key: HBASE-17123
> URL: https://issues.apache.org/jira/browse/HBASE-17123
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 17123.v1.txt, 17123.v3.txt, 17123.v4.txt
>
>
> Currently the postBulkLoadHFile() hook passes the same familyPaths parameter 
> which it receives as method parameter.
> See code in SecureBulkLoadManager :
> {code}
>loaded = region.getCoprocessorHost().postBulkLoadHFile(familyPaths, 
> loaded);
> {code}
> Meaning, the paths are not final, moved paths of the loaded hfiles.
> This issue is to add a variant which notifies the final paths of the loaded 
> hfiles.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-2631) Decide between "InMB" and "MB" as suffix for field names in ClusterStatus objects

[Jan Hentschel (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-2631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677966#comment-15677966
 ] 

Jan Hentschel commented on HBASE-2631:
--

The ClusterStatus objects seem to don't exist anymore. This ticket can probably 
be closed.

> Decide between "InMB" and "MB" as suffix for field names in ClusterStatus 
> objects
> -
>
> Key: HBASE-2631
> URL: https://issues.apache.org/jira/browse/HBASE-2631
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jeff Hammerbacher
>  Labels: beginner
>
> We vacillate between "InMB" (e.g. HServerLoad.getMemStoreSizeInMB(), 
> HServerLoad.getStorefileIndexSizeInMB()) and "MB" (e.g. 
> HServerLoad.getUsedHeapMB()) in the various ClusterStatus objects 
> (HServerInfo, HServerLoad, HServerLoad.RegionLoad). We should probably pick 
> one and stick to it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-11402) Scanner performs redundand datanode requests

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-11402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-11402.
---
Resolution: Duplicate

Resolving as duplicated by HBASE-10676

Do not take my closing of this issue as a discounting of the work done in here 
[~shmuma]; rather, your work has been carried over into the duplicate and later 
in HBASE-17072. This issue is still a problem. You identified the useless seek. 
The threadlocal in turn is super problematic.

> Scanner performs redundand datanode requests
> 
>
> Key: HBASE-11402
> URL: https://issues.apache.org/jira/browse/HBASE-11402
> Project: HBase
>  Issue Type: Bug
>  Components: HFile, Scanners
>Reporter: Max Lapan
>
> Using hbase 0.94.6 I found duplicate datanode requests of this sort:
> {noformat}
> 2014-06-09 14:12:22,039 INFO 
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: 
> /10.103.0.73:50010, dest: /10.103.0.38:57897, bytes: 1056768, op: HDFS_READ, 
> cliID: DFSClient_NONMAPREDUCE_1702752887_26, offset: 35840, srvID: 
> DS-504316153-10.103.0.73-50010-1342437562377, blockid: 
> BP-404551095-10.103.0.38-1376045452213:blk_3541255952831727320_613837, 
> duration: 109928797000
> 2014-06-09 14:12:22,080 INFO 
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: 
> /10.103.0.73:50010, dest: /10.103.0.38:57910, bytes: 1056768, op: HDFS_READ, 
> cliID: DFSClient_NONMAPREDUCE_1702752887_26, offset: 0, srvID: 
> DS-504316153-10.103.0.73-50010-1342437562377, blockid: 
> BP-404551095-10.103.0.38-1376045452213:blk_3541255952831727320_613837, 
> duration: 3825
> {noformat}
> After short investigation, I found the source of such behaviour:
> * StoreScanner in constructor calls StoreFileScanner::seek, which (after 
> several levels of calls) is calling HFileBlock::readBlockDataInternal which 
> reads block and pre-reads header of the next block.
> * This pre-readed header is stored in ThreadLocal variable 
> and stream is left in a position right behind the header of next block.
> * After constructor finished, scanner code does scanning, and, after 
> pre-readed block data finished, it calls HFileReaderV2::readNextDataBlock, 
> which again calls HFileBlock::readBlockDataInternal, but this call occured 
> from different thread and there is nothing usefull in ThreadLocal variable
> * Due to this, stream is asked to seek backwards, and this cause duplicate DN 
> request.
> As far as I understood from trunk code, the problem hasn't fixed yet.
> Log of calls with process above:
> {noformat}
> 2014-06-18 14:55:36,616 INFO 
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex: loadDataBlockWithScanInfo: 
> entered
> 2014-06-18 14:55:36,616 INFO org.apache.hadoop.hbase.io.hfile.HFileReaderV2: 
> seekTo: readBlock, ofs = 0, size = -1
> 2014-06-18 14:55:36,617 INFO org.apache.hadoop.hbase.io.hfile.HFileReaderV2: 
> Before block read: path = 
> hdfs://tsthdp1.p:9000/hbase/webpagesII/ba16051997b1272f00bed5f65094dc63/p/c866b7b0eded4b
> 2014-06-18 14:55:36,617 INFO org.apache.hadoop.hbase.io.hfile.HFile: 
> readBlockDataInternal. Ofs = 0, is.pos = 137257042, ondDiskSizeWithHeader = -1
> 2014-06-18 14:55:36,617 INFO org.apache.hadoop.hbase.io.hfile.HFile: 
> readBlockDataInternal: prefetchHeader.ofs = -1, thread = 48
> 2014-06-18 14:55:36,617 INFO org.apache.hadoop.hbase.io.hfile.HFile: 
> FSReaderV2: readAtOffset: size = 24, offset = 0, peekNext = false
> 2014-06-18 14:55:36,617 INFO org.apache.hadoop.hdfs.DFSClient: seek: 
> targetPos = 0, pos = 137257042, blockEnd = 137257229
> 2014-06-18 14:55:36,617 INFO org.apache.hadoop.hdfs.DFSClient: seek: not 
> done, blockEnd = -1
> 2014-06-18 14:55:36,617 INFO org.apache.hadoop.hdfs.DFSClient: 
> readWithStrategy: before seek, pos = 0, blockEnd = -1, currentNode = 
> 10.103.0.73:50010
> 2014-06-18 14:55:36,618 INFO org.apache.hadoop.hdfs.DFSClient: getBlockAt: 
> blockEnd updated to 137257229
> 2014-06-18 14:55:36,618 INFO org.apache.hadoop.hdfs.DFSClient: blockSeekTo: 
> loop, target = 0
> 2014-06-18 14:55:36,618 INFO org.apache.hadoop.hdfs.DFSClient: 
> getBlockReader: dn = tsthdp2.p, file = 
> /hbase/webpagesII/ba16051997b1272f00bed5f65094dc63/p/c866b7b0eded4b42bc40aa9e18ac8a4b,
>  bl
> 2014-06-18 14:55:36,627 INFO org.apache.hadoop.hdfs.DFSClient: readBuffer: 
> ofs = 0, len = 24
> 2014-06-18 14:55:36,627 INFO org.apache.hadoop.hdfs.DFSClient: readBuffer: 
> try to read
> 2014-06-18 14:55:36,641 INFO org.apache.hadoop.hdfs.DFSClient: readBuffer: 
> done, len = 24
> 2014-06-18 14:55:36,641 INFO org.apache.hadoop.hbase.io.hfile.HFile: 
> FSReaderV2: readAtOffset: size = 35899, offset = 24, peekNext = true
> 2014-06-18 14:55:36,641 INFO org.apache.hadoop.hdfs.DFSClient: seek: 
> targetPos = 24, pos = 24, blockEnd = 137257229
> 2014-06-18 14:55:36,641 INFO org.apache.hadoop.hdfs.DFSClient: seek: check 

[jira] [Commented] (HBASE-16708) Expose endpoint Coprocessor name in "responseTooSlow" log messages

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677943#comment-15677943
 ] 

Hudson commented on HBASE-16708:


SUCCESS: Integrated in Jenkins build HBase-1.4 #539 (See 
[https://builds.apache.org/job/HBase-1.4/539/])
HBASE-16708 Expose endpoint Coprocessor name in responseTooSlow log (jerryjch: 
rev 26e31644818e8568910f4d74a0a3a2926afc8680)
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java


> Expose endpoint Coprocessor name in "responseTooSlow" log messages
> --
>
> Key: HBASE-16708
> URL: https://issues.apache.org/jira/browse/HBASE-16708
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 1.0.0
>Reporter: Nick Dimiduk
>Assignee: Yi Liang
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16708-Branch1.patch, HBASE-16708-V1.patch, 
> HBASE-16708-V2.patch, HBASE-16708-V3.patch, HBASE-ShortName.patch
>
>
> Operational diagnostics of a Phoenix install would be easier if we included 
> which endpoint coprocessor was being called in this responseTooSlow WARN 
> message.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-17121) Undo the building of xref as part of site build

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-17121.
---
   Resolution: Fixed
Fix Version/s: 2.0.0

Resolving as done.

> Undo the building of xref as part of site build
> ---
>
> Key: HBASE-17121
> URL: https://issues.apache.org/jira/browse/HBASE-17121
> Project: HBase
>  Issue Type: Task
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0
>
>
> Remove xref generation as part of site build. It was useful once before 
> grepcode and easy perusal of src via git views: 
> https://git-wip-us.apache.org/repos/asf?p=hbase.git;a=summary
> Replace the xref link with a pointer to apache git.
> DISCUSS thread on dev list: http://osdir.com/ml/general/2016-11/msg22051.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17121) Undo the building of xref as part of site build

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677851#comment-15677851
 ] 

stack commented on HBASE-17121:
---

This task involves a few commits.

Against the site checkout, I committed an addition to our .htaccess to redirect 
xref references to git tree view instead. I also purged the xref dir from our 
site:

kalashnikov:site.git stack$ git log
commit 7b60414719b142ea072c3d980b0abcd50cf1eda9
Author: Michael Stack 
Date:   Fri Nov 18 10:55:36 2016 -0800

HBASE-17121 Undo the building of xref as part of site build; remove xref 
dirs

commit d6a131e03310fe75b0f4ae0b96190519aa5c467e
Author: Michael Stack 
Date:   Fri Nov 18 10:51:34 2016 -0800

HBASE-17121 Undo the building of xref as part of site build; Add redirect 
for xref/* to git tree view up on apache instead

I think did a commit that removed the xref from doc and from our menus against 
our main repo:

commit bab93359fed414d5f671c673d654517e514a2ddf
Author: Michael Stack 
Date:   Fri Nov 18 13:17:45 2016 -0800

HBASE-17121 Undo the building of xref as part of site build; Remove links 
to xref

Checked site build.



> Undo the building of xref as part of site build
> ---
>
> Key: HBASE-17121
> URL: https://issues.apache.org/jira/browse/HBASE-17121
> Project: HBase
>  Issue Type: Task
>Reporter: stack
>Assignee: stack
>
> Remove xref generation as part of site build. It was useful once before 
> grepcode and easy perusal of src via git views: 
> https://git-wip-us.apache.org/repos/asf?p=hbase.git;a=summary
> Replace the xref link with a pointer to apache git.
> DISCUSS thread on dev list: http://osdir.com/ml/general/2016-11/msg22051.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17095) The ClientSimpleScanner keeps retrying if the hfile is corrupt or cannot found

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677846#comment-15677846
 ] 

stack commented on HBASE-17095:
---

+1


Nice test [~jingcheng...@intel.com]

> The ClientSimpleScanner keeps retrying if the hfile is corrupt or cannot found
> --
>
> Key: HBASE-17095
> URL: https://issues.apache.org/jira/browse/HBASE-17095
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver, scan
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Attachments: HBASE-17095-V2.patch, HBASE-17095-branch-1.patch, 
> HBASE-17095.patch, TestScannerWithCorruptHFile.java
>
>
> In {{RsRPCServices.scan}}, most of IOE are thrown as a 
> {{ScannerResetException}}, even when the hfile is corrupt or it cannot be 
> found. The {{ClientScannr.loadCache}} will keep retrying when the exception 
> is {{ScannerResetException}}. We could throw CorruptHFileException and 
> FileNotFoundException directly from server and don't retry the scan in the 
> client.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (HBASE-16555) JUnit dependency not scoped as test

[Jan Hentschel (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Hentschel updated HBASE-16555:
--
Comment: was deleted

(was: Thanks for the hint, [~busbey]. Do we want to mark this ticket as 
duplicated and close it in favor for HBASE-12909? I will start a discussion on 
the dev list later today.)

> JUnit dependency not scoped as test
> ---
>
> Key: HBASE-16555
> URL: https://issues.apache.org/jira/browse/HBASE-16555
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Affects Versions: 1.2.2
>Reporter: Robert G Duncan
>Assignee: Jan Hentschel
> Attachments: HBASE-16555.master.001.patch
>
>
> *Issue*
> JUnit is included as a compile scope dependency. This increases the size of 
> dependent project shaded/Uber JARs unnecessarily.
> *Actual Entry*
> {code:xml}
>   
> junit
> junit
> ${junit.version}
>   
> {code}
> *Expected Entry*
> {code:xml}
> 
> junit
> junit
> ${junit.version}
> test
> 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16555) JUnit dependency not scoped as test

[Jan Hentschel (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677842#comment-15677842
 ] 

Jan Hentschel commented on HBASE-16555:
---

Thanks for the hint, [~busbey]. Do we want to mark this ticket as duplicated 
and close it in favor for HBASE-12909? I will start a discussion on the dev 
list later today.

> JUnit dependency not scoped as test
> ---
>
> Key: HBASE-16555
> URL: https://issues.apache.org/jira/browse/HBASE-16555
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Affects Versions: 1.2.2
>Reporter: Robert G Duncan
>Assignee: Jan Hentschel
> Attachments: HBASE-16555.master.001.patch
>
>
> *Issue*
> JUnit is included as a compile scope dependency. This increases the size of 
> dependent project shaded/Uber JARs unnecessarily.
> *Actual Entry*
> {code:xml}
>   
> junit
> junit
> ${junit.version}
>   
> {code}
> *Expected Entry*
> {code:xml}
> 
> junit
> junit
> ${junit.version}
> test
> 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16555) JUnit dependency not scoped as test

[Jan Hentschel (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677845#comment-15677845
 ] 

Jan Hentschel commented on HBASE-16555:
---

Thanks for the hint, [~busbey]. Do we want to mark this ticket as duplicated 
and close it in favor for HBASE-12909? I will start a discussion on the dev 
list later today.

> JUnit dependency not scoped as test
> ---
>
> Key: HBASE-16555
> URL: https://issues.apache.org/jira/browse/HBASE-16555
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Affects Versions: 1.2.2
>Reporter: Robert G Duncan
>Assignee: Jan Hentschel
> Attachments: HBASE-16555.master.001.patch
>
>
> *Issue*
> JUnit is included as a compile scope dependency. This increases the size of 
> dependent project shaded/Uber JARs unnecessarily.
> *Actual Entry*
> {code:xml}
>   
> junit
> junit
> ${junit.version}
>   
> {code}
> *Expected Entry*
> {code:xml}
> 
> junit
> junit
> ${junit.version}
> test
> 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16989) RowProcess#postBatchMutate doesn’t be executed before the mvcc transaction completion

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677833#comment-15677833
 ] 

stack commented on HBASE-16989:
---

+1

> RowProcess#postBatchMutate doesn’t be executed before the mvcc transaction 
> completion
> -
>
> Key: HBASE-16989
> URL: https://issues.apache.org/jira/browse/HBASE-16989
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
> Fix For: 2.0.0
>
> Attachments: HBASE-16989.v0.patch, HBASE-16989.v1.patch, 
> HBASE-16989.v2.patch, HBASE-16989.v3.patch, HBASE-16989.v4.patch
>
>
> After the [HBASE-15158|https://issues.apache.org/jira/browse/HBASE-15158], 
> RowProcess#postBatchMutate will be executed “after” the mvcc transaction 
> completion.
> {code:title=HRegion#processRowsWithLocks}
>   // STEP 8. Complete mvcc.
>   mvcc.completeAndWait(writeEntry);
>   writeEntry = null;
> 
>   // STEP 9. Release region lock
>   if (locked) {
> this.updatesLock.readLock().unlock();
> locked = false;
>   }
> 
>   // STEP 10. Release row lock(s)
>   releaseRowLocks(acquiredRowLocks);
> 
>   // STEP 11. call postBatchMutate hook
>   processor.postBatchMutate(this);
> {code}
> {code:title=RowProcess#postBatchMutate}
>   /**
>* The hook to be executed after the process() and applying the Mutations 
> to region. The
>* difference of this one with {@link #postProcess(HRegion, WALEdit, 
> boolean)} is this hook will
>* be executed before the mvcc transaction completion.
>*/
>   void postBatchMutate(HRegion region) throws IOException;
> {code}
> Do we ought to revamp the comment of RowProcess#postBatchMutate or change the 
> call order?
> I prefer the former, because the HRegion#doMiniBatchMutate() also call 
> postBatchMutate() after the mvcc transaction completion.
> Any comment? Thanks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17122) Change in behavior when creating a scanner for a disabled table

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677809#comment-15677809
 ] 

stack commented on HBASE-17122:
---

Do you know what commit broke our behavior [~samarth.j...@gmail.com] You have a 
suggestion for how to fix?

> Change in behavior when creating a scanner for a disabled table
> ---
>
> Key: HBASE-17122
> URL: https://issues.apache.org/jira/browse/HBASE-17122
> Project: HBase
>  Issue Type: Bug
>Reporter: Samarth Jain
>
> {code}
> @Test
> public void testQueryingDisabledTable() throws Exception {
> try (Connection conn = DriverManager.getConnection(getUrl())) {
> String tableName = generateUniqueName();
> conn.createStatement().execute(
> "CREATE TABLE " + tableName
> + " (k1 VARCHAR NOT NULL, k2 VARCHAR, CONSTRAINT PK 
> PRIMARY KEY(K1,K2)) ");
> try (HBaseAdmin admin = 
> conn.unwrap(PhoenixConnection.class).getQueryServices().getAdmin()) {
> admin.disableTable(Bytes.toBytes(tableName));
> }
> String query = "SELECT * FROM " + tableName + " WHERE 1=1";
> try (Connection conn2 = DriverManager.getConnection(getUrl())) {
> try (ResultSet rs = 
> conn2.createStatement().executeQuery(query)) {
> assertFalse(rs.next());
> }
> }
> }
> }
> {code}
> This is a phoenix specific test case. I will try an come up with something 
> using the HBase API. But the gist is that with HBase 0.98.21 and beyond, we 
> are seeing that creating a scanner is throwing a NotServingRegionException. 
> Stacktrace for NotServingRegionException
> {code}
> org.apache.phoenix.exception.PhoenixIOException: 
> org.apache.phoenix.exception.PhoenixIOException: callTimeout=120, 
> callDuration=9000104: row '' on table 'T01' at 
> region=T01,,1479429739864.643dde31cc19b549192576eea7791a6f., 
> hostname=localhost,60022,1479429692090, seqNum=1
>   at 
> org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:113)
>   at 
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:752)
>   at 
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:696)
>   at 
> org.apache.phoenix.iterate.ConcatResultIterator.getIterators(ConcatResultIterator.java:50)
>   at 
> org.apache.phoenix.iterate.ConcatResultIterator.currentIterator(ConcatResultIterator.java:97)
>   at 
> org.apache.phoenix.iterate.ConcatResultIterator.next(ConcatResultIterator.java:117)
>   at 
> org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:778)
>   at 
> org.apache.phoenix.end2end.PhoenixRuntimeIT.testQueryingDisabledTable(PhoenixRuntimeIT.java:167)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at 
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
>   at 
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
>   at 
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
>   at 
> 

[jira] [Commented] (HBASE-17127) Locate region should fail fast if underlying Connection already closed

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677805#comment-15677805
 ] 

stack commented on HBASE-17127:
---

+1

> Locate region should fail fast if underlying Connection already closed
> --
>
> Key: HBASE-17127
> URL: https://issues.apache.org/jira/browse/HBASE-17127
> Project: HBase
>  Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17127.patch
>
>
> Currently if try to locate region when underlying connection is closed, we 
> will retry and wait at least 10s for each round until exhausted (refer to the 
> catch clause of {{RpcRetryingCallerImpl#callWithRetries}} and 
> {{RegionServerCallable#sleep}} for more details), which is unnecessary and 
> time-costing.
> The issue is caused by user incorrectly manipulating connection, which shows 
> the disadvantage of force user to handle connection life cycle and proves the 
> necessity to support auto-managed connection as we did before, as indicated 
> in HBASE-17009
> In this JIRA we will make it fail fast in the above case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16002) Many DataType constructors are not public

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677792#comment-15677792
 ] 

stack commented on HBASE-16002:
---

Patch LGTM. Needs blessing by [~ndimiduk] before commit. Thanks [~Jan Hentschel]


> Many DataType constructors are not public
> -
>
> Key: HBASE-16002
> URL: https://issues.apache.org/jira/browse/HBASE-16002
> Project: HBase
>  Issue Type: Bug
>Reporter: Nick Dimiduk
>Assignee: Jan Hentschel
> Attachments: HBASE-16002.master.001.patch
>
>
> Using the {{Struct}} class to build a rowkey, I noticed than many of the 
> provided {{DataType}} implementations' constructors are protected. They 
> should be public in most (all?) cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16002) Many DataType constructors are not public

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677744#comment-15677744
 ] 

Hadoop QA commented on HBASE-16002:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 15s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
9s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
31s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 14s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
25m 10s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 41s 
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
6s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 37s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12839612/HBASE-16002.master.001.patch
 |
| JIRA Issue | HBASE-16002 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux a6ec3a556696 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / a8ee83c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4536/testReport/ |
| modules | C: hbase-common U: hbase-common |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4536/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> Many DataType constructors are not public
> -
>
> Key: HBASE-16002
> URL: https://issues.apache.org/jira/browse/HBASE-16002
> Project: HBase
>  Issue Type: Bug
>Reporter: Nick Dimiduk
>Assignee: Jan Hentschel
> Attachments: 

[jira] [Commented] (HBASE-17129) Remove public from methods in DataType interface

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677670#comment-15677670
 ] 

Hadoop QA commented on HBASE-17129:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
8s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
32s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 14s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
25m 12s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 42s 
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
7s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 38s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12839607/HBASE-17129.master.001.patch
 |
| JIRA Issue | HBASE-17129 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 660fff503579 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / a8ee83c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4535/testReport/ |
| modules | C: hbase-common U: hbase-common |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4535/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> Remove public from methods in DataType interface
> 
>
> Key: HBASE-17129
> URL: https://issues.apache.org/jira/browse/HBASE-17129
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jan Hentschel
>Assignee: Jan Hentschel
>  

[jira] [Assigned] (HBASE-16002) Many DataType constructors are not public

[Jan Hentschel (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Hentschel reassigned HBASE-16002:
-

Assignee: Jan Hentschel

> Many DataType constructors are not public
> -
>
> Key: HBASE-16002
> URL: https://issues.apache.org/jira/browse/HBASE-16002
> Project: HBase
>  Issue Type: Bug
>Reporter: Nick Dimiduk
>Assignee: Jan Hentschel
> Attachments: HBASE-16002.master.001.patch
>
>
> Using the {{Struct}} class to build a rowkey, I noticed than many of the 
> provided {{DataType}} implementations' constructors are protected. They 
> should be public in most (all?) cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work started] (HBASE-16002) Many DataType constructors are not public

[Jan Hentschel (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HBASE-16002 started by Jan Hentschel.
-
> Many DataType constructors are not public
> -
>
> Key: HBASE-16002
> URL: https://issues.apache.org/jira/browse/HBASE-16002
> Project: HBase
>  Issue Type: Bug
>Reporter: Nick Dimiduk
>Assignee: Jan Hentschel
> Attachments: HBASE-16002.master.001.patch
>
>
> Using the {{Struct}} class to build a rowkey, I noticed than many of the 
> provided {{DataType}} implementations' constructors are protected. They 
> should be public in most (all?) cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16002) Many DataType constructors are not public

[Jan Hentschel (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Hentschel updated HBASE-16002:
--
Status: Patch Available  (was: In Progress)

> Many DataType constructors are not public
> -
>
> Key: HBASE-16002
> URL: https://issues.apache.org/jira/browse/HBASE-16002
> Project: HBase
>  Issue Type: Bug
>Reporter: Nick Dimiduk
>Assignee: Jan Hentschel
> Attachments: HBASE-16002.master.001.patch
>
>
> Using the {{Struct}} class to build a rowkey, I noticed than many of the 
> provided {{DataType}} implementations' constructors are protected. They 
> should be public in most (all?) cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16002) Many DataType constructors are not public

[Jan Hentschel (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Hentschel updated HBASE-16002:
--
Attachment: HBASE-16002.master.001.patch

> Many DataType constructors are not public
> -
>
> Key: HBASE-16002
> URL: https://issues.apache.org/jira/browse/HBASE-16002
> Project: HBase
>  Issue Type: Bug
>Reporter: Nick Dimiduk
> Attachments: HBASE-16002.master.001.patch
>
>
> Using the {{Struct}} class to build a rowkey, I noticed than many of the 
> provided {{DataType}} implementations' constructors are protected. They 
> should be public in most (all?) cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17123) Add postBulkLoadHFile variant that notifies the final paths for the hfiles

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677591#comment-15677591
 ] 

Hadoop QA commented on HBASE-17123:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 1s 
{color} | {color:blue} The patch file was not named according to hbase's naming 
conventions. Please see 
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for 
instructions. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
7s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
56s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 43s 
{color} | {color:red} The patch causes 14 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 17s 
{color} | {color:red} The patch causes 14 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 0s 
{color} | {color:red} The patch causes 14 errors with Hadoop v2.6.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m 37s 
{color} | {color:red} The patch causes 14 errors with Hadoop v2.6.4. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m 18s 
{color} | {color:red} The patch causes 14 errors with Hadoop v2.6.5. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 9m 55s 
{color} | {color:red} The patch causes 14 errors with Hadoop v2.7.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 11m 35s 
{color} | {color:red} The patch causes 14 errors with Hadoop v2.7.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 13m 11s 
{color} | {color:red} The patch causes 14 errors with Hadoop v2.7.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 14m 52s 
{color} | {color:red} The patch causes 14 errors with Hadoop v3.0.0-alpha1. 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 58s 
{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 
total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 92m 6s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
13s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 119m 52s {color} 
| {color:black} {color} |
\\
\\
|| Reason 

[jira] [Work started] (HBASE-17129) Remove public from methods in DataType interface

[Jan Hentschel (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HBASE-17129 started by Jan Hentschel.
-
> Remove public from methods in DataType interface
> 
>
> Key: HBASE-17129
> URL: https://issues.apache.org/jira/browse/HBASE-17129
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jan Hentschel
>Assignee: Jan Hentschel
>Priority: Minor
> Attachments: HBASE-17129.master.001.patch
>
>
> There is no need to define the methods in the interface DataType as public. 
> The visibility of the declared methods can be removed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17129) Remove public from methods in DataType interface

[Jan Hentschel (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Hentschel updated HBASE-17129:
--
Status: Patch Available  (was: In Progress)

> Remove public from methods in DataType interface
> 
>
> Key: HBASE-17129
> URL: https://issues.apache.org/jira/browse/HBASE-17129
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jan Hentschel
>Assignee: Jan Hentschel
>Priority: Minor
> Attachments: HBASE-17129.master.001.patch
>
>
> There is no need to define the methods in the interface DataType as public. 
> The visibility of the declared methods can be removed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17129) Remove public from methods in DataType interface

[Jan Hentschel (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Hentschel updated HBASE-17129:
--
Attachment: HBASE-17129.master.001.patch

> Remove public from methods in DataType interface
> 
>
> Key: HBASE-17129
> URL: https://issues.apache.org/jira/browse/HBASE-17129
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jan Hentschel
>Assignee: Jan Hentschel
>Priority: Minor
> Attachments: HBASE-17129.master.001.patch
>
>
> There is no need to define the methods in the interface DataType as public. 
> The visibility of the declared methods can be removed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-17129) Remove public from methods in DataType interface

[Jan Hentschel (JIRA)]

Jan Hentschel created HBASE-17129:
-

 Summary: Remove public from methods in DataType interface
 Key: HBASE-17129
 URL: https://issues.apache.org/jira/browse/HBASE-17129
 Project: HBase
  Issue Type: Improvement
Reporter: Jan Hentschel
Assignee: Jan Hentschel
Priority: Minor


There is no need to define the methods in the interface DataType as public. The 
visibility of the declared methods can be removed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16708) Expose endpoint Coprocessor name in "responseTooSlow" log messages

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-16708:
-
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: (was: 1.0.0)
   1.4.0
   2.0.0
   Status: Resolved  (was: Patch Available)

Pushed to master and branch-1. 
Thanks for the patch, [~easyliangjob].   Thanks for the review [~ndimiduk]

> Expose endpoint Coprocessor name in "responseTooSlow" log messages
> --
>
> Key: HBASE-16708
> URL: https://issues.apache.org/jira/browse/HBASE-16708
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 1.0.0
>Reporter: Nick Dimiduk
>Assignee: Yi Liang
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16708-Branch1.patch, HBASE-16708-V1.patch, 
> HBASE-16708-V2.patch, HBASE-16708-V3.patch, HBASE-ShortName.patch
>
>
> Operational diagnostics of a Phoenix install would be easier if we included 
> which endpoint coprocessor was being called in this responseTooSlow WARN 
> message.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16935) deleteColumn/modifyTable don't delete all family's StoreFile from file system

[Matteo Bertozzi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Bertozzi updated HBASE-16935:

Summary: deleteColumn/modifyTable don't delete all family's StoreFile from 
file system  (was: Java API method Admin.deleteColumn(table, columnFamily) 
doesn't delete family's StoreFile from file system.)

> deleteColumn/modifyTable don't delete all family's StoreFile from file system
> -
>
> Key: HBASE-16935
> URL: https://issues.apache.org/jira/browse/HBASE-16935
> Project: HBase
>  Issue Type: New Feature
>  Components: Admin
>Affects Versions: 1.2.3
>Reporter: Mikhail Zvagelsky
> Attachments: Selection_008.png
>
>
> The method deleteColumn(TableName tableName, byte[] columnName) of the class 
> org.apache.hadoop.hbase.client.Admin shoud delete specified column family 
> from specified table. (Despite of its name the method removes the family, not 
> a column - view the [issue| 
> https://issues.apache.org/jira/browse/HBASE-1989].)
> This method changes the table's schema, but it doesn't delete column family's 
> Store File from a file system. To be precise - I run this code:
> {code:|borderStyle=solid}
> import java.io.IOException;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.hbase.HBaseConfiguration;
> import org.apache.hadoop.hbase.HColumnDescriptor;
> import org.apache.hadoop.hbase.HTableDescriptor;
> import org.apache.hadoop.hbase.TableName;
> import org.apache.hadoop.hbase.client.*;
> import org.apache.hadoop.hbase.util.Bytes;
> public class ToHBaseIssueTracker {
> public static void main(String[] args) throws IOException {
> TableName tableName = TableName.valueOf("test_table");
> HTableDescriptor desc = new HTableDescriptor(tableName);
> desc.addFamily(new HColumnDescriptor("cf1"));
> desc.addFamily(new HColumnDescriptor("cf2"));
> Configuration conf = HBaseConfiguration.create();
> Connection connection = ConnectionFactory.createConnection(conf);
> Admin admin = connection.getAdmin();
> admin.createTable(desc);
> HTable table = new HTable(conf, "test_table");
> for (int i = 0; i < 4; i++) {
> Put put = new Put(Bytes.toBytes(i)); // Use i as row key.
> put.addColumn(Bytes.toBytes("cf1"), Bytes.toBytes("a"), 
> Bytes.toBytes("value"));
> put.addColumn(Bytes.toBytes("cf2"), Bytes.toBytes("a"), 
> Bytes.toBytes("value"));
> table.put(put);
> }
> admin.deleteColumn(tableName, Bytes.toBytes("cf2"));
> admin.majorCompact(tableName);
> admin.close();
> }
> }
> {code}
> Then I see that the store file for the "cf2" family persists in file system.
> I observe this effect in standalone hbase installation and in 
> pseudo-distributed mode.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16935) Java API method Admin.deleteColumn(table, columnFamily) doesn't delete family's StoreFile from file system.

[Matteo Bertozzi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677463#comment-15677463
 ] 

Matteo Bertozzi commented on HBASE-16935:
-

from the code we actually remove the family dirs in both cases (table enabled 
or disabled). but since we reopen the regions after deleting the family folders 
we end up with the flushed on-close file in the deleted family.
so, all the files in the family except the flush on-close one will be removed.

basically the simplified order of operation we have (since 0.94 or even before) 
is:
 - change the htd
 - drop the family folder
 - re-open regions (close region will trigger a flush of the family)

we should probably flip the order, so we reopen with the new htd and then we 
remove the dirs.

> Java API method Admin.deleteColumn(table, columnFamily) doesn't delete 
> family's StoreFile from file system.
> ---
>
> Key: HBASE-16935
> URL: https://issues.apache.org/jira/browse/HBASE-16935
> Project: HBase
>  Issue Type: New Feature
>  Components: Admin
>Affects Versions: 1.2.3
>Reporter: Mikhail Zvagelsky
> Attachments: Selection_008.png
>
>
> The method deleteColumn(TableName tableName, byte[] columnName) of the class 
> org.apache.hadoop.hbase.client.Admin shoud delete specified column family 
> from specified table. (Despite of its name the method removes the family, not 
> a column - view the [issue| 
> https://issues.apache.org/jira/browse/HBASE-1989].)
> This method changes the table's schema, but it doesn't delete column family's 
> Store File from a file system. To be precise - I run this code:
> {code:|borderStyle=solid}
> import java.io.IOException;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.hbase.HBaseConfiguration;
> import org.apache.hadoop.hbase.HColumnDescriptor;
> import org.apache.hadoop.hbase.HTableDescriptor;
> import org.apache.hadoop.hbase.TableName;
> import org.apache.hadoop.hbase.client.*;
> import org.apache.hadoop.hbase.util.Bytes;
> public class ToHBaseIssueTracker {
> public static void main(String[] args) throws IOException {
> TableName tableName = TableName.valueOf("test_table");
> HTableDescriptor desc = new HTableDescriptor(tableName);
> desc.addFamily(new HColumnDescriptor("cf1"));
> desc.addFamily(new HColumnDescriptor("cf2"));
> Configuration conf = HBaseConfiguration.create();
> Connection connection = ConnectionFactory.createConnection(conf);
> Admin admin = connection.getAdmin();
> admin.createTable(desc);
> HTable table = new HTable(conf, "test_table");
> for (int i = 0; i < 4; i++) {
> Put put = new Put(Bytes.toBytes(i)); // Use i as row key.
> put.addColumn(Bytes.toBytes("cf1"), Bytes.toBytes("a"), 
> Bytes.toBytes("value"));
> put.addColumn(Bytes.toBytes("cf2"), Bytes.toBytes("a"), 
> Bytes.toBytes("value"));
> table.put(put);
> }
> admin.deleteColumn(tableName, Bytes.toBytes("cf2"));
> admin.majorCompact(tableName);
> admin.close();
> }
> }
> {code}
> Then I see that the store file for the "cf2" family persists in file system.
> I observe this effect in standalone hbase installation and in 
> pseudo-distributed mode.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16708) Expose endpoint Coprocessor name in "responseTooSlow" log messages

[Yi Liang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677438#comment-15677438
 ] 

Yi Liang commented on HBASE-16708:
--

I attache two patches,  V3.patch  is for master branch and Branch1.patch is for 
branch-1, could any one help me to commit? Thanks

> Expose endpoint Coprocessor name in "responseTooSlow" log messages
> --
>
> Key: HBASE-16708
> URL: https://issues.apache.org/jira/browse/HBASE-16708
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 1.0.0
>Reporter: Nick Dimiduk
>Assignee: Yi Liang
> Fix For: 1.0.0
>
> Attachments: HBASE-16708-Branch1.patch, HBASE-16708-V1.patch, 
> HBASE-16708-V2.patch, HBASE-16708-V3.patch, HBASE-ShortName.patch
>
>
> Operational diagnostics of a Phoenix install would be easier if we included 
> which endpoint coprocessor was being called in this responseTooSlow WARN 
> message.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16708) Expose endpoint Coprocessor name in "responseTooSlow" log messages

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677431#comment-15677431
 ] 

Hadoop QA commented on HBASE-16708:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s {color} 
| {color:red} HBASE-16708 does not apply to master. Rebase required? Wrong 
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames 
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12839595/HBASE-16708-Branch1.patch
 |
| JIRA Issue | HBASE-16708 |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4534/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> Expose endpoint Coprocessor name in "responseTooSlow" log messages
> --
>
> Key: HBASE-16708
> URL: https://issues.apache.org/jira/browse/HBASE-16708
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 1.0.0
>Reporter: Nick Dimiduk
>Assignee: Yi Liang
> Fix For: 1.0.0
>
> Attachments: HBASE-16708-Branch1.patch, HBASE-16708-V1.patch, 
> HBASE-16708-V2.patch, HBASE-16708-V3.patch, HBASE-ShortName.patch
>
>
> Operational diagnostics of a Phoenix install would be easier if we included 
> which endpoint coprocessor was being called in this responseTooSlow WARN 
> message.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HBASE-16894) Create more than 1 split per region, generalize HBASE-12590

[Yi Liang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yi Liang reassigned HBASE-16894:


Assignee: Yi Liang

> Create more than 1 split per region, generalize HBASE-12590
> ---
>
> Key: HBASE-16894
> URL: https://issues.apache.org/jira/browse/HBASE-16894
> Project: HBase
>  Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Yi Liang
>  Labels: beginner, beginners
>
> A common request from users is to be able to better control how many map 
> tasks are created per region. Right now, it is always 1 region = 1 input 
> split = 1 map task. Same goes for Spark since it uses the TIF. With region 
> sizes as large as 50 GBs, it is desirable to be able to create more than 1 
> split per region.
> HBASE-12590 adds a config property for MR jobs to be able to handle skew in 
> region sizes. The algorithm is roughly: 
> {code}
> If (region size >= average size*ratio) : cut the region into two MR input 
> splits
> If (average size <= region size < average size*ratio) : one region as one MR 
> input split
> If (sum of several continuous regions size < average size * ratio): combine 
> these regions into one MR input split.
> {code}
> Although we can set data skew ratio to be 0.5 or something to abuse 
> HBASE-12590 into creating more than 1 split task per region, it is not ideal. 
> But there is no way to create more with the patch as it is. For example we 
> cannot create more than 2 tasks per region. 
> If we want to fix this properly, we should extend the approach in 
> HBASE-12590, and make it so that the client can specify the desired num of 
> mappers, or desired split size, and the TIF generates the splits based on the 
> current region sizes very similar to the algorithm in HBASE-12590, but a more 
> generic way. This also would eliminate the hand tuning of data skew ratio.
> We also can think about the guidepost approach that Phoenix has in the stats 
> table which is used for exactly this purpose. Right now, the region can be 
> split into powers of two assuming uniform distribution within the region. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16894) Create more than 1 split per region, generalize HBASE-12590

[Yi Liang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677428#comment-15677428
 ] 

Yi Liang commented on HBASE-16894:
--

   I have not seen any activity on this jira, so I will work on this one. 
[~enis], from your description, if I understand correctly for you intention, 
then I have some proposal for this jira
 (1) To split large regions and maybe merge too small regions:
 (2) It is better for customer to specify desired split size for each split
 (2) Algorithm to extend the algorithm in HBase-12590
   If (region size > split size + threshold) : cut the region into 
(regionsize/splitsize)  MR input splits
   If (split size - threshold <= region size <= split size + threshold) 
: one region as one MR input split
   If (sum of several continuous regions size < split size - 
threshold): combine these continuous regions into one MR input split. 
The threshold may be based on the input split size.
Is there any other suggestion?  Thanks

> Create more than 1 split per region, generalize HBASE-12590
> ---
>
> Key: HBASE-16894
> URL: https://issues.apache.org/jira/browse/HBASE-16894
> Project: HBase
>  Issue Type: Improvement
>Reporter: Enis Soztutar
>  Labels: beginner, beginners
>
> A common request from users is to be able to better control how many map 
> tasks are created per region. Right now, it is always 1 region = 1 input 
> split = 1 map task. Same goes for Spark since it uses the TIF. With region 
> sizes as large as 50 GBs, it is desirable to be able to create more than 1 
> split per region.
> HBASE-12590 adds a config property for MR jobs to be able to handle skew in 
> region sizes. The algorithm is roughly: 
> {code}
> If (region size >= average size*ratio) : cut the region into two MR input 
> splits
> If (average size <= region size < average size*ratio) : one region as one MR 
> input split
> If (sum of several continuous regions size < average size * ratio): combine 
> these regions into one MR input split.
> {code}
> Although we can set data skew ratio to be 0.5 or something to abuse 
> HBASE-12590 into creating more than 1 split task per region, it is not ideal. 
> But there is no way to create more with the patch as it is. For example we 
> cannot create more than 2 tasks per region. 
> If we want to fix this properly, we should extend the approach in 
> HBASE-12590, and make it so that the client can specify the desired num of 
> mappers, or desired split size, and the TIF generates the splits based on the 
> current region sizes very similar to the algorithm in HBASE-12590, but a more 
> generic way. This also would eliminate the hand tuning of data skew ratio.
> We also can think about the guidepost approach that Phoenix has in the stats 
> table which is used for exactly this purpose. Right now, the region can be 
> split into powers of two assuming uniform distribution within the region. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16663) JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677425#comment-15677425
 ] 

Hadoop QA commented on HBASE-16663:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m 44s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
25s {color} | {color:green} branch-1.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s 
{color} | {color:green} branch-1.1 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s 
{color} | {color:green} branch-1.1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
21s {color} | {color:green} branch-1.1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
16s {color} | {color:green} branch-1.1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 48s 
{color} | {color:red} hbase-server in branch-1.1 has 80 extant Findbugs 
warnings. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 25s 
{color} | {color:red} hbase-server in branch-1.1 failed with JDK v1.8.0_111. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s 
{color} | {color:green} branch-1.1 passed with JDK v1.7.0_80 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 22s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 18s 
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_111. 
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 18s {color} 
| {color:red} hbase-server in the patch failed with JDK v1.8.0_111. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 23s 
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_80. 
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 23s {color} 
| {color:red} hbase-server in the patch failed with JDK v1.7.0_80. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 0m 51s 
{color} | {color:red} The patch causes 22 errors with Hadoop v2.4.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 38s 
{color} | {color:red} The patch causes 22 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 26s 
{color} | {color:red} The patch causes 22 errors with Hadoop v2.5.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 15s 
{color} | {color:red} The patch causes 22 errors with Hadoop v2.5.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m 2s 
{color} | {color:red} The patch causes 22 errors with Hadoop v2.5.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m 51s 
{color} | {color:red} The patch causes 22 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 41s 
{color} | {color:red} The patch causes 22 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m 29s 
{color} | {color:red} The patch causes 22 errors with Hadoop v2.6.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 7m 21s 
{color} | {color:red} The patch causes 22 errors with Hadoop v2.7.1. {color} |
| {color:red}-1{color} | {color:red} hbaseprotoc {color} | {color:red} 0m 24s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 22s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 25s 
{color} | 

[jira] [Commented] (HBASE-17127) Locate region should fail fast if underlying Connection already closed

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677422#comment-15677422
 ] 

Hadoop QA commented on HBASE-17127:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
9s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 15s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
53s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 15s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 15s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
27m 46s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 0s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 1s 
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
8s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 15s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12839590/HBASE-17127.patch |
| JIRA Issue | HBASE-17127 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux aa9bc68747a9 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 72438c0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4532/testReport/ |
| modules | C: hbase-client U: hbase-client |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4532/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> Locate region should fail fast if underlying Connection already closed
> --
>
> Key: HBASE-17127
> URL: https://issues.apache.org/jira/browse/HBASE-17127
> Project: HBase
>  Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17127.patch
>
>
> Currently if try to locate region when underlying connection is closed, we 
> will 

[jira] [Commented] (HBASE-17122) Change in behavior when creating a scanner for a disabled table

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677412#comment-15677412
 ] 

Hudson commented on HBASE-17122:


SUCCESS: Integrated in Jenkins build Phoenix-4.8-HBase-1.2 #64 (See 
[https://builds.apache.org/job/Phoenix-4.8-HBase-1.2/64/])
Revert "PHOENIX-3497 Provide a work around for HBASE-17122" (Samarth: rev 
3d4205123f763fdfd875211d551da42deeb78412)
* (edit) phoenix-core/src/main/java/org/apache/phoenix/util/ServerUtil.java
* (edit) 
phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java
* (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/AlterTableIT.java


> Change in behavior when creating a scanner for a disabled table
> ---
>
> Key: HBASE-17122
> URL: https://issues.apache.org/jira/browse/HBASE-17122
> Project: HBase
>  Issue Type: Bug
>Reporter: Samarth Jain
>
> {code}
> @Test
> public void testQueryingDisabledTable() throws Exception {
> try (Connection conn = DriverManager.getConnection(getUrl())) {
> String tableName = generateUniqueName();
> conn.createStatement().execute(
> "CREATE TABLE " + tableName
> + " (k1 VARCHAR NOT NULL, k2 VARCHAR, CONSTRAINT PK 
> PRIMARY KEY(K1,K2)) ");
> try (HBaseAdmin admin = 
> conn.unwrap(PhoenixConnection.class).getQueryServices().getAdmin()) {
> admin.disableTable(Bytes.toBytes(tableName));
> }
> String query = "SELECT * FROM " + tableName + " WHERE 1=1";
> try (Connection conn2 = DriverManager.getConnection(getUrl())) {
> try (ResultSet rs = 
> conn2.createStatement().executeQuery(query)) {
> assertFalse(rs.next());
> }
> }
> }
> }
> {code}
> This is a phoenix specific test case. I will try an come up with something 
> using the HBase API. But the gist is that with HBase 0.98.21 and beyond, we 
> are seeing that creating a scanner is throwing a NotServingRegionException. 
> Stacktrace for NotServingRegionException
> {code}
> org.apache.phoenix.exception.PhoenixIOException: 
> org.apache.phoenix.exception.PhoenixIOException: callTimeout=120, 
> callDuration=9000104: row '' on table 'T01' at 
> region=T01,,1479429739864.643dde31cc19b549192576eea7791a6f., 
> hostname=localhost,60022,1479429692090, seqNum=1
>   at 
> org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:113)
>   at 
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:752)
>   at 
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:696)
>   at 
> org.apache.phoenix.iterate.ConcatResultIterator.getIterators(ConcatResultIterator.java:50)
>   at 
> org.apache.phoenix.iterate.ConcatResultIterator.currentIterator(ConcatResultIterator.java:97)
>   at 
> org.apache.phoenix.iterate.ConcatResultIterator.next(ConcatResultIterator.java:117)
>   at 
> org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:778)
>   at 
> org.apache.phoenix.end2end.PhoenixRuntimeIT.testQueryingDisabledTable(PhoenixRuntimeIT.java:167)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at 

[jira] [Updated] (HBASE-17127) Locate region should fail fast if underlying Connection already closed

[Yu Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yu Li updated HBASE-17127:
--
Description: 
Currently if try to locate region when underlying connection is closed, we will 
retry and wait at least 10s for each round until exhausted (refer to the catch 
clause of {{RpcRetryingCallerImpl#callWithRetries}} and 
{{RegionServerCallable#sleep}} for more details), which is unnecessary and 
time-costing.

The issue is caused by user incorrectly manipulating connection, which shows 
the disadvantage of force user to handle connection life cycle and proves the 
necessity to support auto-managed connection as we did before, as indicated in 
HBASE-17009

In this JIRA we will make it fail fast in the above case.

  was:
Currently if try to locate region when underlying connection is closed, we will 
retry and wait at least 10s for each round until exhausted (refer to the catch 
clause of {{RpcRetryingCallerImpl#callWithRetries}} and 
{{RegionServerCallable#sleep}} for more details), which is unnecessary and 
time-costing.

The issue is caused by incorrect manipulating connection which shows the 
disadvantage of force user to handle connection life cycle and proves the 
necessity to support auto-managed connection as we did before, as indicated in 
HBASE-17009

In this JIRA we will make it fail fast in the above case.


> Locate region should fail fast if underlying Connection already closed
> --
>
> Key: HBASE-17127
> URL: https://issues.apache.org/jira/browse/HBASE-17127
> Project: HBase
>  Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17127.patch
>
>
> Currently if try to locate region when underlying connection is closed, we 
> will retry and wait at least 10s for each round until exhausted (refer to the 
> catch clause of {{RpcRetryingCallerImpl#callWithRetries}} and 
> {{RegionServerCallable#sleep}} for more details), which is unnecessary and 
> time-costing.
> The issue is caused by user incorrectly manipulating connection, which shows 
> the disadvantage of force user to handle connection life cycle and proves the 
> necessity to support auto-managed connection as we did before, as indicated 
> in HBASE-17009
> In this JIRA we will make it fail fast in the above case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16708) Expose endpoint Coprocessor name in "responseTooSlow" log messages

[Yi Liang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yi Liang updated HBASE-16708:
-
Fix Version/s: (was: 2.0.0)
   1.0.0
Affects Version/s: (was: 2.0.0)
   1.0.0
   Status: Patch Available  (was: Open)

> Expose endpoint Coprocessor name in "responseTooSlow" log messages
> --
>
> Key: HBASE-16708
> URL: https://issues.apache.org/jira/browse/HBASE-16708
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 1.0.0
>Reporter: Nick Dimiduk
>Assignee: Yi Liang
> Fix For: 1.0.0
>
> Attachments: HBASE-16708-Branch1.patch, HBASE-16708-V1.patch, 
> HBASE-16708-V2.patch, HBASE-16708-V3.patch, HBASE-ShortName.patch
>
>
> Operational diagnostics of a Phoenix install would be easier if we included 
> which endpoint coprocessor was being called in this responseTooSlow WARN 
> message.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16708) Expose endpoint Coprocessor name in "responseTooSlow" log messages

[Yi Liang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yi Liang updated HBASE-16708:
-
Status: Open  (was: Patch Available)

> Expose endpoint Coprocessor name in "responseTooSlow" log messages
> --
>
> Key: HBASE-16708
> URL: https://issues.apache.org/jira/browse/HBASE-16708
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Nick Dimiduk
>Assignee: Yi Liang
> Fix For: 2.0.0
>
> Attachments: HBASE-16708-Branch1.patch, HBASE-16708-V1.patch, 
> HBASE-16708-V2.patch, HBASE-16708-V3.patch, HBASE-ShortName.patch
>
>
> Operational diagnostics of a Phoenix install would be easier if we included 
> which endpoint coprocessor was being called in this responseTooSlow WARN 
> message.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16708) Expose endpoint Coprocessor name in "responseTooSlow" log messages

[Yi Liang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yi Liang updated HBASE-16708:
-
Attachment: HBASE-16708-Branch1.patch

> Expose endpoint Coprocessor name in "responseTooSlow" log messages
> --
>
> Key: HBASE-16708
> URL: https://issues.apache.org/jira/browse/HBASE-16708
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Nick Dimiduk
>Assignee: Yi Liang
> Fix For: 2.0.0
>
> Attachments: HBASE-16708-Branch1.patch, HBASE-16708-V1.patch, 
> HBASE-16708-V2.patch, HBASE-16708-V3.patch, HBASE-ShortName.patch
>
>
> Operational diagnostics of a Phoenix install would be easier if we included 
> which endpoint coprocessor was being called in this responseTooSlow WARN 
> message.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17127) Locate region should fail fast if underlying Connection already closed

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677385#comment-15677385
 ] 

Ted Yu commented on HBASE-17127:


lgtm , pending QA run.

> Locate region should fail fast if underlying Connection already closed
> --
>
> Key: HBASE-17127
> URL: https://issues.apache.org/jira/browse/HBASE-17127
> Project: HBase
>  Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17127.patch
>
>
> Currently if try to locate region when underlying connection is closed, we 
> will retry and wait at least 10s for each round until exhausted (refer to the 
> catch clause of {{RpcRetryingCallerImpl#callWithRetries}} and 
> {{RegionServerCallable#sleep}} for more details), which is unnecessary and 
> time-costing.
> The issue is caused by incorrect manipulating connection which shows the 
> disadvantage of force user to handle connection life cycle and proves the 
> necessity to support auto-managed connection as we did before, as indicated 
> in HBASE-17009
> In this JIRA we will make it fail fast in the above case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17114) Add an option to set special retry pause when encountering CallQueueTooBigException

[Yu Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677377#comment-15677377
 ] 

Yu Li commented on HBASE-17114:
---

bq. For advanced users who really need to treat CQTBE differently, that should 
be possible by means of an override, but should not be forced on everyone.
Agreed.

bq. I'm all for improving the client/server interactions in these scenarios, 
and what I first outlined in this issue was one idea for how to do that more 
effectively. However, I would also like us to avoid unexpected surprises for 
our users, and regressions in server behavior.
Yep, this is indeed a surprise for us since there wasn't any CQTBE thus no 
special handling in client side codes, and user kept complaining about "what's 
CQTBE and why it's happening when never before"...

One thing to clarify is that I didn't mean to deny the advantage of introducing 
CQTBE, and backporting and using it in our 1.1.2 is a proof (smile). My concern 
lies in the server side behavior change just like you mentioned. I think more 
document in ref guide would help for users upgrading from an old version w/o 
CQTBE.

bq. I'm not sure of the exact symptoms you're trying to solve, but if you're 
seeing issues with meta being overloaded, then I'd suggest tuning the 
configuration for the number of priority handlers and size of the priority 
queues.
Actually we did, we moved meta to an exclusive machine (no other regions on it) 
and increased priority handlers to 128 (and I'm afraid HBASE-15470 only goes 
into branch-1.3 and priority queue not controllable before) but still observed 
a high load, and that's why we further introduce the patch here.

bq. You could also evaluate running with meta hosted on master, which together 
with zk-less assignment can make region assignment much more stable.
This feature is also not available before branch-1.3 I'm afraid, and because 
currently master is light-weight and we could hot-switch it to apply some 
hot-fix, we may also don't want master to carry meta in future.



> Add an option to set special retry pause when encountering 
> CallQueueTooBigException
> ---
>
> Key: HBASE-17114
> URL: https://issues.apache.org/jira/browse/HBASE-17114
> Project: HBase
>  Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17114.patch
>
>
> As titled, after HBASE-15146 we will throw {{CallQueueTooBigException}} 
> instead of dead-wait. This is good for performance for most cases but might 
> cause a side-effect that if too many clients connect to the busy RS, that the 
> retry requests may come over and over again and RS never got the chance for 
> recovering, and the issue will become especially critical when the target 
> region is META.
> So here in this JIRA we propose to supply some special retry pause for CQTBE 
> in name of {{hbase.client.pause.special}}, and by default it will be 500ms (5 
> times of {{hbase.client.pause}} default value)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16708) Expose endpoint Coprocessor name in "responseTooSlow" log messages

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677359#comment-15677359
 ] 

Hadoop QA commented on HBASE-16708:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
1s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 15s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
8s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 14s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
8s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
1s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
27m 45s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 0s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 2s 
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
8s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 28s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12839588/HBASE-16708-V3.patch |
| JIRA Issue | HBASE-16708 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux d6de297f74af 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 046d1a5 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4530/testReport/ |
| modules | C: hbase-client U: hbase-client |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4530/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> Expose endpoint Coprocessor name in "responseTooSlow" log messages
> --
>
> Key: HBASE-16708
> URL: https://issues.apache.org/jira/browse/HBASE-16708
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Nick 

[jira] [Created] (HBASE-17128) Find Cause of a Write Perf Regression in branch-1.2

[stack (JIRA)]

stack created HBASE-17128:
-

 Summary: Find Cause of a Write Perf Regression in branch-1.2
 Key: HBASE-17128
 URL: https://issues.apache.org/jira/browse/HBASE-17128
 Project: HBase
  Issue Type: Task
Reporter: stack


As reported by [~gbaecher] up on the mailing list, there is a regression in 
1.2. The regression is in a CDH version of 1.2 actually but the CDH hbase is a 
near pure 1.2. This is a working issue to figure which of the below changes 
brought on slower writes (The list comes from doing the following...git log 
--oneline  
remotes/origin/cdh5-1.2.0_5.8.0_dev..remotes/origin/cdh5-1.2.0_5.9.0_dev ... I 
stripped the few CDH specific changes, packaging and tagging only, and then 
made two groupings; candidates and the unlikelies):

{code}
  1 bbc6762 HBASE-16023 Fastpath for the FIFO rpcscheduler Adds an executor 
that does balanced queue and fast path handing off requests directly to waiting 
handlers if any present. Idea taken from Apace Kudu (incubating). See 
https://gerr#
  2 a260917 HBASE-16288 HFile intermediate block level indexes might recurse 
forever creating multi TB files
  3 5633281 HBASE-15811 Batch Get after batch Put does not fetch all Cells We 
were not waiting on all executors in a batch to complete. The test for 
no-more-executors was damaged by the 0.99/0.98.4 fix "HBASE-11403 Fix race 
conditions aro#
  4 780f720 HBASE-11625 - Verifies data before building HFileBlock. - Adds 
HFileBlock.Header class which contains information about location of fields. 
Testing: Adds CorruptedFSReaderImpl to TestChecksum. (Apekshit)
  5 d735680 HBASE-12133 Add FastLongHistogram for metric computation (Yi Deng)
  6 c4ee832 HBASE-15222 Use less contended classes for metrics
  7
  8 17320a4 HBASE-15683 Min latency in latency histograms are emitted as 
Long.MAX_VALUE
  9 283b39f HBASE-15396 Enhance mapreduce.TableSplit to add encoded region name
 10 39db592 HBASE-16195 Should not add chunk into chunkQueue if not using chunk 
pool in HeapMemStoreLAB
 11 5ff28b7 HBASE-16194 Should count in MSLAB chunk allocation into heap size 
change when adding duplicate cells
 12 5e3e0d2 HBASE-16318 fail build while rendering velocity template if 
dependency license isn't in whitelist.
 13 3ed66e3 HBASE-16318 consistently use the correct name for 'Apache License, 
Version 2.0'
 14 351832d HBASE-16340 exclude Xerces iplementation jars from coming in 
transitively.
 15 b6aa4be HBASE-16321 ensure no findbugs-jsr305
 16 4f9dde7 HBASE-16317 revert all ESAPI changes
 17 71b6a8a HBASE-16284 Unauthorized client can shutdown the cluster (Deokwoo 
Han)
 18 523753f HBASE-16450 Shell tool to dump replication queues
 19 ca5f2ee HBASE-16379 [replication] Minor improvement to 
replication/copy_tables_desc.rb
 20 effd105 HBASE-16135 PeerClusterZnode under rs of removed peer may never be 
deleted
 21 a5c6610 HBASE-16319 Fix TestCacheOnWrite after HBASE-16288
 22 1956bb0 HBASE-15808 Reduce potential bulk load intermediate space usage and 
waste
 23 031c54e HBASE-16096 Backport. Cleanly remove replication peers from 
ZooKeeper.
 24 60a3b12 HBASE-14963 Remove use of Guava Stopwatch from HBase client code 
(Devaraj Das)
 25 c7724fc HBASE-16207 can't restore snapshot without "Admin" permission
 26 8322a0b HBASE-16227 [Shell] Column value formatter not working in scans. 
Tested : manually using shell.
 27 8f86658 HBASE-14818 user_permission does not list namespace permissions (li 
xiang)
 28 775cd21 HBASE-15465 userPermission returned by getUserPermission() for the 
selected namespace does not have namespace set (li xiang)
 29 8d85aff HBASE-16093 Fix splits failed before creating daughter regions 
leave meta inconsistent
 30 bc41317 HBASE-16140 bump owasp.esapi from 2.1.0 to 2.1.0.1
 31 6fc70cd HBASE-16035 Nested AutoCloseables might not all get closed (Sean 
Mackrory)
 32 fe28fe84 HBASE-15891. Closeable resources potentially not getting closed if 
exception is thrown.
 33 1d2bf3c HBASE-14644 Region in transition metric is broken -- addendum 
(Huaxiang Sun)
 34 fd5f56c HBASE-16056 Procedure v2 - fix master crash for FileNotFound
 35 10cd038 HBASE-16034 Fix ProcedureTestingUtility#LoadCounter.setMaxProcId()
 36 dae4db4 HBASE-15872 Split TestWALProcedureStore
 37 e638d86 HBASE-14644 Region in transition metric is broken (Huaxiang Sun)
 38 f01b01d HBASE-15496 Throw RowTooBigException only for user scan/get 
(Guanghao Zhang)
 39 cc0ce66 HBASE-15746 Remove extra RegionCoprocessor preClose() in 
RSRpcServices#closeRegion (Stephen Yuan Jiang)
 40 923f6d7 HBASE-15873 ACL for snapshot restore / clone is not enforced
 41 62df392 HBASE-15946. Eliminate possible security concerns in Store File 
metrics.
 42 293db90 HBASE-15925 provide default values for hadoop compat module related 
properties that match default hadoop profile.
 43 b1b5b66 HBASE-15889. String case conversions are locale-sensitive, used 
without locale
 44 4a8c4e7 HBASE-15698 

[jira] [Commented] (HBASE-16708) Expose endpoint Coprocessor name in "responseTooSlow" log messages

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677345#comment-15677345
 ] 

Jerry He commented on HBASE-16708:
--

+1 on v3.

> Expose endpoint Coprocessor name in "responseTooSlow" log messages
> --
>
> Key: HBASE-16708
> URL: https://issues.apache.org/jira/browse/HBASE-16708
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Nick Dimiduk
>Assignee: Yi Liang
> Fix For: 2.0.0
>
> Attachments: HBASE-16708-V1.patch, HBASE-16708-V2.patch, 
> HBASE-16708-V3.patch, HBASE-ShortName.patch
>
>
> Operational diagnostics of a Phoenix install would be easier if we included 
> which endpoint coprocessor was being called in this responseTooSlow WARN 
> message.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16663) JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-16663:
---
Attachment: 16663.branch-1.1.patch

> JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
> 
>
> Key: HBASE-16663
> URL: https://issues.apache.org/jira/browse/HBASE-16663
> Project: HBase
>  Issue Type: Bug
>  Components: metrics, security
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.4, 0.98.24, 1.1.8
>
> Attachments: 16663-branch-1.1.00.patch, 16663.branch-1.1.patch, 
> HBASE-16663-0.98-V4.patch, HBASE-16663-0.98.patch, HBASE-16663-V2.patch, 
> HBASE-16663-V3.patch, HBASE-16663-V4.patch, HBASE-16663-branch-1.patch, 
> HBASE-16663.patch
>
>
> After HBASE-16284, unauthorized user will not able allowed to stop 
> HM/RS/cluster, but while executing "cpHost.preStopMaster()", ConnectorServer 
> will be stopped before AccessController validation.
> hbase-site.xml,
> {noformat}
>  
>   hbase.coprocessor.master.classes
> 
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>  
>   
>   hbase.coprocessor.regionserver.classes
> 
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>   
> {noformat}
> HBaseAdmin.stopMaster(),
> {noformat}
> 2016-09-20 21:12:26,796 INFO  
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000] 
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 21:13:55,380 WARN  
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000] 
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for 
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 21:14:00,495 ERROR 
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000] 
> master.MasterRpcServices: Exception occurred while stopping master
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient 
> permissions for user 'P72981' (global, action=ADMIN)
>   at 
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
>   at 
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
>   at 
> org.apache.hadoop.hbase.security.access.AccessController.preStopMaster(AccessController.java:1297)
>   at 
> org.apache.hadoop.hbase.master.MasterCoprocessorHost$68.call(MasterCoprocessorHost.java:821)
>   at 
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1188)
>   at 
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.preStopMaster(MasterCoprocessorHost.java:817)
>   at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2352)
>   at 
> org.apache.hadoop.hbase.master.MasterRpcServices.stopMaster(MasterRpcServices.java:1364)
> {noformat}
> HBaseAdmin.stopRegionServer(rs-host-port),
> {noformat}
> 2016-09-20 20:59:01,234 INFO  
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020] 
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 20:59:01,250 WARN  
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020] 
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for 
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 20:59:01,253 WARN  
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020] 
> regionserver.HRegionServer: The region server did not stop
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient 
> permissions for user 'P72981' (global, action=ADMIN)
>   at 
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
>   at 
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
>   at 
> org.apache.hadoop.hbase.security.access.AccessController.preStopRegionServer(AccessController.java:2501)
>   at 
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$1.call(RegionServerCoprocessorHost.java:84)
>   at 
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.execOperation(RegionServerCoprocessorHost.java:256)
>   at 
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.preStop(RegionServerCoprocessorHost.java:80)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.stop(HRegionServer.java:1905)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.stopServer(RSRpcServices.java:1961)
> {noformat}
> HBaseAdmin.shutdown(),
> {noformat}
> 2016-09-21 12:09:08,259 INFO  
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000] 
> 

[jira] [Updated] (HBASE-17080) rest.TestTableResource fails in master branch

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-17080:
---
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Thanks for the patch, ChiaPing

> rest.TestTableResource fails in master branch
> -
>
> Key: HBASE-17080
> URL: https://issues.apache.org/jira/browse/HBASE-17080
> Project: HBase
>  Issue Type: Test
>Affects Versions: 2.0.0
>Reporter: Ted Yu
>Assignee: ChiaPing Tsai
> Fix For: 2.0.0
>
> Attachments: HBASE-17080.v0.patch
>
>
> The test fails consistently.
> {code}
> testTableInfoPB(org.apache.hadoop.hbase.rest.TestTableResource)  Time 
> elapsed: 0.118 sec  <<< FAILURE!
> org.junit.ComparisonFailure: expected:<192.168.0.14:5055[8]> but 
> was:<192.168.0.14:5055[5]>
>   at org.junit.Assert.assertEquals(Assert.java:115)
>   at org.junit.Assert.assertEquals(Assert.java:144)
>   at 
> org.apache.hadoop.hbase.rest.TestTableResource.checkTableInfo(TestTableResource.java:191)
>   at 
> org.apache.hadoop.hbase.rest.TestTableResource.testTableInfoPB(TestTableResource.java:272)
> testTableInfoXML(org.apache.hadoop.hbase.rest.TestTableResource)  Time 
> elapsed: 0.084 sec  <<< FAILURE!
> org.junit.ComparisonFailure: expected:<192.168.0.14:5055[8]> but 
> was:<192.168.0.14:5055[5]>
>   at org.junit.Assert.assertEquals(Assert.java:115)
>   at org.junit.Assert.assertEquals(Assert.java:144)
>   at 
> org.apache.hadoop.hbase.rest.TestTableResource.checkTableInfo(TestTableResource.java:191)
>   at 
> org.apache.hadoop.hbase.rest.TestTableResource.testTableInfoXML(TestTableResource.java:255)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17114) Add an option to set special retry pause when encountering CallQueueTooBigException

[Yu Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677310#comment-15677310
 ] 

Yu Li commented on HBASE-17114:
---

All points on current patch makes sense to me, let me update the patch. Thanks 
for review [~ghelmling].

> Add an option to set special retry pause when encountering 
> CallQueueTooBigException
> ---
>
> Key: HBASE-17114
> URL: https://issues.apache.org/jira/browse/HBASE-17114
> Project: HBase
>  Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17114.patch
>
>
> As titled, after HBASE-15146 we will throw {{CallQueueTooBigException}} 
> instead of dead-wait. This is good for performance for most cases but might 
> cause a side-effect that if too many clients connect to the busy RS, that the 
> retry requests may come over and over again and RS never got the chance for 
> recovering, and the issue will become especially critical when the target 
> region is META.
> So here in this JIRA we propose to supply some special retry pause for CQTBE 
> in name of {{hbase.client.pause.special}}, and by default it will be 500ms (5 
> times of {{hbase.client.pause}} default value)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HBASE-17123) Add postBulkLoadHFile variant that notifies the final paths for the hfiles

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu reassigned HBASE-17123:
--

Assignee: Ted Yu

> Add postBulkLoadHFile variant that notifies the final paths for the hfiles
> --
>
> Key: HBASE-17123
> URL: https://issues.apache.org/jira/browse/HBASE-17123
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 17123.v1.txt, 17123.v3.txt
>
>
> Currently the postBulkLoadHFile() hook passes the same familyPaths parameter 
> which it receives as method parameter.
> See code in SecureBulkLoadManager :
> {code}
>loaded = region.getCoprocessorHost().postBulkLoadHFile(familyPaths, 
> loaded);
> {code}
> Meaning, the paths are not final, moved paths of the loaded hfiles.
> This issue is to add a variant which notifies the final paths of the loaded 
> hfiles.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17118) StoreScanner leaked in KeyValueHeap

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677305#comment-15677305
 ] 

Ted Yu commented on HBASE-17118:


[~aoxiang]:
Mind bringing this to other branches now that you have the commit bit ?

For 1.3, wait for Mikhail's signal.

Thanks

> StoreScanner leaked in KeyValueHeap
> ---
>
> Key: HBASE-17118
> URL: https://issues.apache.org/jira/browse/HBASE-17118
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: binlijin
>Assignee: binlijin
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17118-master_v1.patch, 
> HBASE-17118-master_v2.patch, HBASE-17118-master_v3.patch, 
> HBASE-17118-master_v4.patch, HBASE-17118-master_v5.patch, 
> HBASE-17118.branch-1.v1.patch, StoreScanner.png, StoreScannerLeakHeap.png
>
>
> KeyValueHeap#generalizedSeek
>   KeyValueScanner scanner = current;
>   while (scanner != null) {
> Cell topKey = scanner.peek();
> ..
> boolean seekResult;
> if (isLazy && heap.size() > 0) {
>   // If there is only one scanner left, we don't do lazy seek.
>   seekResult = scanner.requestSeek(seekKey, forward, useBloom);
> } else {
>   seekResult = NonLazyKeyValueScanner.doRealSeek(scanner, seekKey,
>   forward);
> }
> ..
> scanner = heap.poll();
>   }
> (1) scanner = heap.poll();  Retrieves and removes the head of this queue
> (2) scanner.requestSeek(seekKey, forward, useBloom); or 
> NonLazyKeyValueScanner.doRealSeek(scanner, seekKey, forward);
> throw exception, and scanner will have no chance to close, so will cause the 
> scanner leak.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17114) Add an option to set special retry pause when encountering CallQueueTooBigException

[Gary Helmling (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677304#comment-15677304
 ] 

Gary Helmling commented on HBASE-17114:
---

bq. Well, if checking the uploaded patch, it's indeed tied to CQTBE only. 
Introducing a new property is only for making things more flexible, and of 
course we could use a hard-coded, like 5 times than the existing pause, for 
CQTBE. But I'd say this is a trade-off, waiting longer for CQTBE could prevent 
the vicious circle but is also causing a higher latency, and IMHO user should 
be able to control such trade-off. If they don't want CQTBE to be special, they 
could set hbase.client.pause.special to the same value as hbase.client.pause, 
which gives them more options.

I agree with allowing the user to control the behavior here, but this is also 
increasing complexity and knowledge needed for configuration tuning, which we 
already have way too much of.  In general, we should be moving in the direction 
of making the system dynamically tune itself according to load instead of 
forcing all users to grapple with yet another configuration property.  By 
default the configuration should be simple to provide the best experience to 
all users.  For advanced users who really need to treat CQTBE differently, that 
should be possible by means of an override, but should not be forced on 
everyone.

bq. Sorry but I don't see any difference in "should not clear the client meta 
cache" and "should not retry so frequently", both trying to resolve some 
problem and make things better.

These are two completely different things.  I don't see the equivalence.  We 
don't clear the meta cache because we don't have an indication that the region 
has moved, so there is no need to go back to meta.  The meta cache handling is 
completely independent of what is appropriate in terms of retries.

bq. No offense but I'm even thinking of making CQTBE thrown optional, because 
for some case dead-wait for the request to be executed in RpcServer until 
time-out is preferable by user rather than receiving some exception and retry 
and fail again, but obviously this is another topic (Smile).

Blocking the RpcServer Reader threads indefinitely when the queue is full, 
making the server completely unresponsive and spilling overflow back in to the 
OS networking buffers is pretty poor behavior.  CQTBE is a crude mechanism for 
back-pressure to the client, but at least it gets the client a response and 
allows it to make an informed decision about how to proceed.  In the case where 
the application implements its own retries the client may want to simply fail 
and kick the exception back up the stack, allowing other layers to retry.  Or 
the client could decide to retry for a fixed duration.  But in either case I 
think CQTBE provides a very clear improvement in overall server behavior.  
Another part of the puzzle is the CoDel scheduler which will allow more useful 
work to get done in overloaded situations.

I'm all for improving the client/server interactions in these scenarios, and 
what I first outlined in this issue was one idea for how to do that more 
effectively.  However, I would also like us to avoid unexpected surprises for 
our users, and regressions in server behavior.

I'm not sure of the exact symptoms you're trying to solve, but if you're seeing 
issues with meta being overloaded, then I'd suggest tuning the configuration 
for the number of priority handlers and size of the priority queues.  You could 
also evaluate running with meta hosted on master, which together with zk-less 
assignment can make region assignment much more stable.

> Add an option to set special retry pause when encountering 
> CallQueueTooBigException
> ---
>
> Key: HBASE-17114
> URL: https://issues.apache.org/jira/browse/HBASE-17114
> Project: HBase
>  Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17114.patch
>
>
> As titled, after HBASE-15146 we will throw {{CallQueueTooBigException}} 
> instead of dead-wait. This is good for performance for most cases but might 
> cause a side-effect that if too many clients connect to the busy RS, that the 
> retry requests may come over and over again and RS never got the chance for 
> recovering, and the issue will become especially critical when the target 
> region is META.
> So here in this JIRA we propose to supply some special retry pause for CQTBE 
> in name of {{hbase.client.pause.special}}, and by default it will be 500ms (5 
> times of {{hbase.client.pause}} default value)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)