[jira] [Commented] (HBASE-18180) Possible connection leak while closing BufferedMutator in TableOutputFormat

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050022#comment-16050022
 ] 

Hadoop QA commented on HBASE-18180:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 
8s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 36s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 
2s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 5s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 36s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 36s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
61m 20s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha3. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 33m 57s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
15s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 135m 57s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.locking.TestLockProcedure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:757bf37 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12873053/HBASE-18180.patch |
| JIRA Issue | HBASE-18180 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 713ca0264761 4.8.3-std-1 #1 SMP Fri Oct 21 11:15:43 UTC 2016 
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 8b36da1 |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/7189/artifact/patchprocess/patch-unit-hbase-server.txt
 |
| unit test logs |  
https://builds.apache.org/job/PreCommit-HBASE-Build/7189/artifact/patchprocess/patch-unit-hbase-server.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/7189/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/7189/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> Possible connection leak while 

[jira] [Commented] (HBASE-18105) [AMv2] Split/Merge need cleanup; currently they diverge and do not fully embrace AMv2 world

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050009#comment-16050009
 ] 

stack commented on HBASE-18105:
---

bq.  I think we need to support this in splitProcedure.

Agree.

bq.  I have a idea to solve this problem, which is add a property called 
bestSplitPoints in HRegionInfo, since splitProcedure always send a rpc request 
getRegionInfoRequest to RSrpcservice to check the region is splitable or not. 

Agree w/ the basic idea. FYI, the rpc to ask the region if it is splittable is 
a new addition added a few weeks ago. The master-side was splitting whenever it 
was asked whether a region was splittable or not. Rather than add to 
HRegionInfo, add to the GetRegionInfoResponse protobuf?


> [AMv2] Split/Merge need cleanup; currently they diverge and do not fully 
> embrace AMv2 world
> ---
>
> Key: HBASE-18105
> URL: https://issues.apache.org/jira/browse/HBASE-18105
> Project: HBase
>  Issue Type: Sub-task
>  Components: Region Assignment
>Affects Versions: 2.0.0
>Reporter: stack
> Fix For: 2.0.0
>
>
> Region Split and Merge work on the new AMv2 but they work differently. This 
> issue is about bringing them back together and fully embracing the AMv2 
> program.
> They both have issues mostly the fact that they carry around baggage no 
> longer necessary in the new world of assignment.
> Here are some of the items:
> Split and Merge metadata modifications are done by the Master now but we have 
> vestige of Split/Merge on RS still; e.g. when we SPLIT, we ask the Master 
> which asks the RS, which turns around, and asks the Master to run the 
> operation. Fun. MERGE is all done Master-side.
>  
> Clean this up. Remove asking RS to run SPLIT and remove RegionMergeRequest, 
> etc. on RS-side. Also remove PONR. We don’t Points-Of-No-Return now we are up 
> on Pv2. Remove all calls in Interfaces; they are unused. Make RS still able 
> to detect when split, but have it be a client of Master like anyone else.
> Split is Async but does not return procId
> Split is async. Doesn’t return the procId though. Merge does. Fix. Only hard 
> part here I think is the Admin API does not allow procid return.
> Flags
> Currently OFFLINE is determined by looking either at the master instance of 
> HTD (isOffline) and/or at the RegionState#state. Ditto for SPLIT. For MERGE, 
> we rely on RegionState#state. Related is a note above on how split works -- 
> there is a split flag in HTD when there should not be.
>  
> TODO is move to rely on RegionState#state exclusively in Master.
> From Split/Merge Procedures need finishing in 
> https://docs.google.com/document/d/1eVKa7FHdeoJ1-9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#heading=h.4b60dc1h4m1f



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HBASE-18010) Connect CellChunkMap to be used for flattening in CompactingMemStore

[ramkrishna.s.vasudevan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan reassigned HBASE-18010:
--

Assignee: Anastasia Braginsky

> Connect CellChunkMap to be used for flattening in CompactingMemStore
> 
>
> Key: HBASE-18010
> URL: https://issues.apache.org/jira/browse/HBASE-18010
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>Assignee: Anastasia Braginsky
>
> The CellChunkMap helps to create a new type of ImmutableSegment, where the 
> index (CellSet's delegatee) is going to be CellChunkMap. No big cells or 
> upserted cells are going to be supported here.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18010) Connect CellChunkMap to be used for flattening in CompactingMemStore

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1605#comment-1605
 ] 

ramkrishna.s.vasudevan commented on HBASE-18010:


Will spend some time this week and early next week. Thanks for the RB link too.

> Connect CellChunkMap to be used for flattening in CompactingMemStore
> 
>
> Key: HBASE-18010
> URL: https://issues.apache.org/jira/browse/HBASE-18010
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>
> The CellChunkMap helps to create a new type of ImmutableSegment, where the 
> index (CellSet's delegatee) is going to be CellChunkMap. No big cells or 
> upserted cells are going to be supported here.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Encryption of exisiting data in Stripe Compaction

[ramkrishna vasudevan]

Hi
Very interesting case. Ya Stripe compaction does not need to under go a
major compaction if it already running under stripe compaction (reading the
docs I get this).
Since you have enable encryption at a later point of time you face this
issue I believe. The naive workaround I can think of is that do a alter
table with default compaction and it will do a major compaction and once
that is done again move back to Stripe compaction?  Will that work?

I would like to hear opinion of others who have experience with Strip
compaction.

Regards
Ram

On Wed, Jun 14, 2017 at 10:25 AM, Karthick Ram 
wrote:

> We have a table which has time series data with Stripe Compaction enabled.
> After encryption has been enabled for this table the newer entries are
> encrypted and inserted. However to encrypt the existing data in the table,
> a major compaction has to run. Since, stripe compaction doesn't allow a
> major compaction to run, we are unable to encrypt the previous data. Please
> suggest some ways to rectify this problem.
>
> Regards,
> Karthick R
>

[jira] [Commented] (HBASE-18219) Fix typo in constant HConstants.HBASE_CLIENT_MEAT_REPLICA_SCAN_TIMEOUT

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049976#comment-16049976
 ] 

Hudson commented on HBASE-18219:


SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #3197 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/3197/])
HBASE-18219 Fix typo in constant (apurtell: rev 
50e28d62a6f21997cb6ff929d84e497d7ccb204c)
* (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestReplicaWithCluster.java
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionConfiguration.java


> Fix typo in constant HConstants.HBASE_CLIENT_MEAT_REPLICA_SCAN_TIMEOUT
> --
>
> Key: HBASE-18219
> URL: https://issues.apache.org/jira/browse/HBASE-18219
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 3.0.0, 1.4.0
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Fix For: 2.0.0, 3.0.0, 1.4.0
>
> Attachments: HBASE-18219-branch-1.patch, HBASE-18219.patch
>
>
> HConstants has a constant HBASE_CLIENT_MEAT_REPLICA_SCAN_TIMEOUT that should 
> be HBASE_CLIENT_META_REPLICA_SCAN_TIMEOUT



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18209) Include httpclient / httpcore jars in build artifacts

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-18209:
---
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 2.0.0-alpha-2
   3.0.0
   Status: Resolved  (was: Patch Available)

Thanks for the review, Josh.

> Include httpclient / httpcore jars in build artifacts
> -
>
> Key: HBASE-18209
> URL: https://issues.apache.org/jira/browse/HBASE-18209
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: 18209.v1.txt, 18209.v2.txt
>
>
> We need httpclient & httpcore jars to be present when rootdir is placed on 
> s3(a).
> Attempts to move to the fully shaded amazon-SDK JAR caused problems of its 
> own. (according to [~steve_l])
> Here are the versions we should use:
> 4.5.2
> 4.4.4
> Currently they are declared test dependency.
> This JIRA is to move to compile time dependency so that the corresponding 
> jars are bundled in lib directory.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18217) Correct comment errors in TestScannerHeartbeatMessages

[Phil Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049971#comment-16049971
 ] 

Phil Yang commented on HBASE-18217:
---

Time limit is half of timeout, see RSRpcServices

> Correct comment errors in TestScannerHeartbeatMessages
> --
>
> Key: HBASE-18217
> URL: https://issues.apache.org/jira/browse/HBASE-18217
> Project: HBase
>  Issue Type: Bug
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
>
> 1. For CLIENT_TIMEOUT
> {code}
>   // Time, in milliseconds, that the client will wait for a response from the 
> server before timing
>   // out. This value is used server side to determine when it is necessary to 
> send a heartbeat
>   // message to the client. Time limit will be 500 ms.
>   private static int CLIENT_TIMEOUT = 1000;
> {code}
> I think “Time limit will be 500 ms.” should be removed
> 2. For DEFAULT_ROW_SLEEP_TIME
> Typo, "same" -> "some", in
> // In this test, we sleep after reading each row. So we should make sure 
> after we get some number
> // of rows and sleep {color:red}same{color} times we must reach time limit, 
> and do not timeout after next sleeping.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18219) Fix typo in constant HConstants.HBASE_CLIENT_MEAT_REPLICA_SCAN_TIMEOUT

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049970#comment-16049970
 ] 

Hudson commented on HBASE-18219:


SUCCESS: Integrated in Jenkins build HBase-2.0 #45 (See 
[https://builds.apache.org/job/HBase-2.0/45/])
HBASE-18219 Fix typo in constant (apurtell: rev 
6af3ee44c0138ac5c0e05b0bbfd68fc703137631)
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionConfiguration.java
* (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestReplicaWithCluster.java


> Fix typo in constant HConstants.HBASE_CLIENT_MEAT_REPLICA_SCAN_TIMEOUT
> --
>
> Key: HBASE-18219
> URL: https://issues.apache.org/jira/browse/HBASE-18219
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 3.0.0, 1.4.0
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Fix For: 2.0.0, 3.0.0, 1.4.0
>
> Attachments: HBASE-18219-branch-1.patch, HBASE-18219.patch
>
>
> HConstants has a constant HBASE_CLIENT_MEAT_REPLICA_SCAN_TIMEOUT that should 
> be HBASE_CLIENT_META_REPLICA_SCAN_TIMEOUT



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18209) Include httpclient / httpcore jars in build artifacts

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049969#comment-16049969
 ] 

Hadoop QA commented on HBASE-18209:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 1s 
{color} | {color:blue} The patch file was not named according to hbase's naming 
conventions. Please see 
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for 
instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 23s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 5s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 47s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 3s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
26m 49s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha3. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 122m 4s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s 
{color} | {color:green} hbase-assembly in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
37s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 159m 11s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.coprocessor.TestRegionObserverInterface |
|   | hadoop.hbase.master.procedure.TestMasterProcedureWalLease |
| Timed out junit tests | 
org.apache.hadoop.hbase.coprocessor.TestCoprocessorMetrics |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:757bf37 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12873049/18209.v2.txt |
| JIRA Issue | HBASE-18209 |
| Optional Tests |  asflicense  javac  javadoc  unit  xml  compile  |
| uname | Linux b033c37e475a 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 50e28d6 |
| Default Java | 1.8.0_131 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/7188/artifact/patchprocess/patch-unit-hbase-server.txt
 |
| unit test logs |  
https://builds.apache.org/job/PreCommit-HBASE-Build/7188/artifact/patchprocess/patch-unit-hbase-server.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/7188/testReport/ |
| 

[jira] [Updated] (HBASE-18161) MultiHFileOutputFormat - comprehensive incremental load support

[Densel Santhmayor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Densel Santhmayor updated HBASE-18161:
--
Attachment: MultiHFileOutputFormatSupport_HBASE_18161_v5.patch

> MultiHFileOutputFormat - comprehensive incremental load support
> ---
>
> Key: HBASE-18161
> URL: https://issues.apache.org/jira/browse/HBASE-18161
> Project: HBase
>  Issue Type: New Feature
>Reporter: Densel Santhmayor
>Priority: Minor
> Attachments: MultiHFileOutputFormatSupport_HBASE_18161.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v2.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v3.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v4.patch, 
> MultiHFileOutputFormatSupport_HBASE_18161_v5.patch
>
>
> h2. Introduction
> MapReduce currently supports the ability to write HBase records in bulk to 
> HFiles for a single table. The file(s) can then be uploaded to the relevant 
> RegionServers information with reasonable latency. This feature is useful to 
> make a large set of data available for queries at the same time as well as 
> provides a way to efficiently process very large input into HBase without 
> affecting query latencies.
> There is, however, no support to write variations of the same record key to 
> HFiles belonging to multiple HBase tables from within the same MapReduce job. 
>  
> h2. Goal
> The goal of this JIRA is to extend HFileOutputFormat2 to support writing to 
> HFiles for different tables within the same MapReduce job while single-table 
> HFile features backwards-compatible. 
> For our use case, we needed to write a record key to a smaller HBase table 
> for quicker access, and the same record key with a date appended to a larger 
> table for longer term storage with chronological access. Each of these tables 
> would have different TTL and other settings to support their respective 
> access patterns. We also needed to be able to bulk write records to multiple 
> tables with different subsets of very large input as efficiently as possible. 
> Rather than run the MapReduce job multiple times (one for each table or 
> record structure), it would be useful to be able to parse the input a single 
> time and write to multiple tables simultaneously.
> Additionally, we'd like to maintain backwards compatibility with the existing 
> heavily-used HFileOutputFormat2 interface to allow benefits such as locality 
> sensitivity (that was introduced long after we implemented support for 
> multiple tables) to support both single table and multi table hfile writes. 
> h2. Proposal
> * Backwards compatibility for existing single table support in 
> HFileOutputFormat2 will be maintained and in this case, mappers will need to 
> emit the table rowkey as before. However, a new class - 
> MultiHFileOutputFormat - will provide a helper function to generate a rowkey 
> for mappers that prefixes the desired tablename to the existing rowkey as 
> well as provides configureIncrementalLoad support for multiple tables.
> * HFileOutputFormat2 will be updated in the following way:
> ** configureIncrementalLoad will now accept multiple table descriptor and 
> region locator pairs, analogous to the single pair currently accepted by 
> HFileOutputFormat2. 
> ** Compression, Block Size, Bloom Type and Datablock settings PER column 
> family that are set in the Configuration object are now indexed and retrieved 
> by tablename AND column family
> ** getRegionStartKeys will now support multiple regionlocators and calculate 
> split points and therefore partitions collectively for all tables. Similarly, 
> now the eventual number of Reducers will be equal to the total number of 
> partitions across all tables. 
> ** The RecordWriter class will be able to process rowkeys either with or 
> without the tablename prepended depending on how configureIncrementalLoad was 
> configured with MultiHFileOutputFormat or HFileOutputFormat2.
> * The use of MultiHFileOutputFormat will write the output into HFiles which 
> will match the output format of HFileOutputFormat2. However, while the 
> default use case will keep the existing directory structure with column 
> family name as the directory and HFiles within that directory, in the case of 
> MultiHFileOutputFormat, it will output HFiles in the output directory with 
> the following relative paths: 
> {noformat}
>  --table1 
>--family1 
>  --HFiles 
>  --table2 
>--family1 
>--family2 
>  --HFiles
> {noformat}
> This aims to be a comprehensive solution to the original tickets - HBASE-3727 
> and HBASE-16261. Thanks to [~clayb] for his support.
> The patch will be attached shortly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18212) In Standalone mode with local filesystem HBase logs Warning message:Failed to invoke 'unbuffer' method in class class org.apache.hadoop.fs.FSDataInputStream

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049964#comment-16049964
 ] 

Ashish Singhi commented on HBASE-18212:
---

Shall we move it to trace level ?

> In Standalone mode with local filesystem HBase logs Warning message:Failed to 
> invoke 'unbuffer' method in class class org.apache.hadoop.fs.FSDataInputStream
> 
>
> Key: HBASE-18212
> URL: https://issues.apache.org/jira/browse/HBASE-18212
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Umesh Agashe
>
> New users may get nervous after seeing following warning level log messages 
> (considering new users will most likely run HBase in Standalone mode first):
> {code}
> WARN  [MemStoreFlusher.1] io.FSDataInputStreamWrapper: Failed to invoke 
> 'unbuffer' method in class class org.apache.hadoop.fs.FSDataInputStream . So 
> there may be a TCP socket connection left open in CLOSE_WAIT state.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18180) Possible connection leak while closing BufferedMutator in TableOutputFormat

[Pankaj Kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pankaj Kumar updated HBASE-18180:
-
Fix Version/s: 3.0.0

> Possible connection leak while closing BufferedMutator in TableOutputFormat
> ---
>
> Key: HBASE-18180
> URL: https://issues.apache.org/jira/browse/HBASE-18180
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 1.4.0, 1.3.1, 1.3.2
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 3.0.0, 1.4.0
>
> Attachments: HBASE-18180-branch-1.patch, HBASE-18180.patch
>
>
> In TableOutputFormat, connection will not be released in case when 
> "mutator.close()" throws exception.
> org.apache.hadoop.hbase.mapreduce.TableOutputFormat
> {code}
> public void close(TaskAttemptContext context)
> throws IOException {
>   mutator.close();
>   connection.close();
> }
> {code}
> org.apache.hadoop.hbase.mapred.TableOutputFormat
> {code}
> public void close(Reporter reporter) throws IOException {
>   this.m_mutator.close();
>   if (connection != null) {
> connection.close();
> connection = null;
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18180) Possible connection leak while closing BufferedMutator in TableOutputFormat

[Pankaj Kumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049956#comment-16049956
 ] 

Pankaj Kumar commented on HBASE-18180:
--

Done.

> Possible connection leak while closing BufferedMutator in TableOutputFormat
> ---
>
> Key: HBASE-18180
> URL: https://issues.apache.org/jira/browse/HBASE-18180
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 1.4.0, 1.3.1, 1.3.2
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 1.4.0
>
> Attachments: HBASE-18180-branch-1.patch, HBASE-18180.patch
>
>
> In TableOutputFormat, connection will not be released in case when 
> "mutator.close()" throws exception.
> org.apache.hadoop.hbase.mapreduce.TableOutputFormat
> {code}
> public void close(TaskAttemptContext context)
> throws IOException {
>   mutator.close();
>   connection.close();
> }
> {code}
> org.apache.hadoop.hbase.mapred.TableOutputFormat
> {code}
> public void close(Reporter reporter) throws IOException {
>   this.m_mutator.close();
>   if (connection != null) {
> connection.close();
> connection = null;
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18180) Possible connection leak while closing BufferedMutator in TableOutputFormat

[Pankaj Kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pankaj Kumar updated HBASE-18180:
-
Attachment: HBASE-18180.patch

> Possible connection leak while closing BufferedMutator in TableOutputFormat
> ---
>
> Key: HBASE-18180
> URL: https://issues.apache.org/jira/browse/HBASE-18180
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 1.4.0, 1.3.1, 1.3.2
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 1.4.0
>
> Attachments: HBASE-18180-branch-1.patch, HBASE-18180.patch
>
>
> In TableOutputFormat, connection will not be released in case when 
> "mutator.close()" throws exception.
> org.apache.hadoop.hbase.mapreduce.TableOutputFormat
> {code}
> public void close(TaskAttemptContext context)
> throws IOException {
>   mutator.close();
>   connection.close();
> }
> {code}
> org.apache.hadoop.hbase.mapred.TableOutputFormat
> {code}
> public void close(Reporter reporter) throws IOException {
>   this.m_mutator.close();
>   if (connection != null) {
> connection.close();
> connection = null;
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18180) Possible connection leak while closing BufferedMutator in TableOutputFormat

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049950#comment-16049950
 ] 

Ted Yu commented on HBASE-18180:


Please re-attach patch for master branch.

> Possible connection leak while closing BufferedMutator in TableOutputFormat
> ---
>
> Key: HBASE-18180
> URL: https://issues.apache.org/jira/browse/HBASE-18180
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 1.4.0, 1.3.1, 1.3.2
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 1.4.0
>
> Attachments: HBASE-18180-branch-1.patch
>
>
> In TableOutputFormat, connection will not be released in case when 
> "mutator.close()" throws exception.
> org.apache.hadoop.hbase.mapreduce.TableOutputFormat
> {code}
> public void close(TaskAttemptContext context)
> throws IOException {
>   mutator.close();
>   connection.close();
> }
> {code}
> org.apache.hadoop.hbase.mapred.TableOutputFormat
> {code}
> public void close(Reporter reporter) throws IOException {
>   this.m_mutator.close();
>   if (connection != null) {
> connection.close();
> connection = null;
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18209) Include httpclient / httpcore jars in build artifacts

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049934#comment-16049934
 ] 

Ted Yu commented on HBASE-18209:


>From the tar ball built with patch:
{code}
-rw-rw-r-- hbase/hadoop  736658 2016-08-20 17:47 
hbase-3.0.0-SNAPSHOT/lib/httpclient-4.5.2.jar
-rw-rw-r-- hbase/hadoop  326724 2016-08-20 17:47 
hbase-3.0.0-SNAPSHOT/lib/httpcore-4.4.4.jar
{code}

> Include httpclient / httpcore jars in build artifacts
> -
>
> Key: HBASE-18209
> URL: https://issues.apache.org/jira/browse/HBASE-18209
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 18209.v1.txt, 18209.v2.txt
>
>
> We need httpclient & httpcore jars to be present when rootdir is placed on 
> s3(a).
> Attempts to move to the fully shaded amazon-SDK JAR caused problems of its 
> own. (according to [~steve_l])
> Here are the versions we should use:
> 4.5.2
> 4.4.4
> Currently they are declared test dependency.
> This JIRA is to move to compile time dependency so that the corresponding 
> jars are bundled in lib directory.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18054) log when we add/remove failed servers in client

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049932#comment-16049932
 ] 

Hadoop QA commented on HBASE-18054:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
29s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 12s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
31s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 
27s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
28m 7s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha3. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s 
{color} | {color:green} hbase-client generated 0 new + 1 unchanged - 1 fixed = 
1 total (was 2) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s 
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 20s 
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 107m 24s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
33s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 157m 27s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | 
org.apache.hadoop.hbase.coprocessor.TestCoprocessorMetrics |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:757bf37 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12873048/HBASE-18054.v4.master.patch
 |
| JIRA Issue | HBASE-18054 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux fe759c7a4a59 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 50e28d6 |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| unit | 

[jira] [Commented] (HBASE-18023) Log multi-* requests for more than threshold number of rows

[Josh Elser (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049913#comment-16049913
 ] 

Josh Elser commented on HBASE-18023:


How about {{hbase.rpc.rows.warning.threshold}} instead of 
{hbase.batch.size.threshold}}. Including some form of "warn" is good and I was 
trying to avoid the confusion (at sight) between size being "number of rows" 
and "number of bytes".

{code}
+  int threshold = conf.getInt(HConstants.BATCH_SIZE_THRESHOLD_NAME, 
HConstants.BATCH_SIZE_THRESHOLD_DEFAULT);
{code}

We should cache this to save on frequently accessing the value from the 
configuration. We can do it when HRegion is constructed (or RSRpcServices is 
constructed, based on more comments to come..).

{code}
+  LOG.warn("Large batch operation detected (greater than 
"+threshold+") (See https://issues.apache.org/jira/browse/HBASE-18023)."
{code}

Personally, I don't think the JIRA URL is going to be of much value to an 
admin. I would strike that part.

{code}
+  + " Length: "+batchOp.operations.length
{code}

How about "Requested number of rows:" instead?

{code}
@@ -3200,6 +3209,7 @@ public class HRegion implements HeapSize, 
PropagatingConfigurationObserver, Regi
 List acquiredRowLocks = 
Lists.newArrayListWithCapacity(batchOp.operations.length);
 MemstoreSize memstoreSize = new MemstoreSize();
 final ObservedExceptionsInBatch observedExceptions = new 
ObservedExceptionsInBatch();
+checkBatchSizeAndLogLargeSize(batchOp);
{code}

I'm a little worried about the case where one RPC contains a updates for a 
number of regions. Your current case properly handles the case where there are 
more than 1000 updates to a single region.

However, multi RPCs can contain actions for many regions in one RPC. So, if I 
crafted a multi RPC in which I included 1000 Region action objects each of 
which only have 999 operations (so, 999 operations for 1000 different regions 
for a total of 999000 actions), the current logging wouldn't catch anything. 
What about lifting this method out of HRegion and into {{RSRpcServices#multi}} 
instead? This way, we can invoke your method on the total RPC request object 
sent in, not a portion of it. Does this make sense?

Nice test addition too by the way. We might be able to make your test a little 
more stable by avoiding writing a test expecting a specific message is logged. 
Instead, what if you moved your {{LOG.warn}} to its own method, mocked out the 
invocation of that method and then verified that the method was called (just a 
different way to show that the log statement would be printed in a real system).

> Log multi-* requests for more than threshold number of rows
> ---
>
> Key: HBASE-18023
> URL: https://issues.apache.org/jira/browse/HBASE-18023
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: Clay B.
>Assignee: David Harju
>Priority: Minor
> Attachments: HBASE-18023.master.001.patch
>
>
> Today, if a user happens to do something like a large multi-put, they can get 
> through request throttling (e.g. it is one request) but still crash a region 
> server with a garbage storm. We have seen regionservers hit this issue and it 
> is silent and deadly. The RS will report nothing more than a mysterious 
> garbage collection and exit out.
> Ideally, we could report a large multi-* request before starting it, in case 
> it happens to be deadly. Knowing the client, user and how many rows are 
> affected would be a good start to tracking down painful users.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18209) Include httpclient / httpcore jars in build artifacts

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-18209:
---
Summary: Include httpclient / httpcore jars in build artifacts  (was: Make 
httpclient / httpcore compile time dependency)

> Include httpclient / httpcore jars in build artifacts
> -
>
> Key: HBASE-18209
> URL: https://issues.apache.org/jira/browse/HBASE-18209
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 18209.v1.txt, 18209.v2.txt
>
>
> We need httpclient & httpcore jars to be present when rootdir is placed on 
> s3(a).
> Attempts to move to the fully shaded amazon-SDK JAR caused problems of its 
> own. (according to [~steve_l])
> Here are the versions we should use:
> 4.5.2
> 4.4.4
> Currently they are declared test dependency.
> This JIRA is to move to compile time dependency so that the corresponding 
> jars are bundled in lib directory.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18209) Include httpclient / httpcore jars in build artifacts

[Josh Elser (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049904#comment-16049904
 ] 

Josh Elser commented on HBASE-18209:


Great! v2 looks good to me, Ted.

Please just verify the jars are actually in lib/ before you commit :)

> Include httpclient / httpcore jars in build artifacts
> -
>
> Key: HBASE-18209
> URL: https://issues.apache.org/jira/browse/HBASE-18209
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 18209.v1.txt, 18209.v2.txt
>
>
> We need httpclient & httpcore jars to be present when rootdir is placed on 
> s3(a).
> Attempts to move to the fully shaded amazon-SDK JAR caused problems of its 
> own. (according to [~steve_l])
> Here are the versions we should use:
> 4.5.2
> 4.4.4
> Currently they are declared test dependency.
> This JIRA is to move to compile time dependency so that the corresponding 
> jars are bundled in lib directory.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18219) Fix typo in constant HConstants.HBASE_CLIENT_MEAT_REPLICA_SCAN_TIMEOUT

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049893#comment-16049893
 ] 

Hudson commented on HBASE-18219:


FAILURE: Integrated in Jenkins build HBase-1.4 #776 (See 
[https://builds.apache.org/job/HBase-1.4/776/])
HBASE-18219 Fix typo in constant (apurtell: rev 
316e02e3d8b68fdfc6fdda59b1e79f1b67517964)
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestReplicaWithCluster.java
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionConfiguration.java
* (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java


> Fix typo in constant HConstants.HBASE_CLIENT_MEAT_REPLICA_SCAN_TIMEOUT
> --
>
> Key: HBASE-18219
> URL: https://issues.apache.org/jira/browse/HBASE-18219
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 3.0.0, 1.4.0
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Fix For: 2.0.0, 3.0.0, 1.4.0
>
> Attachments: HBASE-18219-branch-1.patch, HBASE-18219.patch
>
>
> HConstants has a constant HBASE_CLIENT_MEAT_REPLICA_SCAN_TIMEOUT that should 
> be HBASE_CLIENT_META_REPLICA_SCAN_TIMEOUT



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18023) Log multi-* requests for more than threshold number of rows

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049884#comment-16049884
 ] 

Hadoop QA commented on HBASE-18023:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
30s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
24s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 12 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
29m 14s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha3. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 49s 
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 140m 13s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
29s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 187m 40s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | 
org.apache.hadoop.hbase.coprocessor.TestCoprocessorMetrics |
|   | org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan2 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:757bf37 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12873039/HBASE-18023.master.001.patch
 |
| JIRA Issue | HBASE-18023 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  xml  |
| uname | Linux 3ba5a83b97d3 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 58b3777 |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| 

[jira] [Updated] (HBASE-18209) Make httpclient / httpcore compile time dependency

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-18209:
---
Attachment: 18209.v2.txt

> Make httpclient / httpcore compile time dependency
> --
>
> Key: HBASE-18209
> URL: https://issues.apache.org/jira/browse/HBASE-18209
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 18209.v1.txt, 18209.v2.txt
>
>
> We need httpclient & httpcore jars to be present when rootdir is placed on 
> s3(a).
> Attempts to move to the fully shaded amazon-SDK JAR caused problems of its 
> own. (according to [~steve_l])
> Here are the versions we should use:
> 4.5.2
> 4.4.4
> Currently they are declared test dependency.
> This JIRA is to move to compile time dependency so that the corresponding 
> jars are bundled in lib directory.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18209) Make httpclient / httpcore compile time dependency

[Josh Elser (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049846#comment-16049846
 ] 

Josh Elser commented on HBASE-18209:


bq. This JIRA is to move to compile time dependency so that the corresponding 
jars are bundled in lib directory.

If we just need to get these jars included in the tarball, we should update 
hbase-assembly/pom.xml.

With your current patch, this would create a situation where the 
maven-dependency-plugin's analyze phase would throw an error because 
hbase-server would depend on JARs that it doesn't actually need. Since we 
*know* that we need them at runtime, we should have hbase-assembly depend on 
these two artifacts and (if necessary, I don't think it would be) add them to 
{{hbase-assembly/src/main/assembly/hadoop-two-compat.xml}}.

So, we leave httpclient and httpcore as test-dependencies (as only 
hbase-server/src/test/java requires it, not hbase-server/src/main/java) and 
list the dependencies in hbase-assembly/pom.xml instead. Does this make sense? 
I can put up a patch if not :)

Aside, it looks like commons-httpclient was upgraded in HBASE-16267 which means 
that the version override (and comment) in hbase-server/pom.xml are 
unnecessary. The version element and the comment should be removed as well for 
org.apache.httpcomponents:httpclient.

> Make httpclient / httpcore compile time dependency
> --
>
> Key: HBASE-18209
> URL: https://issues.apache.org/jira/browse/HBASE-18209
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 18209.v1.txt
>
>
> We need httpclient & httpcore jars to be present when rootdir is placed on 
> s3(a).
> Attempts to move to the fully shaded amazon-SDK JAR caused problems of its 
> own. (according to [~steve_l])
> Here are the versions we should use:
> 4.5.2
> 4.4.4
> Currently they are declared test dependency.
> This JIRA is to move to compile time dependency so that the corresponding 
> jars are bundled in lib directory.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18219) Fix typo in constant HConstants.HBASE_CLIENT_MEAT_REPLICA_SCAN_TIMEOUT

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049835#comment-16049835
 ] 

Hadoop QA commented on HBASE-18219:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m 19s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 22s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
47s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 52s 
{color} | {color:green} branch-1 passed with JDK v1.8.0_131 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 12s 
{color} | {color:green} branch-1 passed with JDK v1.7.0_131 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 
14s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 
19s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 46s 
{color} | {color:red} hbase-server in branch-1 has 1 extant Findbugs warnings. 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 48s 
{color} | {color:red} hbase-server in branch-1 has 1 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 48s 
{color} | {color:green} branch-1 passed with JDK v1.8.0_131 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 18s 
{color} | {color:green} branch-1 passed with JDK v1.7.0_131 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s 
{color} | {color:green} the patch passed with JDK v1.8.0_131 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 14s 
{color} | {color:green} hbase-common-jdk1.8.0_131 with JDK v1.8.0_131 generated 
0 new + 18 unchanged - 18 fixed = 18 total (was 36) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s 
{color} | {color:green} hbase-client in the patch passed with JDK v1.8.0_131. 
{color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s 
{color} | {color:green} hbase-server-jdk1.8.0_131 with JDK v1.8.0_131 generated 
0 new + 5 unchanged - 5 fixed = 5 total (was 10) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 5s 
{color} | {color:green} the patch passed with JDK v1.7.0_131 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s 
{color} | {color:green} hbase-common-jdk1.7.0_131 with JDK v1.7.0_131 generated 
0 new + 18 unchanged - 18 fixed = 18 total (was 36) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s 
{color} | {color:green} hbase-client in the patch passed with JDK v1.7.0_131. 
{color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s 
{color} | {color:green} hbase-server-jdk1.7.0_131 with JDK v1.7.0_131 generated 
0 new + 5 unchanged - 5 fixed = 5 total (was 10) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
3s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
15m 4s {color} | {color:green} The patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 7s 
{color} | {color:green} the patch passed {color} |
| 

[jira] [Updated] (HBASE-18054) log when we add/remove failed servers in client

[Ali (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ali updated HBASE-18054:

Attachment: HBASE-18054.v4.master.patch

A throwable argument passed to specify which exception led to failure of 
server. New tests written to test this. Arguments to older tests that use 
addToFailedServers have been modified

> log when we add/remove failed servers in client
> ---
>
> Key: HBASE-18054
> URL: https://issues.apache.org/jira/browse/HBASE-18054
> Project: HBase
>  Issue Type: Bug
>  Components: Client, Operability
>Affects Versions: 1.3.0
>Reporter: Sean Busbey
>Assignee: Ali
> Attachments: HBASE-18054.patch, HBASE-18054.v2.master.patch, 
> HBASE-18054.v3.master.patch, HBASE-18054.v4.master.patch
>
>
> Currently we log if a server is in the failed server list when we go to 
> connect to it, but we don't log anything about when the server got into the 
> list.
> This means we have to search the log for errors involving the same server 
> name that (hopefully) managed to get into the log within 
> {{FAILED_SERVER_EXPIRY_KEY}} milliseconds earlier (default 2 seconds).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18212) In Standalone mode with local filesystem HBase logs Warning message:Failed to invoke 'unbuffer' method in class class org.apache.hadoop.fs.FSDataInputStream

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049789#comment-16049789
 ] 

Andrew Purtell commented on HBASE-18212:


This came in on HBASE-9393. Should we just eliminate this warning? 

> In Standalone mode with local filesystem HBase logs Warning message:Failed to 
> invoke 'unbuffer' method in class class org.apache.hadoop.fs.FSDataInputStream
> 
>
> Key: HBASE-18212
> URL: https://issues.apache.org/jira/browse/HBASE-18212
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Umesh Agashe
>
> New users may get nervous after seeing following warning level log messages 
> (considering new users will most likely run HBase in Standalone mode first):
> {code}
> WARN  [MemStoreFlusher.1] io.FSDataInputStreamWrapper: Failed to invoke 
> 'unbuffer' method in class class org.apache.hadoop.fs.FSDataInputStream . So 
> there may be a TCP socket connection left open in CLOSE_WAIT state.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18219) Fix typo in constant HConstants.HBASE_CLIENT_MEAT_REPLICA_SCAN_TIMEOUT

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-18219:
---
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Thanks [~enis]. Pushed to branch-1, branch-2, and master

> Fix typo in constant HConstants.HBASE_CLIENT_MEAT_REPLICA_SCAN_TIMEOUT
> --
>
> Key: HBASE-18219
> URL: https://issues.apache.org/jira/browse/HBASE-18219
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 3.0.0, 1.4.0
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Fix For: 2.0.0, 3.0.0, 1.4.0
>
> Attachments: HBASE-18219-branch-1.patch, HBASE-18219.patch
>
>
> HConstants has a constant HBASE_CLIENT_MEAT_REPLICA_SCAN_TIMEOUT that should 
> be HBASE_CLIENT_META_REPLICA_SCAN_TIMEOUT



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18218) List replication peers for the cluster

[Maddineni Sukumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049774#comment-16049774
 ] 

Maddineni Sukumar commented on HBASE-18218:
---

And also status like enabled/disabled.

> List replication peers for the cluster
> --
>
> Key: HBASE-18218
> URL: https://issues.apache.org/jira/browse/HBASE-18218
> Project: HBase
>  Issue Type: New Feature
>Reporter: Ali
>Assignee: Ali
> Attachments: HBASE-18218.v1.branch-1.2.patch, screenshot-1.png
>
>
> HBase Master page that listed all the replication peers for a cluster, with 
> their associated metadata



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18218) List replication peers for the cluster

[Vincent Poon (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049761#comment-16049761
 ] 

Vincent Poon commented on HBASE-18218:
--

[~aky] Can you include the peer configs as well?  
(ReplicationPeerConfig#getConfiguration)

> List replication peers for the cluster
> --
>
> Key: HBASE-18218
> URL: https://issues.apache.org/jira/browse/HBASE-18218
> Project: HBase
>  Issue Type: New Feature
>Reporter: Ali
>Assignee: Ali
> Attachments: HBASE-18218.v1.branch-1.2.patch, screenshot-1.png
>
>
> HBase Master page that listed all the replication peers for a cluster, with 
> their associated metadata



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18119) Improve HFile readability and modify ChecksumUtil log level

[Zach York (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049757#comment-16049757
 ] 

Zach York commented on HBASE-18119:
---

[~Qilin Cao] The code simplification makes sense to me. 
A few things:

Why change that message to trace? What is the rationale?

bq. +HFile.checkHFileVersion(conf);
bq. +conf.setInt(HFile.FORMAT_VERSION_KEY, HFile.MIN_FORMAT_VERSION);
bq. +HFile.checkHFileVersion(conf);

This test could pass erroneously. If the first checkHFileVersion threw an 
IllegalArgumentException, the test would still pass even though this would be 
incorrect. Perhaps split it into two tests (1 that shows it works normally and 
another one that has the exception thrown).



> Improve HFile readability and modify ChecksumUtil log level
> ---
>
> Key: HBASE-18119
> URL: https://issues.apache.org/jira/browse/HBASE-18119
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Qilin Cao
>Assignee: Qilin Cao
>Priority: Minor
> Attachments: HBASE-18119-v1.patch
>
>
> It is confused when I read the HFile.checkHFileVersion method, so I change 
> the if expression. Simultaneously, I change ChecksumUtil the info log level 
> to trace.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18219) Fix typo in constant HConstants.HBASE_CLIENT_MEAT_REPLICA_SCAN_TIMEOUT

[Enis Soztutar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049738#comment-16049738
 ] 

Enis Soztutar commented on HBASE-18219:
---

+1. Sad to see meat replicas go. 

> Fix typo in constant HConstants.HBASE_CLIENT_MEAT_REPLICA_SCAN_TIMEOUT
> --
>
> Key: HBASE-18219
> URL: https://issues.apache.org/jira/browse/HBASE-18219
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 3.0.0, 1.4.0
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Fix For: 2.0.0, 3.0.0, 1.4.0
>
> Attachments: HBASE-18219-branch-1.patch, HBASE-18219.patch
>
>
> HConstants has a constant HBASE_CLIENT_MEAT_REPLICA_SCAN_TIMEOUT that should 
> be HBASE_CLIENT_META_REPLICA_SCAN_TIMEOUT



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18023) Log multi-* requests for more than threshold number of rows

[Josh Elser (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser updated HBASE-18023:
---
Status: Patch Available  (was: Open)

Thanks for the patch. I've made you the assignee and marked this as Patch 
Available to get a QA report.

> Log multi-* requests for more than threshold number of rows
> ---
>
> Key: HBASE-18023
> URL: https://issues.apache.org/jira/browse/HBASE-18023
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: Clay B.
>Assignee: David Harju
>Priority: Minor
> Attachments: HBASE-18023.master.001.patch
>
>
> Today, if a user happens to do something like a large multi-put, they can get 
> through request throttling (e.g. it is one request) but still crash a region 
> server with a garbage storm. We have seen regionservers hit this issue and it 
> is silent and deadly. The RS will report nothing more than a mysterious 
> garbage collection and exit out.
> Ideally, we could report a large multi-* request before starting it, in case 
> it happens to be deadly. Knowing the client, user and how many rows are 
> affected would be a good start to tracking down painful users.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HBASE-18023) Log multi-* requests for more than threshold number of rows

[Josh Elser (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser reassigned HBASE-18023:
--

Assignee: David Harju  (was: Josh Elser)

> Log multi-* requests for more than threshold number of rows
> ---
>
> Key: HBASE-18023
> URL: https://issues.apache.org/jira/browse/HBASE-18023
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: Clay B.
>Assignee: David Harju
>Priority: Minor
> Attachments: HBASE-18023.master.001.patch
>
>
> Today, if a user happens to do something like a large multi-put, they can get 
> through request throttling (e.g. it is one request) but still crash a region 
> server with a garbage storm. We have seen regionservers hit this issue and it 
> is silent and deadly. The RS will report nothing more than a mysterious 
> garbage collection and exit out.
> Ideally, we could report a large multi-* request before starting it, in case 
> it happens to be deadly. Knowing the client, user and how many rows are 
> affected would be a good start to tracking down painful users.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18023) Log multi-* requests for more than threshold number of rows

[David Harju (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Harju updated HBASE-18023:

Attachment: HBASE-18023.master.001.patch

Added log warning with default configuration thresholds and unit tests

> Log multi-* requests for more than threshold number of rows
> ---
>
> Key: HBASE-18023
> URL: https://issues.apache.org/jira/browse/HBASE-18023
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: Clay B.
>Assignee: Josh Elser
>Priority: Minor
> Attachments: HBASE-18023.master.001.patch
>
>
> Today, if a user happens to do something like a large multi-put, they can get 
> through request throttling (e.g. it is one request) but still crash a region 
> server with a garbage storm. We have seen regionservers hit this issue and it 
> is silent and deadly. The RS will report nothing more than a mysterious 
> garbage collection and exit out.
> Ideally, we could report a large multi-* request before starting it, in case 
> it happens to be deadly. Knowing the client, user and how many rows are 
> affected would be a good start to tracking down painful users.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18218) List replication peers for the cluster

[Ali (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ali updated HBASE-18218:

Status: Patch Available  (was: Open)

> List replication peers for the cluster
> --
>
> Key: HBASE-18218
> URL: https://issues.apache.org/jira/browse/HBASE-18218
> Project: HBase
>  Issue Type: New Feature
>Reporter: Ali
>Assignee: Ali
> Attachments: HBASE-18218.v1.branch-1.2.patch, screenshot-1.png
>
>
> HBase Master page that listed all the replication peers for a cluster, with 
> their associated metadata



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18218) List replication peers for the cluster

[Ali (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ali updated HBASE-18218:

Attachment: screenshot-1.png

> List replication peers for the cluster
> --
>
> Key: HBASE-18218
> URL: https://issues.apache.org/jira/browse/HBASE-18218
> Project: HBase
>  Issue Type: New Feature
>Reporter: Ali
>Assignee: Ali
> Attachments: HBASE-18218.v1.branch-1.2.patch, screenshot-1.png
>
>
> HBase Master page that listed all the replication peers for a cluster, with 
> their associated metadata



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18218) List replication peers for the cluster

[Ali (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049692#comment-16049692
 ] 

Ali commented on HBASE-18218:
-

screenshot attached

> List replication peers for the cluster
> --
>
> Key: HBASE-18218
> URL: https://issues.apache.org/jira/browse/HBASE-18218
> Project: HBase
>  Issue Type: New Feature
>Reporter: Ali
>Assignee: Ali
> Attachments: HBASE-18218.v1.branch-1.2.patch
>
>
> HBase Master page that listed all the replication peers for a cluster, with 
> their associated metadata



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18218) List replication peers for the cluster

[Ali (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ali updated HBASE-18218:

Attachment: HBASE-18218.v1.branch-1.2.patch

Patch includes changes made to front end only, therefore no unit tests were 
written.

> List replication peers for the cluster
> --
>
> Key: HBASE-18218
> URL: https://issues.apache.org/jira/browse/HBASE-18218
> Project: HBase
>  Issue Type: New Feature
>Reporter: Ali
>Assignee: Ali
> Attachments: HBASE-18218.v1.branch-1.2.patch
>
>
> HBase Master page that listed all the replication peers for a cluster, with 
> their associated metadata



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18219) Fix typo in constant HConstants.HBASE_CLIENT_MEAT_REPLICA_SCAN_TIMEOUT

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-18219:
---
Attachment: HBASE-18219-branch-1.patch
HBASE-18219.patch

Trivial patches. Master patch applies to branch-2 too. 

> Fix typo in constant HConstants.HBASE_CLIENT_MEAT_REPLICA_SCAN_TIMEOUT
> --
>
> Key: HBASE-18219
> URL: https://issues.apache.org/jira/browse/HBASE-18219
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 3.0.0, 1.4.0
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Fix For: 2.0.0, 3.0.0, 1.4.0
>
> Attachments: HBASE-18219-branch-1.patch, HBASE-18219.patch
>
>
> HConstants has a constant HBASE_CLIENT_MEAT_REPLICA_SCAN_TIMEOUT that should 
> be HBASE_CLIENT_META_REPLICA_SCAN_TIMEOUT



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18219) Fix typo in constant HConstants.HBASE_CLIENT_MEAT_REPLICA_SCAN_TIMEOUT

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-18219:
---
Status: Patch Available  (was: Open)

> Fix typo in constant HConstants.HBASE_CLIENT_MEAT_REPLICA_SCAN_TIMEOUT
> --
>
> Key: HBASE-18219
> URL: https://issues.apache.org/jira/browse/HBASE-18219
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 3.0.0, 1.4.0
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Fix For: 2.0.0, 3.0.0, 1.4.0
>
> Attachments: HBASE-18219-branch-1.patch, HBASE-18219.patch
>
>
> HConstants has a constant HBASE_CLIENT_MEAT_REPLICA_SCAN_TIMEOUT that should 
> be HBASE_CLIENT_META_REPLICA_SCAN_TIMEOUT



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18219) Fix typo in constant HConstants.HBASE_CLIENT_MEAT_REPLICA_SCAN_TIMEOUT

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049675#comment-16049675
 ] 

Andrew Purtell commented on HBASE-18219:


Pretty sure we call 'meat replicas' by the industry term DBA :-)

> Fix typo in constant HConstants.HBASE_CLIENT_MEAT_REPLICA_SCAN_TIMEOUT
> --
>
> Key: HBASE-18219
> URL: https://issues.apache.org/jira/browse/HBASE-18219
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 3.0.0, 1.4.0
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Fix For: 2.0.0, 3.0.0, 1.4.0
>
>
> HConstants has a constant HBASE_CLIENT_MEAT_REPLICA_SCAN_TIMEOUT that should 
> be HBASE_CLIENT_META_REPLICA_SCAN_TIMEOUT



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-18219) Fix typo in constant HConstants.HBASE_CLIENT_MEAT_REPLICA_SCAN_TIMEOUT

[Andrew Purtell (JIRA)]

Andrew Purtell created HBASE-18219:
--

 Summary: Fix typo in constant 
HConstants.HBASE_CLIENT_MEAT_REPLICA_SCAN_TIMEOUT
 Key: HBASE-18219
 URL: https://issues.apache.org/jira/browse/HBASE-18219
 Project: HBase
  Issue Type: Bug
Affects Versions: 2.0.0, 3.0.0, 1.4.0
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Minor
 Fix For: 2.0.0, 3.0.0, 1.4.0


HConstants has a constant HBASE_CLIENT_MEAT_REPLICA_SCAN_TIMEOUT that should be 
HBASE_CLIENT_META_REPLICA_SCAN_TIMEOUT



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18214) Replace the folly::AtomicHashMap usage in the RPC layer

[Enis Soztutar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049607#comment-16049607
 ] 

Enis Soztutar commented on HBASE-18214:
---

Indeed, this fails with 
{code}
Exception AtomicHashMap is full. 
{code}

If we use std::unordered_map, we need to use a mutex as well. One other option 
is to bring in the TBB library as a dependency. I would say that let's do the 
mutex + unordered_map for now, and worry about optimizing this later if we 
measure that this part is a bottleneck. 

> Replace the folly::AtomicHashMap usage in the RPC layer
> ---
>
> Key: HBASE-18214
> URL: https://issues.apache.org/jira/browse/HBASE-18214
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Devaraj Das
>Assignee: Devaraj Das
>
> In my tests, I saw that folly::AtomicHashMap usage is not appropriate for 
> one, rather common use case. It'd become sort of unusable (inserts would 
> hang) after a bunch of inserts and erases. This hashmap is used to keep track 
> of call-Id after a connection is set up in the RPC layer (insert a 
> call-id/msg pair when an RPC is sent, and erase the pair when the 
> corresponding response is received). Here is a simple program that will 
> demonstrate the issue:
> {code}
> folly::AtomicHashMap f(100);
> int i = 0;
> while (i < 1) {
> try {
>   f.insert(i,100);
>   LOG(INFO) << "Inserted " << i << "  " << f.size();
>   f.erase(i);
>   LOG(INFO) << "Deleted " << i << "  " << f.size();
>   i++;
> } catch (const std::exception ) {
>   LOG(INFO) << "Exception " << e.what();
>   break;
> }
> }
> {code}
> After poking around a little bit, it is indeed called out as a limitation 
> here 
> https://github.com/facebook/folly/blob/master/folly/docs/AtomicHashMap.md 
> (grep for 'erase'). Proposal is to replace this with something that will fit 
> in in the above usecase (thinking of using std::unordered_map).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (HBASE-18105) [AMv2] Split/Merge need cleanup; currently they diverge and do not fully embrace AMv2 world

[Yi Liang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049446#comment-16049446
 ] 

Yi Liang edited comment on HBASE-18105 at 6/14/17 8:05 PM:
---

Hi [~stack],
   I have a problem now, in the current split logic, if the split rowkey = 
null, the split will find the best split points for that region, and split it. 
However this logic does not exist in SplitProcedure.  I think we need to 
support this in splitProcedure. I have a idea to solve this problem, which is 
add a property called bestSplitPoints in HRegionInfo, since splitProcedure 
always send a rpc request getRegionInfoRequest to RSrpcservice to check the 
region is splitable or not. We can get this bestSplitPoints from this rpc 
request as well, or do you have other good idea?   

or We can have another define another RPC call to get bestSplitPoints from rs

  in the current split logic, it is easy  to get the best split points, since 
they happen at RSRpcServices.splitRegion(see path1 for split), which is in rs 
side, they can get region server and region info easily. 


was (Author: easyliangjob):
Hi [~stack],
   I have a problem now, in the current split logic, if the split rowkey = 
null, the split will find the best split points for that region, and split it. 
However this logic does not exist in SplitProcedure.  I think we need to 
support this in splitProcedure. I have a idea to solve this problem, which is 
add a property called bestSplitPoints in HRegionInfo, since splitProcedure 
always send a rpc request getRegionInfoRequest to RSrpcservice to check the 
region is splitable or not. We can get this bestSplitPoints from this rpc 
request as well, or do you have other good idea?   

  in the current split logic, it is easy  to get the best split points, since 
they happen at RSRpcServices.splitRegion(see path1 for split), which is in rs 
side, they can get region server and region info easily. 

> [AMv2] Split/Merge need cleanup; currently they diverge and do not fully 
> embrace AMv2 world
> ---
>
> Key: HBASE-18105
> URL: https://issues.apache.org/jira/browse/HBASE-18105
> Project: HBase
>  Issue Type: Sub-task
>  Components: Region Assignment
>Affects Versions: 2.0.0
>Reporter: stack
> Fix For: 2.0.0
>
>
> Region Split and Merge work on the new AMv2 but they work differently. This 
> issue is about bringing them back together and fully embracing the AMv2 
> program.
> They both have issues mostly the fact that they carry around baggage no 
> longer necessary in the new world of assignment.
> Here are some of the items:
> Split and Merge metadata modifications are done by the Master now but we have 
> vestige of Split/Merge on RS still; e.g. when we SPLIT, we ask the Master 
> which asks the RS, which turns around, and asks the Master to run the 
> operation. Fun. MERGE is all done Master-side.
>  
> Clean this up. Remove asking RS to run SPLIT and remove RegionMergeRequest, 
> etc. on RS-side. Also remove PONR. We don’t Points-Of-No-Return now we are up 
> on Pv2. Remove all calls in Interfaces; they are unused. Make RS still able 
> to detect when split, but have it be a client of Master like anyone else.
> Split is Async but does not return procId
> Split is async. Doesn’t return the procId though. Merge does. Fix. Only hard 
> part here I think is the Admin API does not allow procid return.
> Flags
> Currently OFFLINE is determined by looking either at the master instance of 
> HTD (isOffline) and/or at the RegionState#state. Ditto for SPLIT. For MERGE, 
> we rely on RegionState#state. Related is a note above on how split works -- 
> there is a split flag in HTD when there should not be.
>  
> TODO is move to rely on RegionState#state exclusively in Master.
> From Split/Merge Procedures need finishing in 
> https://docs.google.com/document/d/1eVKa7FHdeoJ1-9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#heading=h.4b60dc1h4m1f



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18178) [C++] Retrying meta location lookup and zookeeper connection

[Enis Soztutar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049598#comment-16049598
 ] 

Enis Soztutar commented on HBASE-18178:
---

bq. Is the above going to be done in next patch or new JIRA ?
We cannot do this in this jira. Will be follow up jira. 
bq. Why is the retry number increased ?
Because retry number is 35 in the Java side. It should not have been 31. 
bq. Change the log level. (There're other LOG(INFO)'s in the patch)
Log level is correct. It is intentionally INFO. 
bq. Is the return statement effective ?
No it is not. However, we need it because of the static assertion for the 
folly::Future template. 

> [C++] Retrying meta location lookup and zookeeper connection 
> -
>
> Key: HBASE-18178
> URL: https://issues.apache.org/jira/browse/HBASE-18178
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Attachments: hbase-18178-v1.patch
>
>
> Currently location-cache can only do a single lookup to meta. If meta 
> location changes or we have zookeeper connection problems, we never retry. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18188) [C++] Fix Handling do not retry exceptions

[Enis Soztutar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Enis Soztutar updated HBASE-18188:
--
Attachment: hbase-18188_v2.patch

v2 patch. 

> [C++] Fix Handling do not retry exceptions
> --
>
> Key: HBASE-18188
> URL: https://issues.apache.org/jira/browse/HBASE-18188
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Attachments: hbase-18188_v1.patch, hbase-18188_v2.patch
>
>
> Needed for HBASE-18061 and others. 
> Java client relies on the exception hierarchy for DoNotRetryExceptions, which 
> there is 40+ subtypes. The exceptions from the server side are rethrown in 
> the client side (ConnectionUtils.translateException, etc) and the rest of the 
> code deals with do-not-retry information by catching DoNotRetryIOException's 
> (thus relying on exception hierarchy). 
> This of course does not work on the C++ side, since we lack the info for the 
> java class types. In case the exception happens in the RPC response, the 
> server puts the do_not_retry information as a field in PB (see 
> ExceptionResponse::do_not_retry PB message). However, in other settings, we 
> just serialize the exception without do_not_retry information (see 
> ResultOrException PB message). In some other settings, we can raise 
> exceptions from the client-side (for example when table cannot be found in 
> meta). 
> We need a strategy to handle do-not-retry information uniformly, no matter 
> they are coming from client side or server side. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18023) Log multi-* requests for more than threshold number of rows

[David Harju (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049542#comment-16049542
 ] 

David Harju commented on HBASE-18023:
-

Thanks [~stack] and [~clayb]!

 I'll go ahead and make the threshold 1k, and unless there's a better example 
I'll add a link to this issue 
(https://issues.apache.org/jira/browse/HBASE-18023) in the log line.

> Log multi-* requests for more than threshold number of rows
> ---
>
> Key: HBASE-18023
> URL: https://issues.apache.org/jira/browse/HBASE-18023
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: Clay B.
>Assignee: Josh Elser
>Priority: Minor
>
> Today, if a user happens to do something like a large multi-put, they can get 
> through request throttling (e.g. it is one request) but still crash a region 
> server with a garbage storm. We have seen regionservers hit this issue and it 
> is silent and deadly. The RS will report nothing more than a mysterious 
> garbage collection and exit out.
> Ideally, we could report a large multi-* request before starting it, in case 
> it happens to be deadly. Knowing the client, user and how many rows are 
> affected would be a good start to tracking down painful users.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-17988) get-active-master.rb and draining_servers.rb no longer work

[Chinmay Kulkarni (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049522#comment-16049522
 ] 

Chinmay Kulkarni commented on HBASE-17988:
--

I have changed the fix versions. If everything looks good, can we merge this 
patch into _master_?

> get-active-master.rb and draining_servers.rb no longer work
> ---
>
> Key: HBASE-17988
> URL: https://issues.apache.org/jira/browse/HBASE-17988
> Project: HBase
>  Issue Type: Bug
>  Components: scripts
>Affects Versions: 2.0.0
>Reporter: Mike Drob
>Assignee: Chinmay Kulkarni
>Priority: Critical
> Fix For: 2.0.0, 3.0.0
>
> Attachments: HBASE-17988.002.patch, HBASE-17988.patch
>
>
> The scripts {{bin/get-active-master.rb}} and {{bin/draining_servers.rb}} no 
> longer work on current master branch. Here is an example error message:
> {noformat}
> $ bin/hbase-jruby bin/get-active-master.rb 
> NoMethodError: undefined method `masterAddressZNode' for 
> #
>at bin/get-active-master.rb:35
> {noformat}
> My initial probing suggests that this is likely due to movement that happened 
> in HBASE-16690. Perhaps instead of reworking the ruby, there is similar Java 
> functionality already existing somewhere.
> Putting priority at critical since it's impossible to know whether users rely 
> on the scripts.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (HBASE-18105) [AMv2] Split/Merge need cleanup; currently they diverge and do not fully embrace AMv2 world

[Yi Liang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049446#comment-16049446
 ] 

Yi Liang edited comment on HBASE-18105 at 6/14/17 6:03 PM:
---

Hi [~stack],
   I have a problem now, in the current split logic, if the split rowkey = 
null, the split will find the best split points for that region, and split it. 
However this logic does not exist in SplitProcedure.  I think we need to 
support this in splitProcedure. I have a idea to solve this problem, which is 
add a property called bestSplitPoints in HRegionInfo, since splitProcedure 
always send a rpc request getRegionInfoRequest to RSrpcservice to check the 
region is splitable or not. We can get this bestSplitPoints from this rpc 
request as well, or do you have other good idea?   

  in the current split logic, it is easy  to get the best split points, since 
they happen at RSRpcServices.splitRegion(see path1 for split), which is in rs 
side, they can get region server and region info easily. 


was (Author: easyliangjob):
Hi [~stack],
   I have a problem now, in the current split logic, if the split rowkey = 
null, the split will find the best split points for that region, and split it. 
However this logic does not exist in SplitProcedure.  I think we need to 
support this in splitProcedure. I have a idea to solve this problem, which is 
add a property called bestSplitPoints in HRegionInfo, since splitProcedure 
always send a rpc request to RSrpcservice to check the region is splitable or 
not. We can get this bestSplitPoints from this rpc request as well, or do you 
have other good idea?   

  in the current split logic, it is easy  to get the best split points, since 
they happen at RSRpcServices.splitRegion(see path1 for split), which is in rs 
side, they can get region server and region info easily. 

> [AMv2] Split/Merge need cleanup; currently they diverge and do not fully 
> embrace AMv2 world
> ---
>
> Key: HBASE-18105
> URL: https://issues.apache.org/jira/browse/HBASE-18105
> Project: HBase
>  Issue Type: Sub-task
>  Components: Region Assignment
>Affects Versions: 2.0.0
>Reporter: stack
> Fix For: 2.0.0
>
>
> Region Split and Merge work on the new AMv2 but they work differently. This 
> issue is about bringing them back together and fully embracing the AMv2 
> program.
> They both have issues mostly the fact that they carry around baggage no 
> longer necessary in the new world of assignment.
> Here are some of the items:
> Split and Merge metadata modifications are done by the Master now but we have 
> vestige of Split/Merge on RS still; e.g. when we SPLIT, we ask the Master 
> which asks the RS, which turns around, and asks the Master to run the 
> operation. Fun. MERGE is all done Master-side.
>  
> Clean this up. Remove asking RS to run SPLIT and remove RegionMergeRequest, 
> etc. on RS-side. Also remove PONR. We don’t Points-Of-No-Return now we are up 
> on Pv2. Remove all calls in Interfaces; they are unused. Make RS still able 
> to detect when split, but have it be a client of Master like anyone else.
> Split is Async but does not return procId
> Split is async. Doesn’t return the procId though. Merge does. Fix. Only hard 
> part here I think is the Admin API does not allow procid return.
> Flags
> Currently OFFLINE is determined by looking either at the master instance of 
> HTD (isOffline) and/or at the RegionState#state. Ditto for SPLIT. For MERGE, 
> we rely on RegionState#state. Related is a note above on how split works -- 
> there is a split flag in HTD when there should not be.
>  
> TODO is move to rely on RegionState#state exclusively in Master.
> From Split/Merge Procedures need finishing in 
> https://docs.google.com/document/d/1eVKa7FHdeoJ1-9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#heading=h.4b60dc1h4m1f



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18105) [AMv2] Split/Merge need cleanup; currently they diverge and do not fully embrace AMv2 world

[Yi Liang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049446#comment-16049446
 ] 

Yi Liang commented on HBASE-18105:
--

Hi [~stack],
   I have a problem now, in the current split logic, if the split rowkey = 
null, the split will find the best split points for that region, and split it. 
However this logic does not exist in SplitProcedure.  I think we need to 
support this in splitProcedure. I have a idea to solve this problem, which is 
add a property called bestSplitPoints in HRegionInfo, since splitProcedure 
always send a rpc request to RSrpcservice to check the region is splitable or 
not. We can get this bestSplitPoints from this rpc request as well, or do you 
have other good idea?   

  in the current split logic, it is easy  to get the best split points, since 
they happen at RSRpcServices.splitRegion(see path1 for split), which is in rs 
side, they can get region server and region info easily. 

> [AMv2] Split/Merge need cleanup; currently they diverge and do not fully 
> embrace AMv2 world
> ---
>
> Key: HBASE-18105
> URL: https://issues.apache.org/jira/browse/HBASE-18105
> Project: HBase
>  Issue Type: Sub-task
>  Components: Region Assignment
>Affects Versions: 2.0.0
>Reporter: stack
> Fix For: 2.0.0
>
>
> Region Split and Merge work on the new AMv2 but they work differently. This 
> issue is about bringing them back together and fully embracing the AMv2 
> program.
> They both have issues mostly the fact that they carry around baggage no 
> longer necessary in the new world of assignment.
> Here are some of the items:
> Split and Merge metadata modifications are done by the Master now but we have 
> vestige of Split/Merge on RS still; e.g. when we SPLIT, we ask the Master 
> which asks the RS, which turns around, and asks the Master to run the 
> operation. Fun. MERGE is all done Master-side.
>  
> Clean this up. Remove asking RS to run SPLIT and remove RegionMergeRequest, 
> etc. on RS-side. Also remove PONR. We don’t Points-Of-No-Return now we are up 
> on Pv2. Remove all calls in Interfaces; they are unused. Make RS still able 
> to detect when split, but have it be a client of Master like anyone else.
> Split is Async but does not return procId
> Split is async. Doesn’t return the procId though. Merge does. Fix. Only hard 
> part here I think is the Admin API does not allow procid return.
> Flags
> Currently OFFLINE is determined by looking either at the master instance of 
> HTD (isOffline) and/or at the RegionState#state. Ditto for SPLIT. For MERGE, 
> we rely on RegionState#state. Related is a note above on how split works -- 
> there is a split flag in HTD when there should not be.
>  
> TODO is move to rely on RegionState#state exclusively in Master.
> From Split/Merge Procedures need finishing in 
> https://docs.google.com/document/d/1eVKa7FHdeoJ1-9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#heading=h.4b60dc1h4m1f



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18218) List replication peers for the cluster

[Ali (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ali updated HBASE-18218:

Description: HBase Master page that listed all the replication peers for a 
cluster, with their associated metadata

> List replication peers for the cluster
> --
>
> Key: HBASE-18218
> URL: https://issues.apache.org/jira/browse/HBASE-18218
> Project: HBase
>  Issue Type: New Feature
>Reporter: Ali
>Assignee: Ali
>
> HBase Master page that listed all the replication peers for a cluster, with 
> their associated metadata



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18218) List replication peers for the cluster

[Ali (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ali updated HBASE-18218:

Summary: List replication peers for the cluster  (was: HBase Master page 
that listed all the replication peers for a cluster, with their associated 
metadata)

> List replication peers for the cluster
> --
>
> Key: HBASE-18218
> URL: https://issues.apache.org/jira/browse/HBASE-18218
> Project: HBase
>  Issue Type: New Feature
>Reporter: Ali
>Assignee: Ali
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-18218) HBase Master page that listed all the replication peers for a cluster, with their associated metadata

[Ali (JIRA)]

Ali created HBASE-18218:
---

 Summary: HBase Master page that listed all the replication peers 
for a cluster, with their associated metadata
 Key: HBASE-18218
 URL: https://issues.apache.org/jira/browse/HBASE-18218
 Project: HBase
  Issue Type: New Feature
Reporter: Ali
Assignee: Ali






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18180) Possible connection leak while closing BufferedMutator in TableOutputFormat

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049389#comment-16049389
 ] 

Hadoop QA commented on HBASE-18180:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 31s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
15s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s 
{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
32s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
23s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m 56s 
{color} | {color:red} hbase-server in branch-1 has 1 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s 
{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 5s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 5s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
29m 1s {color} | {color:green} The patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 198m 42s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
23s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 250m 5s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.balancer.TestStochasticLoadBalancer2 
|
|   | hadoop.hbase.regionserver.TestRSKilledWhenInitializing |
|   | hadoop.hbase.snapshot.TestExportSnapshot |
|   | hadoop.hbase.TestInfoServers |
|   | hadoop.hbase.master.TestMasterBalanceThrottling |
| Timed out junit tests | 
org.apache.hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpoint
 |
|   | org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot |
|   | org.apache.hadoop.hbase.filter.TestFuzzyRowFilterEndToEnd |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:395d9a0 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12872915/HBASE-18180-branch-1.patch
 |
| JIRA Issue | HBASE-18180 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux b44fa81f16bc 4.8.3-std-1 #1 SMP Fri Oct 21 11:15:43 UTC 2016 
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/hbase.sh |
| git revision | branch-1 / 256fc63 |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| findbugs | 
https://builds.apache.org/job/PreCommit-HBASE-Build/7183/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html
 |
| unit | 

[jira] [Commented] (HBASE-18128) compaction marker could be skipped

[Allan Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049240#comment-16049240
 ] 

Allan Yang commented on HBASE-18128:


Have some chat with [~tedyu]. Ignoring the seqID of compaction marker will 
resulting all compaction marker in the log being replayed
1. Only keep compaction marker with seqid smaller than flushed, but may drop 
the latest compaction marker to replay (very rare case, not a problem since if 
happens, the only result is that some redundant files in the region)
2. keep all compaction marker in the hlogs, writing many useless compaction 
marker entries to the recovered.edits and replay (not a problem either, since 
replay compaction is idempotent)
I'd prefer the first one.

> compaction marker could be skipped 
> ---
>
> Key: HBASE-18128
> URL: https://issues.apache.org/jira/browse/HBASE-18128
> Project: HBase
>  Issue Type: Improvement
>  Components: Compaction, regionserver
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
> Attachments: HBASE-18128-master.patch, HBASE-18128-master-v2.patch, 
> HBASE-18128-master-v3.patch, TestCompactionMarker.java
>
>
> The sequence for a compaction are as follows:
> 1. Compaction writes new files under region/.tmp directory (compaction output)
> 2. Compaction atomically moves the temporary file under region directory
> 3. Compaction appends a WAL edit containing the compaction input and output 
> files. Forces sync on WAL.
> 4. Compaction deletes the input files from the region directory.
> But if a flush happened between 3 and 4, then the regionserver crushed. The 
> compaction marker will be skipped when splitting log because the sequence id 
> of compaction marker is smaller than lastFlushedSequenceId.
> {code}
> if (lastFlushedSequenceId >= entry.getKey().getLogSeqNum()) {
>   editsSkipped++;
>   continue;
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Encryption of exisiting data in Stripe Compaction

[Karthick Ram]

We have a table which has time series data with Stripe Compaction enabled.
After encryption has been enabled for this table the newer entries are
encrypted and inserted. However to encrypt the existing data in the table,
a major compaction has to run. Since, stripe compaction doesn't allow a
major compaction to run, we are unable to encrypt the previous data. Please
suggest some ways to rectify this problem.

Regards,
Karthick R

[jira] [Commented] (HBASE-18010) Connect CellChunkMap to be used for flattening in CompactingMemStore

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049204#comment-16049204
 ] 

Anastasia Braginsky commented on HBASE-18010:
-

Hi [~stack], [~anoop.hbase], [~ram_krish],

I am publishing a big patch including integration of the CellChunkMap, without 
supporting cells bigger than chunks. (https://reviews.apache.org/r/60083/)
Those are changes in the patch:
1. Adding ability to flatten, merge and compact into CellChunkMap
2. Making flattening a creational pattern
3. Adding hirerachy of immutable chunks
4. Adding the process of "strengthening" of the pointers in the ChunkCreator's 
mapping
5. Updating test

Please start reviewing, may be take it part by part...
Thanks!
Anastasia

> Connect CellChunkMap to be used for flattening in CompactingMemStore
> 
>
> Key: HBASE-18010
> URL: https://issues.apache.org/jira/browse/HBASE-18010
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>
> The CellChunkMap helps to create a new type of ImmutableSegment, where the 
> index (CellSet's delegatee) is going to be CellChunkMap. No big cells or 
> upserted cells are going to be supported here.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18089) TestScannerHeartbeatMessages fails in branch-1

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049143#comment-16049143
 ] 

Ted Yu commented on HBASE-18089:


When I ran the test locally, chance of failure was high.

> TestScannerHeartbeatMessages fails in branch-1
> --
>
> Key: HBASE-18089
> URL: https://issues.apache.org/jira/browse/HBASE-18089
> Project: HBase
>  Issue Type: Test
>Reporter: Ted Yu
>Assignee: Xiang Li
> Attachments: test-heartbeat-6860.out
>
>
> From 
> https://builds.apache.org/job/PreCommit-HBASE-Build/6860/artifact/patchprocess/patch-unit-hbase-server.txt
>  :
> {code}
> testScannerHeartbeatMessages(org.apache.hadoop.hbase.regionserver.TestScannerHeartbeatMessages)
>   Time elapsed: 2.376 sec  <<< FAILURE!
> java.lang.AssertionError: Heartbeats messages are disabled, an exception 
> should be thrown. If an exception  is not thrown, the test case is not 
> testing the importance of heartbeat messages
>   at org.junit.Assert.fail(Assert.java:88)
>   at 
> org.apache.hadoop.hbase.regionserver.TestScannerHeartbeatMessages.testImportanceOfHeartbeats(TestScannerHeartbeatMessages.java:237)
>   at 
> org.apache.hadoop.hbase.regionserver.TestScannerHeartbeatMessages.testScannerHeartbeatMessages(TestScannerHeartbeatMessages.java:207)
> {code}
> Similar test failure can be observed in 
> https://builds.apache.org/job/PreCommit-HBASE-Build/6852/artifact/patchprocess/patch-unit-hbase-server.txt



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (HBASE-18217) Correct comment errors in TestScannerHeartbeatMessages

[Xiang Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049140#comment-16049140
 ] 

Xiang Li edited comment on HBASE-18217 at 6/14/17 12:51 PM:


[~yangzhe1991], I need your help to confirm if the comment of "Time limit will 
be 500 ms." is not correct, as it was updated by HBASE-16051. I am a little 
confused why the patch of HBASE-16051 changes CLIENT_TIMEOUT to 1000 and also 
add the comment that "Time limit will be 500 ms.", seem not make sense.

Also, could you please explain a little more about the following comments on 
DEFAULT_ROW_SLEEP_TIME
{code}
  // So set this to 200, we will get 3 rows and reach time limit at the start 
of 4th row, then sleep
  // for the 4th time. Total time is 800 ms so we will not timeout.
{code}
you mentioned that it reaches "time limit" at the start of 4th row, it 
conflicts with the last sentence saying that "Total time is 800 ms so we will 
{color:red}not{color} timeout". If it reaches time limit, it will time out. 
Please correct me if I did not get it right. When setting CLIENT_TIMEOUT to 
1000, does it reach time limit at the start of 4th row (e.g. 600ms). Did you 
write those when you perhaps assumed that CLIENT_TIMEOUT was 500?



was (Author: water):
@phil yang, I need your help to confirm if the comment of "Time limit will be 
500 ms." is not correct, as it was updated by HBASE-16051. I am a little 
confused why the patch of HBASE-16051 changes CLIENT_TIMEOUT to 1000 and also 
add the comment that "Time limit will be 500 ms.", seem not make sense.

Also, could you please explain a little more about the following comments on 
DEFAULT_ROW_SLEEP_TIME
{code}
  // So set this to 200, we will get 3 rows and reach time limit at the start 
of 4th row, then sleep
  // for the 4th time. Total time is 800 ms so we will not timeout.
{code}
you mentioned that it reaches "time limit" at the start of 4th row, it 
conflicts with the last sentence saying that "Total time is 800 ms so we will 
{color:red}not{color} timeout". If it reaches time limit, it will time out. 
Please correct me if I did not get it right. When setting CLIENT_TIMEOUT to 
1000, does it reach time limit at the start of 4th row (e.g. 600ms). Did you 
write those when you perhaps assumed that CLIENT_TIMEOUT was 500?


> Correct comment errors in TestScannerHeartbeatMessages
> --
>
> Key: HBASE-18217
> URL: https://issues.apache.org/jira/browse/HBASE-18217
> Project: HBase
>  Issue Type: Bug
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
>
> 1. For CLIENT_TIMEOUT
> {code}
>   // Time, in milliseconds, that the client will wait for a response from the 
> server before timing
>   // out. This value is used server side to determine when it is necessary to 
> send a heartbeat
>   // message to the client. Time limit will be 500 ms.
>   private static int CLIENT_TIMEOUT = 1000;
> {code}
> I think “Time limit will be 500 ms.” should be removed
> 2. For DEFAULT_ROW_SLEEP_TIME
> Typo, "same" -> "some", in
> // In this test, we sleep after reading each row. So we should make sure 
> after we get some number
> // of rows and sleep {color:red}same{color} times we must reach time limit, 
> and do not timeout after next sleeping.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18217) Correct comment errors in TestScannerHeartbeatMessages

[Xiang Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049141#comment-16049141
 ] 

Xiang Li commented on HBASE-18217:
--

Hi [~ted_yu], I found those when I studied HBASE-18089. If you have any 
comment, please feel free to let me know...

> Correct comment errors in TestScannerHeartbeatMessages
> --
>
> Key: HBASE-18217
> URL: https://issues.apache.org/jira/browse/HBASE-18217
> Project: HBase
>  Issue Type: Bug
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
>
> 1. For CLIENT_TIMEOUT
> {code}
>   // Time, in milliseconds, that the client will wait for a response from the 
> server before timing
>   // out. This value is used server side to determine when it is necessary to 
> send a heartbeat
>   // message to the client. Time limit will be 500 ms.
>   private static int CLIENT_TIMEOUT = 1000;
> {code}
> I think “Time limit will be 500 ms.” should be removed
> 2. For DEFAULT_ROW_SLEEP_TIME
> Typo, "same" -> "some", in
> // In this test, we sleep after reading each row. So we should make sure 
> after we get some number
> // of rows and sleep {color:red}same{color} times we must reach time limit, 
> and do not timeout after next sleeping.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18217) Correct comment errors in TestScannerHeartbeatMessages

[Xiang Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049140#comment-16049140
 ] 

Xiang Li commented on HBASE-18217:
--

@phil yang, I need your help to confirm if the comment of "Time limit will be 
500 ms." is not correct, as it was updated by HBASE-16051. I am a little 
confused why the patch of HBASE-16051 changes CLIENT_TIMEOUT to 1000 and also 
add the comment that "Time limit will be 500 ms.", seem not make sense.

Also, could you please explain a little more about the following comments on 
DEFAULT_ROW_SLEEP_TIME
{code}
  // So set this to 200, we will get 3 rows and reach time limit at the start 
of 4th row, then sleep
  // for the 4th time. Total time is 800 ms so we will not timeout.
{code}
you mentioned that it reaches "time limit" at the start of 4th row, it 
conflicts with the last sentence saying that "Total time is 800 ms so we will 
{color:red}not{color} timeout". If it reaches time limit, it will time out. 
Please correct me if I did not get it right. When setting CLIENT_TIMEOUT to 
1000, does it reach time limit at the start of 4th row (e.g. 600ms). Did you 
write those when you perhaps assumed that CLIENT_TIMEOUT was 500?


> Correct comment errors in TestScannerHeartbeatMessages
> --
>
> Key: HBASE-18217
> URL: https://issues.apache.org/jira/browse/HBASE-18217
> Project: HBase
>  Issue Type: Bug
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
>
> 1. For CLIENT_TIMEOUT
> {code}
>   // Time, in milliseconds, that the client will wait for a response from the 
> server before timing
>   // out. This value is used server side to determine when it is necessary to 
> send a heartbeat
>   // message to the client. Time limit will be 500 ms.
>   private static int CLIENT_TIMEOUT = 1000;
> {code}
> I think “Time limit will be 500 ms.” should be removed
> 2. For DEFAULT_ROW_SLEEP_TIME
> Typo, "same" -> "some", in
> // In this test, we sleep after reading each row. So we should make sure 
> after we get some number
> // of rows and sleep {color:red}same{color} times we must reach time limit, 
> and do not timeout after next sleeping.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18217) Correct comment errors in TestScannerHeartbeatMessages

[Xiang Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiang Li updated HBASE-18217:
-
Description: 
1. For CLIENT_TIMEOUT
{code}
  // Time, in milliseconds, that the client will wait for a response from the 
server before timing
  // out. This value is used server side to determine when it is necessary to 
send a heartbeat
  // message to the client. Time limit will be 500 ms.
  private static int CLIENT_TIMEOUT = 1000;
{code}
I think “Time limit will be 500 ms.” should be removed

2. For DEFAULT_ROW_SLEEP_TIME
Typo, "same" -> "some", in
// In this test, we sleep after reading each row. So we should make sure after 
we get some number
// of rows and sleep {color:red}same{color} times we must reach time limit, and 
do not timeout after next sleeping.

> Correct comment errors in TestScannerHeartbeatMessages
> --
>
> Key: HBASE-18217
> URL: https://issues.apache.org/jira/browse/HBASE-18217
> Project: HBase
>  Issue Type: Bug
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
>
> 1. For CLIENT_TIMEOUT
> {code}
>   // Time, in milliseconds, that the client will wait for a response from the 
> server before timing
>   // out. This value is used server side to determine when it is necessary to 
> send a heartbeat
>   // message to the client. Time limit will be 500 ms.
>   private static int CLIENT_TIMEOUT = 1000;
> {code}
> I think “Time limit will be 500 ms.” should be removed
> 2. For DEFAULT_ROW_SLEEP_TIME
> Typo, "same" -> "some", in
> // In this test, we sleep after reading each row. So we should make sure 
> after we get some number
> // of rows and sleep {color:red}same{color} times we must reach time limit, 
> and do not timeout after next sleeping.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-18217) Correct comment errors in TestScannerHeartbeatMessages

[Xiang Li (JIRA)]

Xiang Li created HBASE-18217:


 Summary: Correct comment errors in TestScannerHeartbeatMessages
 Key: HBASE-18217
 URL: https://issues.apache.org/jira/browse/HBASE-18217
 Project: HBase
  Issue Type: Bug
Reporter: Xiang Li
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HBASE-18217) Correct comment errors in TestScannerHeartbeatMessages

[Xiang Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiang Li reassigned HBASE-18217:


Assignee: Xiang Li

> Correct comment errors in TestScannerHeartbeatMessages
> --
>
> Key: HBASE-18217
> URL: https://issues.apache.org/jira/browse/HBASE-18217
> Project: HBase
>  Issue Type: Bug
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18179) Add new hadoop releases to the pre commit hadoop check

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049116#comment-16049116
 ] 

Hudson commented on HBASE-18179:


SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #3193 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/3193/])
HBASE-18179 Add new hadoop releases to the pre commit hadoop check (zhangduo: 
rev 58b377751aff8909c016beb9dc5ceafa6779593c)
* (edit) dev-support/hbase-personality.sh


> Add new hadoop releases to the pre commit hadoop check
> --
>
> Key: HBASE-18179
> URL: https://issues.apache.org/jira/browse/HBASE-18179
> Project: HBase
>  Issue Type: Bug
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 3.0.0
>
> Attachments: HBASE-18179.patch, test-branch-2.patch
>
>
> 3.0.0-alpha3 is out, we should replace the old alpha2 release with alpha3. 
> And we should add new 2.x releases also.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18089) TestScannerHeartbeatMessages fails in branch-1

[Xiang Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049113#comment-16049113
 ] 

Xiang Li commented on HBASE-18089:
--

[~ted_yu], I am study this JIRA and is also reading the heartbeat related fixes 
recently (since 21 May where this JIRA was filed). 
One quick question: As I can not re-create the error now, when you filed this 
JIRA, is the error repeatable? or it is a environmental issue and only happened 
sometimes?

> TestScannerHeartbeatMessages fails in branch-1
> --
>
> Key: HBASE-18089
> URL: https://issues.apache.org/jira/browse/HBASE-18089
> Project: HBase
>  Issue Type: Test
>Reporter: Ted Yu
>Assignee: Xiang Li
> Attachments: test-heartbeat-6860.out
>
>
> From 
> https://builds.apache.org/job/PreCommit-HBASE-Build/6860/artifact/patchprocess/patch-unit-hbase-server.txt
>  :
> {code}
> testScannerHeartbeatMessages(org.apache.hadoop.hbase.regionserver.TestScannerHeartbeatMessages)
>   Time elapsed: 2.376 sec  <<< FAILURE!
> java.lang.AssertionError: Heartbeats messages are disabled, an exception 
> should be thrown. If an exception  is not thrown, the test case is not 
> testing the importance of heartbeat messages
>   at org.junit.Assert.fail(Assert.java:88)
>   at 
> org.apache.hadoop.hbase.regionserver.TestScannerHeartbeatMessages.testImportanceOfHeartbeats(TestScannerHeartbeatMessages.java:237)
>   at 
> org.apache.hadoop.hbase.regionserver.TestScannerHeartbeatMessages.testScannerHeartbeatMessages(TestScannerHeartbeatMessages.java:207)
> {code}
> Similar test failure can be observed in 
> https://builds.apache.org/job/PreCommit-HBASE-Build/6852/artifact/patchprocess/patch-unit-hbase-server.txt



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (HBASE-18128) compaction marker could be skipped

[Jingyun Tian (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049053#comment-16049053
 ] 

Jingyun Tian edited comment on HBASE-18128 at 6/14/17 11:02 AM:


The failed test is unrelated to my patch and it can pass on my computer.

[~allan163] I think the point is we add the compaction marker is to guarantee 
the old file won't show up. If we don't need to guarantee this, maybe the 
compaction marker we can also skip during splitting log.


was (Author: tianjingyun):
The failed test is unrealted to my patch and it can pass on my computer.

[~allan163] I think the point is we add the compaction marker is to guarantee 
the old file won't show up. If we don't need to guarantee this, maybe the 
compaction marker we can also skip during splitting log.

> compaction marker could be skipped 
> ---
>
> Key: HBASE-18128
> URL: https://issues.apache.org/jira/browse/HBASE-18128
> Project: HBase
>  Issue Type: Improvement
>  Components: Compaction, regionserver
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
> Attachments: HBASE-18128-master.patch, HBASE-18128-master-v2.patch, 
> HBASE-18128-master-v3.patch, TestCompactionMarker.java
>
>
> The sequence for a compaction are as follows:
> 1. Compaction writes new files under region/.tmp directory (compaction output)
> 2. Compaction atomically moves the temporary file under region directory
> 3. Compaction appends a WAL edit containing the compaction input and output 
> files. Forces sync on WAL.
> 4. Compaction deletes the input files from the region directory.
> But if a flush happened between 3 and 4, then the regionserver crushed. The 
> compaction marker will be skipped when splitting log because the sequence id 
> of compaction marker is smaller than lastFlushedSequenceId.
> {code}
> if (lastFlushedSequenceId >= entry.getKey().getLogSeqNum()) {
>   editsSkipped++;
>   continue;
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18128) compaction marker could be skipped

[Jingyun Tian (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049053#comment-16049053
 ] 

Jingyun Tian commented on HBASE-18128:
--

The failed test is unrealted to my patch and it can pass on my computer.

[~allan163] I think the point is we add the compaction marker is to guarantee 
the old file won't show up. If we don't need to guarantee this, maybe the 
compaction marker we can also skip during splitting log.

> compaction marker could be skipped 
> ---
>
> Key: HBASE-18128
> URL: https://issues.apache.org/jira/browse/HBASE-18128
> Project: HBase
>  Issue Type: Improvement
>  Components: Compaction, regionserver
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
> Attachments: HBASE-18128-master.patch, HBASE-18128-master-v2.patch, 
> HBASE-18128-master-v3.patch, TestCompactionMarker.java
>
>
> The sequence for a compaction are as follows:
> 1. Compaction writes new files under region/.tmp directory (compaction output)
> 2. Compaction atomically moves the temporary file under region directory
> 3. Compaction appends a WAL edit containing the compaction input and output 
> files. Forces sync on WAL.
> 4. Compaction deletes the input files from the region directory.
> But if a flush happened between 3 and 4, then the regionserver crushed. The 
> compaction marker will be skipped when splitting log because the sequence id 
> of compaction marker is smaller than lastFlushedSequenceId.
> {code}
> if (lastFlushedSequenceId >= entry.getKey().getLogSeqNum()) {
>   editsSkipped++;
>   continue;
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HBASE-18139) maven-remote-resources-plugin fails with IndexOutOfBoundsException in hbase-assembly

[Xiang Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiang Li reassigned HBASE-18139:


Assignee: Xiang Li

> maven-remote-resources-plugin fails with IndexOutOfBoundsException in 
> hbase-assembly
> 
>
> Key: HBASE-18139
> URL: https://issues.apache.org/jira/browse/HBASE-18139
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Affects Versions: 1.3.2
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
>
> The same as HBASE-14199.
> {code}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process 
> (aggregate-licenses) on project hbase-assembly: Error rendering velocity 
> resource.: Error invoking method 'get(java.lang.Integer)' in 
> java.util.ArrayList at META-INF/LICENSE.vm[line 1678, column 8]: 
> InvocationTargetException: Index: 0, Size: 0 -> [Help 1]
> {code}
> Fail to run mvn install against the latest branch-1 and branch-1.3, with no 
> additional change.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-17008) Examples to make AsyncClient go down easy

[Yu Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16048837#comment-16048837
 ] 

Yu Li commented on HBASE-17008:
---

Sorry for the late +1, the examples look great! Thanks.

> Examples to make AsyncClient go down easy
> -
>
> Key: HBASE-17008
> URL: https://issues.apache.org/jira/browse/HBASE-17008
> Project: HBase
>  Issue Type: Sub-task
>  Components: asyncclient, Client
>Affects Versions: 3.0.0, 2.0.0-alpha-1
>Reporter: stack
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-17008.patch, HBASE-17008-v1.patch
>
>
> The parent issue is about delivering a new, async client. The new client 
> operates at a pretty low level. There will be questions on how best to use it.
> Some have come up already over in HBASE-16991. In particular, [~Apache9] and 
> [~carp84] talk about the tier they have to put on top of hbase because its 
> API is not user-friendly.
> This issue is about adding in the examples, docs., and helper classes we need 
> to make the new async client more palatable to mortals. See HBASE-16991 for 
> instance for an example of how to cache an AsyncConnection that an 
> application might make use of.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18216) [AMv2] Workaround for HBASE-18152, corrupt procedure WAL

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16048832#comment-16048832
 ] 

Hudson commented on HBASE-18216:


FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3192 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/3192/])
HBASE-18216 [AMv2] Workaround for HBASE-18152, corrupt procedure WAL (stack: 
rev 0b43353bf76f19e020e2831691a832722b590915)
* (edit) 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormatReader.java
HBASE-18216 [AMv2] Workaround for HBASE-18152, corrupt procedure WAL; (stack: 
rev 550b6c585e0390bc80516e64df8bd1a3a6e10e23)
* (edit) 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormatReader.java


> [AMv2] Workaround for HBASE-18152, corrupt procedure WAL
> 
>
> Key: HBASE-18216
> URL: https://issues.apache.org/jira/browse/HBASE-18216
> Project: HBase
>  Issue Type: Bug
>  Components: proc-v2
>Affects Versions: 2.0.0
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0
>
>
> Let me commit workaround for the issue up in HBASE-18152, corruption in the 
> master wal procedure files. Testing on cluster shows it helps.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18152) [AMv2] Corrupt Procedure WAL file; procedure data stored out of order

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16048833#comment-16048833
 ] 

Hudson commented on HBASE-18152:


FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3192 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/3192/])
HBASE-18216 [AMv2] Workaround for HBASE-18152, corrupt procedure WAL (stack: 
rev 0b43353bf76f19e020e2831691a832722b590915)
* (edit) 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormatReader.java
HBASE-18216 [AMv2] Workaround for HBASE-18152, corrupt procedure WAL; (stack: 
rev 550b6c585e0390bc80516e64df8bd1a3a6e10e23)
* (edit) 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormatReader.java


> [AMv2] Corrupt Procedure WAL file; procedure data stored out of order
> -
>
> Key: HBASE-18152
> URL: https://issues.apache.org/jira/browse/HBASE-18152
> Project: HBase
>  Issue Type: Sub-task
>  Components: Region Assignment
>Affects Versions: 2.0.0
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-18152.master.001.patch, 
> pv2-0036.log, pv2-0047.log, 
> reading_bad_wal.patch
>
>
> I've seen corruption from time-to-time testing.  Its rare enough. Often we 
> can get over it but sometimes we can't. It took me a while to capture an 
> instance of corruption. Turns out we are write to the WAL out-of-order which 
> undoes a basic tenet; that WAL content is ordered in line w/ execution.
> Below I'll post a corrupt WAL.
> Looking at the write-side, there is a lot going on. I'm not clear on how we 
> could write out of order. Will try and get more insight. Meantime parking 
> this issue here to fill data into.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-17008) Examples to make AsyncClient go down easy

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16048831#comment-16048831
 ] 

Hudson commented on HBASE-17008:


FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3192 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/3192/])
HBASE-17008 Examples to make AsyncClient go down easy (zhangduo: rev 
cb88b6462933ae5a5e79b167e3f924eb08472473)
* (add) 
hbase-examples/src/test/java/org/apache/hadoop/hbase/client/example/TestAsyncClientExample.java
* (add) 
hbase-examples/src/test/java/org/apache/hadoop/hbase/client/example/TestHttpProxyExample.java
* (add) 
hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/HttpProxyExample.java
* (add) 
hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/AsyncClientExample.java


> Examples to make AsyncClient go down easy
> -
>
> Key: HBASE-17008
> URL: https://issues.apache.org/jira/browse/HBASE-17008
> Project: HBase
>  Issue Type: Sub-task
>  Components: asyncclient, Client
>Affects Versions: 3.0.0, 2.0.0-alpha-1
>Reporter: stack
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-17008.patch, HBASE-17008-v1.patch
>
>
> The parent issue is about delivering a new, async client. The new client 
> operates at a pretty low level. There will be questions on how best to use it.
> Some have come up already over in HBASE-16991. In particular, [~Apache9] and 
> [~carp84] talk about the tier they have to put on top of hbase because its 
> API is not user-friendly.
> This issue is about adding in the examples, docs., and helper classes we need 
> to make the new async client more palatable to mortals. See HBASE-16991 for 
> instance for an example of how to cache an AsyncConnection that an 
> application might make use of.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18179) Add new hadoop releases to the pre commit hadoop check

[Duo Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang updated HBASE-18179:
--
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Pushed to master.

Thanks [~busbey] for reviewing.

> Add new hadoop releases to the pre commit hadoop check
> --
>
> Key: HBASE-18179
> URL: https://issues.apache.org/jira/browse/HBASE-18179
> Project: HBase
>  Issue Type: Bug
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 3.0.0
>
> Attachments: HBASE-18179.patch, test-branch-2.patch
>
>
> 3.0.0-alpha3 is out, we should replace the old alpha2 release with alpha3. 
> And we should add new 2.x releases also.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18152) [AMv2] Corrupt Procedure WAL file; procedure data stored out of order

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16048805#comment-16048805
 ] 

Hudson commented on HBASE-18152:


FAILURE: Integrated in Jenkins build HBase-2.0 #40 (See 
[https://builds.apache.org/job/HBase-2.0/40/])
HBASE-18216 [AMv2] Workaround for HBASE-18152, corrupt procedure WAL (stack: 
rev 1eb2f2593d00b6e92f31fe9811fdd656fcf2847b)
* (edit) 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormatReader.java
HBASE-18216 [AMv2] Workaround for HBASE-18152, corrupt procedure WAL; (stack: 
rev 16a5a9db3edf3594b492f548b78bf8342884373f)
* (edit) 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormatReader.java


> [AMv2] Corrupt Procedure WAL file; procedure data stored out of order
> -
>
> Key: HBASE-18152
> URL: https://issues.apache.org/jira/browse/HBASE-18152
> Project: HBase
>  Issue Type: Sub-task
>  Components: Region Assignment
>Affects Versions: 2.0.0
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-18152.master.001.patch, 
> pv2-0036.log, pv2-0047.log, 
> reading_bad_wal.patch
>
>
> I've seen corruption from time-to-time testing.  Its rare enough. Often we 
> can get over it but sometimes we can't. It took me a while to capture an 
> instance of corruption. Turns out we are write to the WAL out-of-order which 
> undoes a basic tenet; that WAL content is ordered in line w/ execution.
> Below I'll post a corrupt WAL.
> Looking at the write-side, there is a lot going on. I'm not clear on how we 
> could write out of order. Will try and get more insight. Meantime parking 
> this issue here to fill data into.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-17008) Examples to make AsyncClient go down easy

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16048803#comment-16048803
 ] 

Hudson commented on HBASE-17008:


FAILURE: Integrated in Jenkins build HBase-2.0 #40 (See 
[https://builds.apache.org/job/HBase-2.0/40/])
HBASE-17008 Examples to make AsyncClient go down easy (zhangduo: rev 
f50fe22196561a8e55cbf14516d7c59dd108c516)
* (add) 
hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/HttpProxyExample.java
* (add) 
hbase-examples/src/test/java/org/apache/hadoop/hbase/client/example/TestAsyncClientExample.java
* (add) 
hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/AsyncClientExample.java
* (add) 
hbase-examples/src/test/java/org/apache/hadoop/hbase/client/example/TestHttpProxyExample.java


> Examples to make AsyncClient go down easy
> -
>
> Key: HBASE-17008
> URL: https://issues.apache.org/jira/browse/HBASE-17008
> Project: HBase
>  Issue Type: Sub-task
>  Components: asyncclient, Client
>Affects Versions: 3.0.0, 2.0.0-alpha-1
>Reporter: stack
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-17008.patch, HBASE-17008-v1.patch
>
>
> The parent issue is about delivering a new, async client. The new client 
> operates at a pretty low level. There will be questions on how best to use it.
> Some have come up already over in HBASE-16991. In particular, [~Apache9] and 
> [~carp84] talk about the tier they have to put on top of hbase because its 
> API is not user-friendly.
> This issue is about adding in the examples, docs., and helper classes we need 
> to make the new async client more palatable to mortals. See HBASE-16991 for 
> instance for an example of how to cache an AsyncConnection that an 
> application might make use of.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18216) [AMv2] Workaround for HBASE-18152, corrupt procedure WAL

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16048804#comment-16048804
 ] 

Hudson commented on HBASE-18216:


FAILURE: Integrated in Jenkins build HBase-2.0 #40 (See 
[https://builds.apache.org/job/HBase-2.0/40/])
HBASE-18216 [AMv2] Workaround for HBASE-18152, corrupt procedure WAL (stack: 
rev 1eb2f2593d00b6e92f31fe9811fdd656fcf2847b)
* (edit) 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormatReader.java
HBASE-18216 [AMv2] Workaround for HBASE-18152, corrupt procedure WAL; (stack: 
rev 16a5a9db3edf3594b492f548b78bf8342884373f)
* (edit) 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormatReader.java


> [AMv2] Workaround for HBASE-18152, corrupt procedure WAL
> 
>
> Key: HBASE-18216
> URL: https://issues.apache.org/jira/browse/HBASE-18216
> Project: HBase
>  Issue Type: Bug
>  Components: proc-v2
>Affects Versions: 2.0.0
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0
>
>
> Let me commit workaround for the issue up in HBASE-18152, corruption in the 
> master wal procedure files. Testing on cluster shows it helps.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18137) Replication gets stuck for empty WALs

[Vincent Poon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vincent Poon updated HBASE-18137:
-
Release Note: 0-length WAL files can potentially cause the replication 
queue to get stuck.  A new config "replication.source.eof.autorecovery" has 
been added: if set to true (default is false), the 0-length WAL file will be 
skipped after 1) the max number of retries has been hit, and 2) there are more 
WAL files in the queue.  The risk of enabling this is that there is a chance 
the 0-length WAL file actually has some data (e.g. block went missing and will 
come back once a datanode is recovered).  (was: 0-length WAL files can 
potentially cause the replication queue to get stuck.  A new config 
"replication.source.eof.autorecovery" has been added, if set to true (default 
is false), the 0-length WAL file will be skipped after 1) the max number of 
retries has been hit, and 2) there are more WAL files in the queue.)

> Replication gets stuck for empty WALs
> -
>
> Key: HBASE-18137
> URL: https://issues.apache.org/jira/browse/HBASE-18137
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 1.3.1
>Reporter: Ashu Pachauri
>Assignee: Vincent Poon
>Priority: Critical
> Fix For: 2.0.0, 1.4.0, 1.3.2, 1.2.7
>
> Attachments: HBASE-18137.branch-1.3.v1.patch, 
> HBASE-18137.branch-1.3.v2.patch, HBASE-18137.branch-1.3.v3.patch, 
> HBASE-18137.branch-1.v1.patch, HBASE-18137.branch-1.v2.patch, 
> HBASE-18137.master.v1.patch
>
>
> Replication assumes that only the last WAL of a recovered queue can be empty. 
> But, intermittent DFS issues may cause empty WALs being created (without the 
> PWAL magic), and a roll of WAL to happen without a regionserver crash. This 
> will cause recovered queues to have empty WALs in the middle. This cause 
> replication to get stuck:
> {code}
> TRACE regionserver.ReplicationSource: Opening log 
> WARN regionserver.ReplicationSource: - Got: 
> java.io.EOFException
>   at java.io.DataInputStream.readFully(DataInputStream.java:197)
>   at java.io.DataInputStream.readFully(DataInputStream.java:169)
>   at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1915)
>   at 
> org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1880)
>   at 
> org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1829)
>   at 
> org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1843)
>   at 
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.(SequenceFileLogReader.java:70)
>   at 
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.reset(SequenceFileLogReader.java:168)
>   at 
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.initReader(SequenceFileLogReader.java:177)
>   at 
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:66)
>   at 
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:312)
>   at 
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:276)
>   at 
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:264)
>   at 
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:423)
>   at 
> org.apache.hadoop.hbase.replication.regionserver.ReplicationWALReaderManager.openReader(ReplicationWALReaderManager.java:70)
>   at 
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceWorkerThread.openReader(ReplicationSource.java:830)
>   at 
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceWorkerThread.run(ReplicationSource.java:572)
> {code}
> The WAL in question was completely empty but there were other WALs in the 
> recovered queue which were newer and non-empty.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)