[jira] [Commented] (HBASE-13770) Programmatic JAAS configuration option for secure zookeeper may be broken

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-13770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14900268#comment-14900268
 ] 

Ashish Singhi commented on HBASE-13770:
---

+1 (non-binding)

> Programmatic JAAS configuration option for secure zookeeper may be broken
> -
>
> Key: HBASE-13770
> URL: https://issues.apache.org/jira/browse/HBASE-13770
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.0.1, 1.1.0, 0.98.13, 1.2.0
>Reporter: Andrew Purtell
>Assignee: Maddineni Sukumar
> Fix For: 0.98.13
>
> Attachments: HBASE-13770-0.98.patch, HBASE-13770-0.98.patch, 
> HBASE-13770-v1.patch, HBASE-13770-v2-0.98.patch, HBASE-13770-v2.patch, 
> HBASE-13770-v3-0.98.patch, HBASE-13770-v4-0.98.patch
>
>
> While verifying the patch fix for HBASE-13768 we were unable to successfully 
> test the programmatic JAAS configuration option for secure ZooKeeper 
> integration. Unclear if that was due to a bug or incorrect test configuration.
> Update the security section of the online book with clear instructions for 
> setting up the programmatic JAAS configuration option for secure ZooKeeper 
> integration.
> Verify it works.
> Fix as necessary.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14425) In Secure Zookeeper cluster superuser will not have sufficient permission if multiple values are configured in "hbase.superuser"

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14900275#comment-14900275
 ] 

Ashish Singhi commented on HBASE-14425:
---

Nit:
bq.  String[] superUsers = zkw.getConfiguration().getStrings("hbase.superuser");
May be instead of hard coding property name, can you use 
{{Superusers.SUPERUSER_CONF_KEY}} ? I know it was already existing but at that 
time we did not had this constant in place.
Also looks like QA bot did not pick the patch last time, mind attaching again.
Otherwise lgtm.

> In Secure Zookeeper cluster superuser will not have sufficient permission if 
> multiple values are configured in "hbase.superuser"
> 
>
> Key: HBASE-14425
> URL: https://issues.apache.org/jira/browse/HBASE-14425
> Project: HBase
>  Issue Type: Bug
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 2.0.0
>
> Attachments: HBASE-14425.patch
>
>
> During master intialization we are setting ACLs for the znodes.
> In ZKUtil.createACL(ZooKeeperWatcher zkw, String node, boolean 
> isSecureZooKeeper),
> {code}
>   String superUser = zkw.getConfiguration().get("hbase.superuser");
>   ArrayList acls = new ArrayList();
>   // add permission to hbase supper user
>   if (superUser != null) {
> acls.add(new ACL(Perms.ALL, new Id("auth", superUser)));
>   }
> {code}
> Here we are directly setting "hbase.superuser" value to Znode which will 
> cause an issue when multiple values are configured. In "hbase.superuser" 
> multiple superusers and supergroups can be configured separated by comma. We 
> need to iterate them and set ACL.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14454) TestFailedAppendAndSync fail

[Heng Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14900294#comment-14900294
 ] 

Heng Chen commented on HBASE-14454:
---

I think it is caused by issue 
https://issues.apache.org/jira/browse/HBASE-14391

This issue's patch add some code in {{FSHLog.close}} like 
{code}
+if (!fs.delete(this.fullPathLogDir, false)) {
+  LOG.warn("Unable to delete " + this.fullPathLogDir);
+}
{code}

> TestFailedAppendAndSync fail
> 
>
> Key: HBASE-14454
> URL: https://issues.apache.org/jira/browse/HBASE-14454
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
>
> https://builds.apache.org/view/H-L/view/HBase/job/PreCommit-HBASE-Build/15633/testReport/org.apache.hadoop.hbase.regionserver/TestFailedAppendAndSync/testLockupAroundBadAssignSync/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14454) TestFailedAppendAndSync fail

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14900297#comment-14900297
 ] 

Ashish Singhi commented on HBASE-14454:
---

{quote}I think it is caused by issue 
https://issues.apache.org/jira/browse/HBASE-14391
{quote}
No, issue is yet to resolve.

> TestFailedAppendAndSync fail
> 
>
> Key: HBASE-14454
> URL: https://issues.apache.org/jira/browse/HBASE-14454
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
>
> https://builds.apache.org/view/H-L/view/HBase/job/PreCommit-HBASE-Build/15633/testReport/org.apache.hadoop.hbase.regionserver/TestFailedAppendAndSync/testLockupAroundBadAssignSync/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14454) TestFailedAppendAndSync fail

[Heng Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14900301#comment-14900301
 ] 

Heng Chen commented on HBASE-14454:
---

{quote}
No, issue is yet to resolve.
{quote}
 Yes,  it is not resolved, but the url of this issue's description is about  
HBASE-14391

> TestFailedAppendAndSync fail
> 
>
> Key: HBASE-14454
> URL: https://issues.apache.org/jira/browse/HBASE-14454
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
>
> https://builds.apache.org/view/H-L/view/HBase/job/PreCommit-HBASE-Build/15633/testReport/org.apache.hadoop.hbase.regionserver/TestFailedAppendAndSync/testLockupAroundBadAssignSync/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13770) Programmatic JAAS configuration option for secure zookeeper may be broken

[Maddineni Sukumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maddineni Sukumar updated HBASE-13770:
--
Attachment: HBASE-13770-v4-0.98.patch

Changing config variable name of zk client and server principals as suggested 
by [~apurtell] and [~ashish singhi]. Thanks a lot for review comments. 

> Programmatic JAAS configuration option for secure zookeeper may be broken
> -
>
> Key: HBASE-13770
> URL: https://issues.apache.org/jira/browse/HBASE-13770
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.0.1, 1.1.0, 0.98.13, 1.2.0
>Reporter: Andrew Purtell
>Assignee: Maddineni Sukumar
> Fix For: 0.98.13
>
> Attachments: HBASE-13770-0.98.patch, HBASE-13770-0.98.patch, 
> HBASE-13770-v1.patch, HBASE-13770-v2-0.98.patch, HBASE-13770-v2.patch, 
> HBASE-13770-v3-0.98.patch, HBASE-13770-v4-0.98.patch
>
>
> While verifying the patch fix for HBASE-13768 we were unable to successfully 
> test the programmatic JAAS configuration option for secure ZooKeeper 
> integration. Unclear if that was due to a bug or incorrect test configuration.
> Update the security section of the online book with clear instructions for 
> setting up the programmatic JAAS configuration option for secure ZooKeeper 
> integration.
> Verify it works.
> Fix as necessary.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14454) TestFailedAppendAndSync fail

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14900360#comment-14900360
 ] 

Ashish Singhi commented on HBASE-14454:
---

I ran the tests locally 10 times and it passes all the time for me, 
unfortunately I am not able to reproduce it locally.
The test fail for this reason,
{noformat}
java.io.IOException: Directory 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/target/test-data/1eac55dd-3ea0-4e40-94e8-51af55755e9a/TestHRegiontestLockupAroundBadAssignSync/testLockupAroundBadAssignSync
 is not empty
{noformat}
The path it failed to delete is {{FSHLog#fullPathLogDir}} and in the code I did 
not see anywhere we are explicitly trying to delete {{FSHLog#fullPathLogDir}} 
path.
Am I missing anything here ?

> TestFailedAppendAndSync fail
> 
>
> Key: HBASE-14454
> URL: https://issues.apache.org/jira/browse/HBASE-14454
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
>
> https://builds.apache.org/view/H-L/view/HBase/job/PreCommit-HBASE-Build/15633/testReport/org.apache.hadoop.hbase.regionserver/TestFailedAppendAndSync/testLockupAroundBadAssignSync/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Fwd: Subscribe

[ramkrishna vasudevan]

-- Forwarded message --
From: ramkrishna vasudevan 
Date: Fri, Sep 18, 2015 at 10:34 AM
Subject: Subscribe
To: issues-subscr...@hbase.apache.org

[jira] [Commented] (HBASE-14425) In Secure Zookeeper cluster superuser will not have sufficient permission if multiple values are configured in "hbase.superuser"

[Pankaj Kumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14900314#comment-14900314
 ] 

Pankaj Kumar commented on HBASE-14425:
--

Thanks [~ashish singhi] for reviewing the patch. I will address this in the 
another patch version.

> In Secure Zookeeper cluster superuser will not have sufficient permission if 
> multiple values are configured in "hbase.superuser"
> 
>
> Key: HBASE-14425
> URL: https://issues.apache.org/jira/browse/HBASE-14425
> Project: HBase
>  Issue Type: Bug
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 2.0.0
>
> Attachments: HBASE-14425.patch
>
>
> During master intialization we are setting ACLs for the znodes.
> In ZKUtil.createACL(ZooKeeperWatcher zkw, String node, boolean 
> isSecureZooKeeper),
> {code}
>   String superUser = zkw.getConfiguration().get("hbase.superuser");
>   ArrayList acls = new ArrayList();
>   // add permission to hbase supper user
>   if (superUser != null) {
> acls.add(new ACL(Perms.ALL, new Id("auth", superUser)));
>   }
> {code}
> Here we are directly setting "hbase.superuser" value to Znode which will 
> cause an issue when multiple values are configured. In "hbase.superuser" 
> multiple superusers and supergroups can be configured separated by comma. We 
> need to iterate them and set ACL.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14391) Empty regionserver WAL will never be deleted although the coresponding regionserver has been stale

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901849#comment-14901849
 ] 

stack commented on HBASE-14391:
---

For sure failures are related [~jerryhe]? Often we see failure on 
integration

> Empty regionserver WAL will never be deleted although the coresponding 
> regionserver has been stale
> --
>
> Key: HBASE-14391
> URL: https://issues.apache.org/jira/browse/HBASE-14391
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 1.0.2
>Reporter: Qianxi Zhang
>Assignee: Qianxi Zhang
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>
> Attachments: HBASE-14391-master-v3.patch, 
> HBASE_14391_master_v4.patch, HBASE_14391_trunk_v1.patch, 
> HBASE_14391_trunk_v2.patch, WALs-leftover-dir.txt
>
>
> When I restarted the hbase cluster in which there was few data, I found there 
> are two directories for one host with different timestamp which indicates 
> that the old regionserver wal directory is not deleted.
> FHLog#989
> {code}
>  @Override
>   public void close() throws IOException {
> shutdown();
> final FileStatus[] files = getFiles();
> if (null != files && 0 != files.length) {
>   for (FileStatus file : files) {
> Path p = getWALArchivePath(this.fullPathArchiveDir, file.getPath());
> // Tell our listeners that a log is going to be archived.
> if (!this.listeners.isEmpty()) {
>   for (WALActionsListener i : this.listeners) {
> i.preLogArchive(file.getPath(), p);
>   }
> }
> if (!FSUtils.renameAndSetModifyTime(fs, file.getPath(), p)) {
>   throw new IOException("Unable to rename " + file.getPath() + " to " 
> + p);
> }
> // Tell our listeners that a log was archived.
> if (!this.listeners.isEmpty()) {
>   for (WALActionsListener i : this.listeners) {
> i.postLogArchive(file.getPath(), p);
>   }
> }
>   }
>   LOG.debug("Moved " + files.length + " WAL file(s) to " +
> FSUtils.getPath(this.fullPathArchiveDir));
> }
> LOG.info("Closed WAL: " + toString());
>   }
> {code}
> When regionserver is stopped, the hlog will be archived, so wal/regionserver 
> is empty in hdfs.
> MasterFileSystem#252
> {code}
> if (curLogFiles == null || curLogFiles.length == 0) {
> // Empty log folder. No recovery needed
> continue;
>   }
> {code}
> The regionserver directory will be not splitted, it makes sense. But it will 
> be not deleted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12790) Support fairness across parallelized scans

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-12790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901876#comment-14901876
 ] 

ramkrishna.s.vasudevan commented on HBASE-12790:


Updated the patch in RB. RB link is 
https://reviews.apache.org/r/32447/diff/#

> Support fairness across parallelized scans
> --
>
> Key: HBASE-12790
> URL: https://issues.apache.org/jira/browse/HBASE-12790
> Project: HBase
>  Issue Type: New Feature
>Reporter: James Taylor
>Assignee: ramkrishna.s.vasudevan
>  Labels: Phoenix
> Attachments: AbstractRoundRobinQueue.java, HBASE-12790.patch, 
> HBASE-12790_1.patch, HBASE-12790_5.patch, HBASE-12790_callwrapper.patch, 
> HBASE-12790_trunk_1.patch
>
>
> Some HBase clients parallelize the execution of a scan to reduce latency in 
> getting back results. This can lead to starvation with a loaded cluster and 
> interleaved scans, since the RPC queue will be ordered and processed on a 
> FIFO basis. For example, if there are two clients, A & B that submit largish 
> scans at the same time. Say each scan is broken down into 100 scans by the 
> client (broken down into equal depth chunks along the row key), and the 100 
> scans of client A are queued first, followed immediately by the 100 scans of 
> client B. In this case, client B will be starved out of getting any results 
> back until the scans for client A complete.
> One solution to this is to use the attached AbstractRoundRobinQueue instead 
> of the standard FIFO queue. The queue to be used could be (maybe it already 
> is) configurable based on a new config parameter. Using this queue would 
> require the client to have the same identifier for all of the 100 parallel 
> scans that represent a single logical scan from the clients point of view. 
> With this information, the round robin queue would pick off a task from the 
> queue in a round robin fashion (instead of a strictly FIFO manner) to prevent 
> starvation over interleaved parallelized scans.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12751) Allow RowLock to be reader writer

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-12751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-12751:
--
Attachment: 12751.v40.txt

The javadoc WARNINGs are from the REST patch that just went in. Fix them here 
anyways. Try again.

> Allow RowLock to be reader writer
> -
>
> Key: HBASE-12751
> URL: https://issues.apache.org/jira/browse/HBASE-12751
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.3.0
>
> Attachments: 12751.rebased.v25.txt, 12751.rebased.v26.txt, 
> 12751.rebased.v26.txt, 12751.rebased.v27.txt, 12751.rebased.v29.txt, 
> 12751.rebased.v31.txt, 12751.rebased.v32.txt, 12751.rebased.v32.txt, 
> 12751.rebased.v33.txt, 12751.rebased.v34.txt, 12751.rebased.v35.txt, 
> 12751.rebased.v35.txt, 12751.rebased.v35.txt, 12751.v37.txt, 12751.v38.txt, 
> 12751.v39.txt, 12751.v40.txt, 12751v22.txt, 12751v23.txt, 12751v23.txt, 
> 12751v23.txt, 12751v23.txt, 12751v36.txt, HBASE-12751-v1.patch, 
> HBASE-12751-v10.patch, HBASE-12751-v10.patch, HBASE-12751-v11.patch, 
> HBASE-12751-v12.patch, HBASE-12751-v13.patch, HBASE-12751-v14.patch, 
> HBASE-12751-v15.patch, HBASE-12751-v16.patch, HBASE-12751-v17.patch, 
> HBASE-12751-v18.patch, HBASE-12751-v19 (1).patch, HBASE-12751-v19.patch, 
> HBASE-12751-v2.patch, HBASE-12751-v20.patch, HBASE-12751-v20.patch, 
> HBASE-12751-v21.patch, HBASE-12751-v3.patch, HBASE-12751-v4.patch, 
> HBASE-12751-v5.patch, HBASE-12751-v6.patch, HBASE-12751-v7.patch, 
> HBASE-12751-v8.patch, HBASE-12751-v9.patch, HBASE-12751.patch
>
>
> Right now every write operation grabs a row lock. This is to prevent values 
> from changing during a read modify write operation (increment or check and 
> put). However it limits parallelism in several different scenarios.
> If there are several puts to the same row but different columns or stores 
> then this is very limiting.
> If there are puts to the same column then mvcc number should ensure a 
> consistent ordering. So locking is not needed.
> However locking for check and put or increment is still needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12751) Allow RowLock to be reader writer

[Elliott Clark (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-12751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901978#comment-14901978
 ] 

Elliott Clark commented on HBASE-12751:
---

+1 commit it.

> Allow RowLock to be reader writer
> -
>
> Key: HBASE-12751
> URL: https://issues.apache.org/jira/browse/HBASE-12751
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.3.0
>
> Attachments: 12751.rebased.v25.txt, 12751.rebased.v26.txt, 
> 12751.rebased.v26.txt, 12751.rebased.v27.txt, 12751.rebased.v29.txt, 
> 12751.rebased.v31.txt, 12751.rebased.v32.txt, 12751.rebased.v32.txt, 
> 12751.rebased.v33.txt, 12751.rebased.v34.txt, 12751.rebased.v35.txt, 
> 12751.rebased.v35.txt, 12751.rebased.v35.txt, 12751.v37.txt, 12751.v38.txt, 
> 12751.v39.txt, 12751.v40.txt, 12751v22.txt, 12751v23.txt, 12751v23.txt, 
> 12751v23.txt, 12751v23.txt, 12751v36.txt, HBASE-12751-v1.patch, 
> HBASE-12751-v10.patch, HBASE-12751-v10.patch, HBASE-12751-v11.patch, 
> HBASE-12751-v12.patch, HBASE-12751-v13.patch, HBASE-12751-v14.patch, 
> HBASE-12751-v15.patch, HBASE-12751-v16.patch, HBASE-12751-v17.patch, 
> HBASE-12751-v18.patch, HBASE-12751-v19 (1).patch, HBASE-12751-v19.patch, 
> HBASE-12751-v2.patch, HBASE-12751-v20.patch, HBASE-12751-v20.patch, 
> HBASE-12751-v21.patch, HBASE-12751-v3.patch, HBASE-12751-v4.patch, 
> HBASE-12751-v5.patch, HBASE-12751-v6.patch, HBASE-12751-v7.patch, 
> HBASE-12751-v8.patch, HBASE-12751-v9.patch, HBASE-12751.patch
>
>
> Right now every write operation grabs a row lock. This is to prevent values 
> from changing during a read modify write operation (increment or check and 
> put). However it limits parallelism in several different scenarios.
> If there are several puts to the same row but different columns or stores 
> then this is very limiting.
> If there are puts to the same column then mvcc number should ensure a 
> consistent ordering. So locking is not needed.
> However locking for check and put or increment is still needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13770) Programmatic JAAS configuration option for secure zookeeper may be broken

[Maddineni Sukumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maddineni Sukumar updated HBASE-13770:
--
Attachment: HBASE-13770-v4-master.patch

patch for master branch.
fyi - [~apurtell] 


> Programmatic JAAS configuration option for secure zookeeper may be broken
> -
>
> Key: HBASE-13770
> URL: https://issues.apache.org/jira/browse/HBASE-13770
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.0.1, 1.1.0, 0.98.13, 1.2.0
>Reporter: Andrew Purtell
>Assignee: Maddineni Sukumar
> Fix For: 0.98.13
>
> Attachments: HBASE-13770-0.98.patch, HBASE-13770-0.98.patch, 
> HBASE-13770-v1.patch, HBASE-13770-v2-0.98.patch, HBASE-13770-v2.patch, 
> HBASE-13770-v3-0.98.patch, HBASE-13770-v4-0.98.patch, 
> HBASE-13770-v4-master.patch
>
>
> While verifying the patch fix for HBASE-13768 we were unable to successfully 
> test the programmatic JAAS configuration option for secure ZooKeeper 
> integration. Unclear if that was due to a bug or incorrect test configuration.
> Update the security section of the online book with clear instructions for 
> setting up the programmatic JAAS configuration option for secure ZooKeeper 
> integration.
> Verify it works.
> Fix as necessary.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13031) Ability to snapshot based on a key range

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-13031:
---
Fix Version/s: (was: 0.98.15)
   0.98.16

> Ability to snapshot based on a key range
> 
>
> Key: HBASE-13031
> URL: https://issues.apache.org/jira/browse/HBASE-13031
> Project: HBase
>  Issue Type: Improvement
>Reporter: churro morales
>Assignee: churro morales
> Fix For: 2.0.0, 1.3.0, 0.98.16
>
> Attachments: HBASE-13031-v1.patch, HBASE-13031.patch
>
>
> Posted on the mailing list and seems like some people are interested.  A 
> little background for everyone.
> We have a very large table, we would like to snapshot and transfer the data 
> to another cluster (compressed data is always better to ship).  Our problem 
> lies in the fact it could take many weeks to transfer all of the data and 
> during that time with major compactions, the data stored in dfs has the 
> potential to double which would cause us to run out of disk space.
> So we were thinking about allowing the ability to snapshot a specific key 
> range.  
> Ideally I feel the approach is that the user would specify a start and stop 
> key, those would be associated with a region boundary.  If between the time 
> the user submits the request and the snapshot is taken the boundaries change 
> (due to merging or splitting of regions) the snapshot should fail.
> We would know which regions to snapshot and if those changed between when the 
> request was submitted and the regions locked, the snapshot could simply fail 
> and the user would try again, instead of potentially giving the user more / 
> less than what they had anticipated.  I was planning on storing the start / 
> stop key in the SnapshotDescription and from there it looks pretty straight 
> forward where we just have to change the verifier code to accommodate the key 
> ranges.  
> If this design sounds good to anyone, or if I am overlooking anything please 
> let me know.  Once we agree on the design, I'll write and submit the patches.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12912) StoreScanner calls Configuration for Boolean Check on each initialization

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-12912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-12912:
---
Fix Version/s: (was: 0.98.15)
   0.98.16

> StoreScanner calls Configuration for Boolean Check on each initialization
> -
>
> Key: HBASE-12912
> URL: https://issues.apache.org/jira/browse/HBASE-12912
> Project: HBase
>  Issue Type: Bug
>Reporter: John Leach
>Assignee: John Leach
> Fix For: 2.0.0, 1.3.0, 0.98.16
>
> Attachments: StoreScannerStall.tiff
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> There is a clear CPU drain and iterator creation when creating store scanners 
> under high load.  Splice was running a TPCC test of our database and we are 
> seeing object creation and CPU waste on the boolean check
> Code Snippet...
> if (store != null && ((HStore)store).getHRegion() != null
> && store.getStorefilesCount() > 1) {
>   RegionServerServices rsService = 
> ((HStore)store).getHRegion().getRegionServerServices();
>   if (rsService == null || !rsService.getConfiguration().getBoolean(
> STORESCANNER_PARALLEL_SEEK_ENABLE, false)) return;
>   isParallelSeekEnabled = true;
>   executor = rsService.getExecutorService();
> }
> Will attach profile...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14459) Add request and response sizes metrics

[Otis Gospodnetic (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Otis Gospodnetic updated HBASE-14459:
-
Issue Type: New Feature  (was: Bug)

> Add request and response sizes metrics
> --
>
> Key: HBASE-14459
> URL: https://issues.apache.org/jira/browse/HBASE-14459
> Project: HBase
>  Issue Type: New Feature
>  Components: metrics
>Affects Versions: 1.2.0
>Reporter: Sanjeev Srivatsa
>Assignee: Sanjeev Srivatsa
> Attachments: HBASE-14459-v1.patch
>
>
> Adding metrics that should be useful:
> Request size
> Response size



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12751) Allow RowLock to be reader writer

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-12751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-12751:
--
Attachment: 12751.v39.txt

Nvm. TestServerCrashProcedure and TestDistributedLogsplitting were also doing 
DLR. That leaves TestReplicasClient. It actually passes locally. It just runs a 
long time. Lets see how we do now.

> Allow RowLock to be reader writer
> -
>
> Key: HBASE-12751
> URL: https://issues.apache.org/jira/browse/HBASE-12751
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.3.0
>
> Attachments: 12751.rebased.v25.txt, 12751.rebased.v26.txt, 
> 12751.rebased.v26.txt, 12751.rebased.v27.txt, 12751.rebased.v29.txt, 
> 12751.rebased.v31.txt, 12751.rebased.v32.txt, 12751.rebased.v32.txt, 
> 12751.rebased.v33.txt, 12751.rebased.v34.txt, 12751.rebased.v35.txt, 
> 12751.rebased.v35.txt, 12751.rebased.v35.txt, 12751.v37.txt, 12751.v38.txt, 
> 12751.v39.txt, 12751v22.txt, 12751v23.txt, 12751v23.txt, 12751v23.txt, 
> 12751v23.txt, 12751v36.txt, HBASE-12751-v1.patch, HBASE-12751-v10.patch, 
> HBASE-12751-v10.patch, HBASE-12751-v11.patch, HBASE-12751-v12.patch, 
> HBASE-12751-v13.patch, HBASE-12751-v14.patch, HBASE-12751-v15.patch, 
> HBASE-12751-v16.patch, HBASE-12751-v17.patch, HBASE-12751-v18.patch, 
> HBASE-12751-v19 (1).patch, HBASE-12751-v19.patch, HBASE-12751-v2.patch, 
> HBASE-12751-v20.patch, HBASE-12751-v20.patch, HBASE-12751-v21.patch, 
> HBASE-12751-v3.patch, HBASE-12751-v4.patch, HBASE-12751-v5.patch, 
> HBASE-12751-v6.patch, HBASE-12751-v7.patch, HBASE-12751-v8.patch, 
> HBASE-12751-v9.patch, HBASE-12751.patch
>
>
> Right now every write operation grabs a row lock. This is to prevent values 
> from changing during a read modify write operation (increment or check and 
> put). However it limits parallelism in several different scenarios.
> If there are several puts to the same row but different columns or stores 
> then this is very limiting.
> If there are puts to the same column then mvcc number should ensure a 
> consistent ordering. So locking is not needed.
> However locking for check and put or increment is still needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14445) ExportSnapshot does not honor -chuser, -chgroup, -chmod options

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901895#comment-14901895
 ] 

Ted Yu commented on HBASE-14445:


Turns out that HBASE-13250 didn't cover -chmod.

> ExportSnapshot does not honor -chuser, -chgroup, -chmod options
> ---
>
> Key: HBASE-14445
> URL: https://issues.apache.org/jira/browse/HBASE-14445
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.98.4
>Reporter: Ted Yu
> Attachments: 14445-v1.txt
>
>
> Create a snapshot of an existing HBase table, export the snapshot using the 
> -chuser, -chgroup, -chmod options.
> Look in hdfs filesystem for export. The files do not have the correct 
> ownership, group, permissions
> Thanks to Ian Roberts who first reported the issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14445) ExportSnapshot does not honor -chmod option

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-14445:
---
Status: Patch Available  (was: Reopened)

> ExportSnapshot does not honor -chmod option
> ---
>
> Key: HBASE-14445
> URL: https://issues.apache.org/jira/browse/HBASE-14445
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.98.4
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 14445-v1.txt
>
>
> Create a snapshot of an existing HBase table, export the snapshot using the 
> -chuser, -chgroup, -chmod options.
> Look in hdfs filesystem for export. The files do not have the correct 
> ownership, group, permissions
> Thanks to Ian Roberts who first reported the issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14445) ExportSnapshot does not honor -chmod option

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-14445:
---
Attachment: 14445-v1.txt

> ExportSnapshot does not honor -chmod option
> ---
>
> Key: HBASE-14445
> URL: https://issues.apache.org/jira/browse/HBASE-14445
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.98.4
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 14445-v1.txt
>
>
> Create a snapshot of an existing HBase table, export the snapshot using the 
> -chuser, -chgroup, -chmod options.
> Look in hdfs filesystem for export. The files do not have the correct 
> ownership, group, permissions
> Thanks to Ian Roberts who first reported the issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14445) ExportSnapshot does not honor -chmod option

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-14445:
---
Attachment: (was: 14445-v1.txt)

> ExportSnapshot does not honor -chmod option
> ---
>
> Key: HBASE-14445
> URL: https://issues.apache.org/jira/browse/HBASE-14445
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.98.4
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 14445-v1.txt
>
>
> Create a snapshot of an existing HBase table, export the snapshot using the 
> -chuser, -chgroup, -chmod options.
> Look in hdfs filesystem for export. The files do not have the correct 
> ownership, group, permissions
> Thanks to Ian Roberts who first reported the issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (HBASE-14445) ExportSnapshot does not honor -chmod option

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu reopened HBASE-14445:

  Assignee: Ted Yu

> ExportSnapshot does not honor -chmod option
> ---
>
> Key: HBASE-14445
> URL: https://issues.apache.org/jira/browse/HBASE-14445
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.98.4
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 14445-v1.txt
>
>
> Create a snapshot of an existing HBase table, export the snapshot using the 
> -chuser, -chgroup, -chmod options.
> Look in hdfs filesystem for export. The files do not have the correct 
> ownership, group, permissions
> Thanks to Ian Roberts who first reported the issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14445) ExportSnapshot does not honor -chmod option

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-14445:
---
Summary: ExportSnapshot does not honor -chmod option  (was: ExportSnapshot 
does not honor -chuser, -chgroup, -chmod options)

> ExportSnapshot does not honor -chmod option
> ---
>
> Key: HBASE-14445
> URL: https://issues.apache.org/jira/browse/HBASE-14445
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.98.4
>Reporter: Ted Yu
> Attachments: 14445-v1.txt
>
>
> Create a snapshot of an existing HBase table, export the snapshot using the 
> -chuser, -chgroup, -chmod options.
> Look in hdfs filesystem for export. The files do not have the correct 
> ownership, group, permissions
> Thanks to Ian Roberts who first reported the issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14445) ExportSnapshot does not honor -chmod option

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-14445:
---
Attachment: 14445-v1.txt

> ExportSnapshot does not honor -chmod option
> ---
>
> Key: HBASE-14445
> URL: https://issues.apache.org/jira/browse/HBASE-14445
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.98.4
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 14445-v1.txt
>
>
> Create a snapshot of an existing HBase table, export the snapshot using the 
> -chuser, -chgroup, -chmod options.
> Look in hdfs filesystem for export. The files do not have the correct 
> ownership, group, permissions
> Thanks to Ian Roberts who first reported the issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14374) Backport parent 'HBASE-14317 Stuck FSHLog' issue to 1.1 and 1.0

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-14374:
--
Attachment: 14373.branch-1.1.v7.txt

I think the failures above are what [~jerryhe] may have been referring too. 
Retrying. Tests pass locally.

> Backport parent 'HBASE-14317 Stuck FSHLog' issue to 1.1 and 1.0
> ---
>
> Key: HBASE-14374
> URL: https://issues.apache.org/jira/browse/HBASE-14374
> Project: HBase
>  Issue Type: Sub-task
>Reporter: stack
>Assignee: stack
> Fix For: 1.0.3, 1.1.3
>
> Attachments: 14317-branch-1.1.txt, 14317.branch-1.1.v2.txt, 
> 14317.branch-1.1.v2.txt, 14317.branch-1.1.v2.txt, 14317.branch-1.1.v6.txt, 
> 14373.branch-1.1.v7.txt, 14373.branch-1.1.v7.txt, 14374.branch-1.1.v3.txt, 
> 14374.branch-1.1.v4.txt, 14374.branch-1.1.v4.txt, 14374.branch-1.1.v4.txt, 
> 14374.branch-1.1.v4.txt, 14374.branch-1.1.v4.txt, 14374.branch-1.1.v4.txt, 
> 14374.branch-1.1.v5.txt, 14374.branch-1.1.v6.txt, 14374.branch-1.v6.txt
>
>
> Backport parent issue to branch-1.1. and branch-1.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14451) Move on to htrace-4.0.0 (from htrace-3.2.0)

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-14451:
--
Attachment: 14451v6.txt

Rebase

> Move on to htrace-4.0.0 (from htrace-3.2.0)
> ---
>
> Key: HBASE-14451
> URL: https://issues.apache.org/jira/browse/HBASE-14451
> Project: HBase
>  Issue Type: Task
>Reporter: stack
>Assignee: stack
> Attachments: 14451.txt, 14451v2.txt, 14451v3.txt, 14451v4.txt, 
> 14451v5.txt, 14451v6.txt
>
>
> htrace-4.0.0 was just release with a new API. Get up on it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14460) [Perf Regression] Merge of MVCC and SequenceId (HBASE-HBASE-8763) slowed Increments, CheckAndPuts, batch operations

[stack (JIRA)]

stack created HBASE-14460:
-

 Summary: [Perf Regression] Merge of MVCC and SequenceId 
(HBASE-HBASE-8763) slowed Increments, CheckAndPuts, batch operations
 Key: HBASE-14460
 URL: https://issues.apache.org/jira/browse/HBASE-14460
 Project: HBase
  Issue Type: Bug
  Components: Performance
Reporter: stack
Assignee: stack
Priority: Critical


As reported by 鈴木俊裕 up on the mailing list -- see "Performance degradation 
between CDH5.3.1(HBase0.98.6) and CDH5.4.5(HBase1.0.0)" -- our unification of 
sequenceid and MVCC slows Increments (and other ops) as the mvcc needs to 
'catch up' to our current point before we can read the last Increment value 
that we need to update.

We can say that our Increment is just done wrong, we should just be writing 
Increments and summing on read, but checkAndPut as well as batching operations 
have the same issue. Fix.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14147) REST Support for Namespaces

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901949#comment-14901949
 ] 

Hudson commented on HBASE-14147:


FAILURE: Integrated in HBase-TRUNK #6826 (See 
[https://builds.apache.org/job/HBase-TRUNK/6826/])
HBASE-14147 Add namespace CRUD functionality to REST (apurtell: rev 
783e20e1a7ca2c796b8d9c9e938e5c5779aa30c7)
* 
hbase-rest/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/NamespacesMessage.proto
* 
hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/model/NamespacesInstanceModel.java
* 
hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/model/NamespacesModel.java
* 
hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/protobuf/generated/NamespacesMessage.java
* 
hbase-rest/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/NamespacePropertiesMessage.proto
* 
hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/NamespacesInstanceResource.java
* 
hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/provider/JAXBContextResolver.java
* hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/NamespacesResource.java
* hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/RootResource.java
* 
hbase-rest/src/test/java/org/apache/hadoop/hbase/rest/model/TestNamespacesModel.java
* 
hbase-rest/src/test/java/org/apache/hadoop/hbase/rest/TestNamespacesResource.java
* 
hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/protobuf/generated/NamespacePropertiesMessage.java
* hbase-rest/src/main/resources/org/apache/hadoop/hbase/rest/XMLSchema.xsd
* 
hbase-rest/src/test/java/org/apache/hadoop/hbase/rest/model/TestNamespacesInstanceModel.java
* 
hbase-rest/src/test/java/org/apache/hadoop/hbase/rest/TestNamespacesInstanceResource.java
* hbase-rest/pom.xml


> REST Support for Namespaces
> ---
>
> Key: HBASE-14147
> URL: https://issues.apache.org/jira/browse/HBASE-14147
> Project: HBase
>  Issue Type: Sub-task
>  Components: REST
>Affects Versions: 1.1.1
>Reporter: Rick Kellogg
>Assignee: Matt Warhaftig
>Priority: Minor
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.15
>
> Attachments: SuccessfulLocalBuild.txt, hbase-14147-v1.patch, 
> hbase-14147-v1.patch, hbase-14147-v1.patch
>
>
> Expand REST services to include addition features:
> * Create namespace
> * Alter namespace
> * Describe namespace
> * Drop namespace
> * List tables in a specific namespace
> * List all namespaces.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14460) [Perf Regression] Merge of MVCC and SequenceId (HBASE-HBASE-8763) slowed Increments, CheckAndPuts, batch operations

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-14460:
--
Attachment: region_lock.png

Here is link to the mailing list where 鈴木俊裕 describes the issue:

http://mail-archives.apache.org/mod_mbox/hbase-dev/201509.mbox/%3ccangerjyo+k+cpskvoqxf7qvk9wzvsnm9jwdnd4q8d11y3mf...@mail.gmail.com%3E

I've attached here the nice diagram he made to illustrate the problem.


> [Perf Regression] Merge of MVCC and SequenceId (HBASE-HBASE-8763) slowed 
> Increments, CheckAndPuts, batch operations
> ---
>
> Key: HBASE-14460
> URL: https://issues.apache.org/jira/browse/HBASE-14460
> Project: HBase
>  Issue Type: Bug
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 14460.txt, region_lock.png
>
>
> As reported by 鈴木俊裕 up on the mailing list -- see "Performance degradation 
> between CDH5.3.1(HBase0.98.6) and CDH5.4.5(HBase1.0.0)" -- our unification of 
> sequenceid and MVCC slows Increments (and other ops) as the mvcc needs to 
> 'catch up' to our current point before we can read the last Increment value 
> that we need to update.
> We can say that our Increment is just done wrong, we should just be writing 
> Increments and summing on read, but checkAndPut as well as batching 
> operations have the same issue. Fix.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-14028) DistributedLogReplay drops edits when ITBLL 125M

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-14028.
---
Resolution: Later

> DistributedLogReplay drops edits when ITBLL 125M
> 
>
> Key: HBASE-14028
> URL: https://issues.apache.org/jira/browse/HBASE-14028
> Project: HBase
>  Issue Type: Bug
>  Components: Recovery
>Affects Versions: 1.2.0
>Reporter: stack
> Attachments: 14028.logging.txt
>
>
> Testing DLR before 1.2.0RC gets cut, we are dropping edits.
> Issue seems to be around replay into a deployed region that is on a server 
> that dies before all edits have finished replaying. Logging is sparse on 
> sequenceid accounting so can't tell for sure how it is happening (and if our 
> now accounting by Store is messing up DLR). Digging.
> I notice also that DLR does not refresh its cache of region location on error 
> -- it just keeps trying till whole WAL fails 8 retries...about 30 
> seconds. We could do a bit of refactor and have the replay find region in new 
> location if moved during DLR replay.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-11889) Turn distributedLogReplay off by default for HBase1.0

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-11889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-11889.
---
Resolution: Incomplete

DLR was off in 1.0. We should have closed this then. Closing now.

> Turn distributedLogReplay off by default for HBase1.0
> -
>
> Key: HBASE-11889
> URL: https://issues.apache.org/jira/browse/HBASE-11889
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.99.0
>Reporter: Jeffrey Zhong
>
> The reason is that we want to support rolling upgrade while 
> distributedLogReplay supports rolling upgrade only since 
> hbase0.98.4(HBASE-11094). So keep it off by default for 1.0 because we want 
> to rolling upgrade from 0.98.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-13567) [DLR] Region stuck in recovering mode

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-13567.
---
Resolution: Duplicate

Addressed by HBASE-13616

> [DLR] Region stuck in recovering mode
> -
>
> Key: HBASE-13567
> URL: https://issues.apache.org/jira/browse/HBASE-13567
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 1.1.0
>Reporter: stack
>Assignee: stack
> Attachments: 13567.test.txt, 13567v2.txt, dlr.test.txt
>
>
> On the hosting server, it says:
> {code}
> 2015-04-24 22:47:27,736 DEBUG 
> [PriorityRpcServer.handler=2,queue=0,port=16020] ipc.RpcServer: 
> PriorityRpcServer.handler=2,queue=0,port=16020: callId: 84 service: 
> ClientService methodName: Get size: 99 connection: 10.20.84.26:52860
> org.apache.hadoop.hbase.exceptions.RegionInRecoveryException: 
> hbase:namespace,,1425361558502.57b3dc571e5753b306509b5740cd25b9. is 
> recovering; cannot take reads
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.startRegionOperation(HRegion.java:7530)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2427)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2422)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6451)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6430)
> at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:1898)
> at 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32201)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2112)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> at 
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> whenever someone tries to read the region. In the above case, it is the 
> master trying to initialize after being killed by a monkey. It is trying to 
> set up the TableNamespaceManager. Eventually it fails after 350 attempts:
> {code}
> 2015-04-25 00:35:58,750 WARN  [c2020:16000.activeMasterManager] 
> master.TableNamespaceManager: Caught exception in initializing namespace 
> table manager
> 193959 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after 
> attempts=350, exceptions:
> 193960 Fri Apr 24 22:40:57 PDT 2015, 
> RpcRetryingCaller{globalStartTime=1429940457781, pause=100, retries=350}, 
> org.apache.hadoop.hbase.exceptions.RegionInRecoveryException: 
> org.apache.hadoop.hbase.excepti   ons.RegionInRecoveryException: 
> hbase:namespace,,1425361558502.57b3dc571e5753b306509b5740cd25b9. is 
> recovering; cannot take reads
> 193961 at 
> org.apache.hadoop.hbase.regionserver.HRegion.startRegionOperation(HRegion.java:7530)
> 193962 at 
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2427)
> 193963 at 
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2422)
> 193964 at 
> org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6451)
> 193965 at 
> org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6430)
> 193966 at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:1898)
> 193967 at 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32201)
> {code}
> The master is supposed to have 'processed' the region -- the hbase:namespace 
> in this case -- but the regionserver did not get the notification:
> 184849 2015-04-24 22:46:31,650 DEBUG [M_LOG_REPLAY_OPS-c2020:16000-8] 
> coordination.SplitLogManagerCoordination: Processing recovering 
> [3fbee1781e0c2ded3cc30b701a03797d, 3f4ea5ea14653cee6006f13c7d06d10b, e   
> 52b81449d08921c49625620cfc7ace7, 07459e75bef40ec82b6d4267c9de9971, 
> b26731667a5e0f15162ad4fa3408b99c, 57b3dc571e5753b306509b5740cd25b9, 
> 349eadd360d57083a88e0d84bcb29351, 99e3eb2bcf44bccded24103e351c   96b6] 
> and servers [c2021.halxg.cloudera.com,16020,1429940280208], 
> isMetaRecovery=false
> Do we need to keep sending notification until acknowledged by the 
> regionserver as we do w/ split?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14459) Add request and response sizes metrics

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901899#comment-14901899
 ] 

Hadoop QA commented on HBASE-14459:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12761532/HBASE-14459-v1.patch
  against master branch at commit 8765ffb0ca2b1c39d864ac495f0b3ee2abc520ed.
  ATTACHMENT ID: 12761532

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 4 new 
or modified tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
   org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat2
  
org.apache.hadoop.hbase.master.procedure.TestWALProcedureStoreOnHDFS

 {color:red}-1 core zombie tests{color}.  There are 5 zombie test(s):   
at 
org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite.testNotCachingDataBlocksDuringCompactionInternals(TestCacheOnWrite.java:484)
at 
org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite.testNotCachingDataBlocksDuringCompaction(TestCacheOnWrite.java:509)
at 
org.apache.hadoop.hbase.replication.regionserver.TestReplicationWALReaderManager.test(TestReplicationWALReaderManager.java:184)
at 
org.apache.hadoop.hbase.io.encoding.TestDataBlockEncoders.testSeekingOnSample(TestDataBlockEncoders.java:217)

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15658//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15658//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15658//artifact/patchprocess/checkstyle-aggregate.html

  Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15658//console

This message is automatically generated.

> Add request and response sizes metrics
> --
>
> Key: HBASE-14459
> URL: https://issues.apache.org/jira/browse/HBASE-14459
> Project: HBase
>  Issue Type: New Feature
>  Components: metrics
>Affects Versions: 1.2.0
>Reporter: Sanjeev Srivatsa
>Assignee: Sanjeev Srivatsa
> Attachments: HBASE-14459-v1.patch
>
>
> Adding metrics that should be useful:
> Request size
> Response size



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14391) Empty regionserver WAL will never be deleted although the coresponding regionserver has been stale

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901938#comment-14901938
 ] 

Jerry He commented on HBASE-14391:
--

Yes. They are.
The original patch is possibly deleting newly created logDir that does not yet 
have a writer created yet in a race.  My bad.

> Empty regionserver WAL will never be deleted although the coresponding 
> regionserver has been stale
> --
>
> Key: HBASE-14391
> URL: https://issues.apache.org/jira/browse/HBASE-14391
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 1.0.2
>Reporter: Qianxi Zhang
>Assignee: Qianxi Zhang
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>
> Attachments: HBASE-14391-master-v3.patch, 
> HBASE_14391_master_v4.patch, HBASE_14391_trunk_v1.patch, 
> HBASE_14391_trunk_v2.patch, WALs-leftover-dir.txt
>
>
> When I restarted the hbase cluster in which there was few data, I found there 
> are two directories for one host with different timestamp which indicates 
> that the old regionserver wal directory is not deleted.
> FHLog#989
> {code}
>  @Override
>   public void close() throws IOException {
> shutdown();
> final FileStatus[] files = getFiles();
> if (null != files && 0 != files.length) {
>   for (FileStatus file : files) {
> Path p = getWALArchivePath(this.fullPathArchiveDir, file.getPath());
> // Tell our listeners that a log is going to be archived.
> if (!this.listeners.isEmpty()) {
>   for (WALActionsListener i : this.listeners) {
> i.preLogArchive(file.getPath(), p);
>   }
> }
> if (!FSUtils.renameAndSetModifyTime(fs, file.getPath(), p)) {
>   throw new IOException("Unable to rename " + file.getPath() + " to " 
> + p);
> }
> // Tell our listeners that a log was archived.
> if (!this.listeners.isEmpty()) {
>   for (WALActionsListener i : this.listeners) {
> i.postLogArchive(file.getPath(), p);
>   }
> }
>   }
>   LOG.debug("Moved " + files.length + " WAL file(s) to " +
> FSUtils.getPath(this.fullPathArchiveDir));
> }
> LOG.info("Closed WAL: " + toString());
>   }
> {code}
> When regionserver is stopped, the hlog will be archived, so wal/regionserver 
> is empty in hdfs.
> MasterFileSystem#252
> {code}
> if (curLogFiles == null || curLogFiles.length == 0) {
> // Empty log folder. No recovery needed
> continue;
>   }
> {code}
> The regionserver directory will be not splitted, it makes sense. But it will 
> be not deleted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14460) [Perf Regression] Merge of MVCC and SequenceId (HBASE-HBASE-8763) slowed Increments, CheckAndPuts, batch operations

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-14460:
--
Attachment: 14460.txt

Silly test to demonstrate the problem.

> [Perf Regression] Merge of MVCC and SequenceId (HBASE-HBASE-8763) slowed 
> Increments, CheckAndPuts, batch operations
> ---
>
> Key: HBASE-14460
> URL: https://issues.apache.org/jira/browse/HBASE-14460
> Project: HBase
>  Issue Type: Bug
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 14460.txt
>
>
> As reported by 鈴木俊裕 up on the mailing list -- see "Performance degradation 
> between CDH5.3.1(HBase0.98.6) and CDH5.4.5(HBase1.0.0)" -- our unification of 
> sequenceid and MVCC slows Increments (and other ops) as the mvcc needs to 
> 'catch up' to our current point before we can read the last Increment value 
> that we need to update.
> We can say that our Increment is just done wrong, we should just be writing 
> Increments and summing on read, but checkAndPut as well as batching 
> operations have the same issue. Fix.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12751) Allow RowLock to be reader writer

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-12751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901956#comment-14901956
 ] 

Hadoop QA commented on HBASE-12751:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12761563/12751.v39.txt
  against master branch at commit 45d67435adc9195325bfccc78b7e1a0202a446e5.
  ATTACHMENT ID: 12761563

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 110 
new or modified tests.

{color:red}-1 Anti-pattern{color}.  The patch appears to 
have anti-pattern where BYTES_COMPARATOR was omitted:
 -getRegionInfo(), -1, new TreeMap());.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 2 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
 

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15660//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15660//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15660//artifact/patchprocess/checkstyle-aggregate.html

  Javadoc warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15660//artifact/patchprocess/patchJavadocWarnings.txt
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15660//console

This message is automatically generated.

> Allow RowLock to be reader writer
> -
>
> Key: HBASE-12751
> URL: https://issues.apache.org/jira/browse/HBASE-12751
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.3.0
>
> Attachments: 12751.rebased.v25.txt, 12751.rebased.v26.txt, 
> 12751.rebased.v26.txt, 12751.rebased.v27.txt, 12751.rebased.v29.txt, 
> 12751.rebased.v31.txt, 12751.rebased.v32.txt, 12751.rebased.v32.txt, 
> 12751.rebased.v33.txt, 12751.rebased.v34.txt, 12751.rebased.v35.txt, 
> 12751.rebased.v35.txt, 12751.rebased.v35.txt, 12751.v37.txt, 12751.v38.txt, 
> 12751.v39.txt, 12751v22.txt, 12751v23.txt, 12751v23.txt, 12751v23.txt, 
> 12751v23.txt, 12751v36.txt, HBASE-12751-v1.patch, HBASE-12751-v10.patch, 
> HBASE-12751-v10.patch, HBASE-12751-v11.patch, HBASE-12751-v12.patch, 
> HBASE-12751-v13.patch, HBASE-12751-v14.patch, HBASE-12751-v15.patch, 
> HBASE-12751-v16.patch, HBASE-12751-v17.patch, HBASE-12751-v18.patch, 
> HBASE-12751-v19 (1).patch, HBASE-12751-v19.patch, HBASE-12751-v2.patch, 
> HBASE-12751-v20.patch, HBASE-12751-v20.patch, HBASE-12751-v21.patch, 
> HBASE-12751-v3.patch, HBASE-12751-v4.patch, HBASE-12751-v5.patch, 
> HBASE-12751-v6.patch, HBASE-12751-v7.patch, HBASE-12751-v8.patch, 
> HBASE-12751-v9.patch, HBASE-12751.patch
>
>
> Right now every write operation grabs a row lock. This is to prevent values 
> from changing during a read modify write operation (increment or check and 
> put). However it limits parallelism in several different scenarios.
> If there are several puts to the same row but different columns or stores 
> then this is very limiting.
> If there are puts to the same column then mvcc number should ensure a 
> consistent ordering. So locking is not needed.
> However locking for check and put or increment is still needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14221) Reduce the number of time row comparison is done in a Scan

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901980#comment-14901980
 ] 

ramkrishna.s.vasudevan commented on HBASE-14221:


As am not getting notifications on the JIRA updates I  missed this review. 
Sorry about that.
bq.How does this extend to the MultiCF case?
Ideally the multiCF case also the same should apply. As we tend to move from CF 
to CF till we complete the current row, this same logic should be applicable. 
But I need to work on some more complex cases to see if all fits into the same 
umbrella.
bq.So, about 10% difference for this added complexity?
Do you think this is going to be more complex for the gain we get? I thought 
avoiding as many compares as possible would be beneficial. 
bq.Why need for two flags? Why not isSingleColumnFamily test not enough? When 
would we have a single store heap scanner but then a joined heap would have 
more than one?
I created one more specifically for the joined scanner case. Because by code 
the normal scanner and joined scanner are treated as special cases.
bq.Why not just have the flag be in the scanner context?
Yes. We can try to have the flag in the scanner context too I think. Let me try 
 it out.  
But before I update the patch do you think we can get this in Stack?  The gain 
is quite siginificant in almost all the tests that I did.

> Reduce the number of time row comparison is done in a Scan
> --
>
> Key: HBASE-14221
> URL: https://issues.apache.org/jira/browse/HBASE-14221
> Project: HBase
>  Issue Type: Sub-task
>  Components: Scanners
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-14221.patch, HBASE-14221_1.patch, 
> HBASE-14221_1.patch, withmatchingRowspatch.png, withoutmatchingRowspatch.png
>
>
> When we tried to do some profiling with the PE tool found this.
> Currently we do row comparisons in 3 places in a simple Scan case.
> 1) ScanQueryMatcher
> {code}
>int ret = this.rowComparator.compareRows(curCell, cell);
> if (!this.isReversed) {
>   if (ret <= -1) {
> return MatchCode.DONE;
>   } else if (ret >= 1) {
> // could optimize this, if necessary?
> // Could also be called SEEK_TO_CURRENT_ROW, but this
> // should be rare/never happens.
> return MatchCode.SEEK_NEXT_ROW;
>   }
> } else {
>   if (ret <= -1) {
> return MatchCode.SEEK_NEXT_ROW;
>   } else if (ret >= 1) {
> return MatchCode.DONE;
>   }
> }
> {code}
> 2) In StoreScanner next() while starting to scan the row
> {code}
> if (!scannerContext.hasAnyLimit(LimitScope.BETWEEN_CELLS) || 
> matcher.curCell == null ||
> isNewRow || !CellUtil.matchingRow(peeked, matcher.curCell)) {
>   this.countPerRow = 0;
>   matcher.setToNewRow(peeked);
> }
> {code}
> Particularly to see if we are in a new row.
> 3) In HRegion
> {code}
>   scannerContext.setKeepProgress(true);
>   heap.next(results, scannerContext);
>   scannerContext.setKeepProgress(tmpKeepProgress);
>   nextKv = heap.peek();
> moreCellsInRow = moreCellsInRow(nextKv, currentRowCell);
> {code}
> Here again there are cases where we need to careful for a MultiCF case.  Was 
> trying to solve this for the MultiCF case but is having lot of cases to 
> solve. But atleast for a single CF case I think these comparison can be 
> reduced.
> So for a single CF case in the SQM we are able to find if we have crossed a 
> row using the code pasted above in SQM. That comparison is definitely needed.
> Now in case of a single CF the HRegion is going to have only one element in 
> the heap and so the 3rd comparison can surely be avoided if the 
> StoreScanner.next() was over due to MatchCode.DONE caused by SQM.
> Coming to the 2nd compareRows that we do in StoreScanner. next() - even that 
> can be avoided if we know that the previous next() call was over due to a new 
> row. Doing all this I found that the compareRows in the profiler which was 
> 19% got reduced to 13%. Initially we can solve for single CF case which can 
> be extended to MultiCF cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14391) Empty regionserver WAL will never be deleted although the coresponding regionserver has been stale

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901992#comment-14901992
 ] 

Hudson commented on HBASE-14391:


FAILURE: Integrated in HBase-1.1 #672 (See 
[https://builds.apache.org/job/HBase-1.1/672/])
Revert: HBASE-14391 Empty regionserver WAL will never be deleted although the 
coresponding regionserver has been stale (jerryjch: rev 
489dd6427a499d09fb8cde4fbdd46303f0a57b20)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java


> Empty regionserver WAL will never be deleted although the coresponding 
> regionserver has been stale
> --
>
> Key: HBASE-14391
> URL: https://issues.apache.org/jira/browse/HBASE-14391
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 1.0.2
>Reporter: Qianxi Zhang
>Assignee: Qianxi Zhang
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>
> Attachments: HBASE-14391-master-v3.patch, 
> HBASE_14391_master_v4.patch, HBASE_14391_trunk_v1.patch, 
> HBASE_14391_trunk_v2.patch, WALs-leftover-dir.txt
>
>
> When I restarted the hbase cluster in which there was few data, I found there 
> are two directories for one host with different timestamp which indicates 
> that the old regionserver wal directory is not deleted.
> FHLog#989
> {code}
>  @Override
>   public void close() throws IOException {
> shutdown();
> final FileStatus[] files = getFiles();
> if (null != files && 0 != files.length) {
>   for (FileStatus file : files) {
> Path p = getWALArchivePath(this.fullPathArchiveDir, file.getPath());
> // Tell our listeners that a log is going to be archived.
> if (!this.listeners.isEmpty()) {
>   for (WALActionsListener i : this.listeners) {
> i.preLogArchive(file.getPath(), p);
>   }
> }
> if (!FSUtils.renameAndSetModifyTime(fs, file.getPath(), p)) {
>   throw new IOException("Unable to rename " + file.getPath() + " to " 
> + p);
> }
> // Tell our listeners that a log was archived.
> if (!this.listeners.isEmpty()) {
>   for (WALActionsListener i : this.listeners) {
> i.postLogArchive(file.getPath(), p);
>   }
> }
>   }
>   LOG.debug("Moved " + files.length + " WAL file(s) to " +
> FSUtils.getPath(this.fullPathArchiveDir));
> }
> LOG.info("Closed WAL: " + toString());
>   }
> {code}
> When regionserver is stopped, the hlog will be archived, so wal/regionserver 
> is empty in hdfs.
> MasterFileSystem#252
> {code}
> if (curLogFiles == null || curLogFiles.length == 0) {
> // Empty log folder. No recovery needed
> continue;
>   }
> {code}
> The regionserver directory will be not splitted, it makes sense. But it will 
> be not deleted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14367) Add normalization support to shell

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901563#comment-14901563
 ] 

Hadoop QA commented on HBASE-14367:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12761478/HBASE-14367-branch-1.2.v1.patch
  against branch-1.2 branch at commit 54e8c4bf23294b53bc50e15aa92148f7994e8962.
  ATTACHMENT ID: 12761478

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:red}-1 checkstyle{color}.  The applied patch generated 
3810 checkstyle errors (more than the master's current 3808 errors).

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 lineLengths{color}.  The patch introduces the following lines 
longer than 100:
+   * rpc SetNormalizerRunning(.SetNormalizerRunningRequest) 
returns (.SetNormalizerRunningResponse);
+   * rpc IsNormalizerEnabled(.IsNormalizerEnabledRequest) returns 
(.IsNormalizerEnabledResponse);
+ * rpc SetNormalizerRunning(.SetNormalizerRunningRequest) returns 
(.SetNormalizerRunningResponse);
+ * rpc IsNormalizerEnabled(.IsNormalizerEnabledRequest) returns 
(.IsNormalizerEnabledResponse);
+
htd.setNormalizationEnabled(JBoolean.valueOf(arg[NORMALIZATION_ENABLED])) if 
arg[NORMALIZATION_ENABLED]

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
   
org.apache.hadoop.hbase.master.TestDistributedLogSplitting

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15654//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15654//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15654//artifact/patchprocess/checkstyle-aggregate.html

Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15654//console

This message is automatically generated.

> Add normalization support to shell
> --
>
> Key: HBASE-14367
> URL: https://issues.apache.org/jira/browse/HBASE-14367
> Project: HBase
>  Issue Type: Bug
>  Components: Balancer, shell
>Affects Versions: 1.1.2
>Reporter: Lars George
>Assignee: Mikhail Antonov
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14367-branch-1.2.v1.patch, HBASE-14367.patch
>
>
> https://issues.apache.org/jira/browse/HBASE-13103 adds support for setting a 
> normalization flag per {{HTableDescriptor}}, along with the server side chore 
> to do the work.
> What is lacking is to easily set this from the shell, right now you need to 
> use the Java API to modify the descriptor. This issue is to add the flag as a 
> known attribute key and/or other means to toggle this per table.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14459) Add request and response sizes metrics

[Sanjeev Srivatsa (JIRA)]

Sanjeev Srivatsa created HBASE-14459:


 Summary: Add request and response sizes metrics
 Key: HBASE-14459
 URL: https://issues.apache.org/jira/browse/HBASE-14459
 Project: HBase
  Issue Type: Bug
Reporter: Sanjeev Srivatsa
Assignee: Sanjeev Srivatsa


Adding metrics that should be useful:
Request size
Response size



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14374) Backport parent 'HBASE-14317 Stuck FSHLog' issue to 1.1 and 1.0

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901719#comment-14901719
 ] 

Hadoop QA commented on HBASE-14374:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12761508/14373.branch-1.1.v7.txt
  against branch-1.1 branch at commit 8765ffb0ca2b1c39d864ac495f0b3ee2abc520ed.
  ATTACHMENT ID: 12761508

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 19 new 
or modified tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
   org.apache.hadoop.hbase.TestHBaseTestingUtility

 {color:red}-1 core zombie tests{color}.  There are 5 zombie test(s):   
at 
org.apache.hadoop.hbase.mapreduce.TestImportTsv.testMROnTableWithCustomMapper(TestImportTsv.java:154)
at 
org.apache.hadoop.hbase.mapreduce.TestImportExport.testExportScannerBatching(TestImportExport.java:271)
at 
org.apache.reef.io.network.DeprecatedNetworkConnectionServiceTest.testMultithreadedSharedConnMessagingNetworkConnServiceRate(DeprecatedNetworkConnectionServiceTest.java:343)
at 
org.apache.hadoop.hbase.mapreduce.TestRowCounter.testRowCounterTimeRange(TestRowCounter.java:179)
at 
org.apache.hadoop.hbase.client.TestFastFail.testFastFail(TestFastFail.java:244)

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15657//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15657//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15657//artifact/patchprocess/checkstyle-aggregate.html

  Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15657//console

This message is automatically generated.

> Backport parent 'HBASE-14317 Stuck FSHLog' issue to 1.1 and 1.0
> ---
>
> Key: HBASE-14374
> URL: https://issues.apache.org/jira/browse/HBASE-14374
> Project: HBase
>  Issue Type: Sub-task
>Reporter: stack
>Assignee: stack
> Fix For: 1.0.3, 1.1.3
>
> Attachments: 14317-branch-1.1.txt, 14317.branch-1.1.v2.txt, 
> 14317.branch-1.1.v2.txt, 14317.branch-1.1.v2.txt, 14317.branch-1.1.v6.txt, 
> 14373.branch-1.1.v7.txt, 14374.branch-1.1.v3.txt, 14374.branch-1.1.v4.txt, 
> 14374.branch-1.1.v4.txt, 14374.branch-1.1.v4.txt, 14374.branch-1.1.v4.txt, 
> 14374.branch-1.1.v4.txt, 14374.branch-1.1.v4.txt, 14374.branch-1.1.v5.txt, 
> 14374.branch-1.1.v6.txt, 14374.branch-1.v6.txt
>
>
> Backport parent issue to branch-1.1. and branch-1.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14391) Empty regionserver WAL will never be deleted although the coresponding regionserver has been stale

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901720#comment-14901720
 ] 

Hudson commented on HBASE-14391:


SUCCESS: Integrated in HBase-1.3-IT #171 (See 
[https://builds.apache.org/job/HBase-1.3-IT/171/])
HBASE-14391 Empty regionserver WAL will never be deleted although the 
coresponding regionserver has been stale (Qianxi Zhang) (jerryjch: rev 
9574c67610cf30eaad58de64171726eac954be9c)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java


> Empty regionserver WAL will never be deleted although the coresponding 
> regionserver has been stale
> --
>
> Key: HBASE-14391
> URL: https://issues.apache.org/jira/browse/HBASE-14391
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 1.0.2
>Reporter: Qianxi Zhang
>Assignee: Qianxi Zhang
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>
> Attachments: HBASE-14391-master-v3.patch, 
> HBASE_14391_master_v4.patch, HBASE_14391_trunk_v1.patch, 
> HBASE_14391_trunk_v2.patch, WALs-leftover-dir.txt
>
>
> When I restarted the hbase cluster in which there was few data, I found there 
> are two directories for one host with different timestamp which indicates 
> that the old regionserver wal directory is not deleted.
> FHLog#989
> {code}
>  @Override
>   public void close() throws IOException {
> shutdown();
> final FileStatus[] files = getFiles();
> if (null != files && 0 != files.length) {
>   for (FileStatus file : files) {
> Path p = getWALArchivePath(this.fullPathArchiveDir, file.getPath());
> // Tell our listeners that a log is going to be archived.
> if (!this.listeners.isEmpty()) {
>   for (WALActionsListener i : this.listeners) {
> i.preLogArchive(file.getPath(), p);
>   }
> }
> if (!FSUtils.renameAndSetModifyTime(fs, file.getPath(), p)) {
>   throw new IOException("Unable to rename " + file.getPath() + " to " 
> + p);
> }
> // Tell our listeners that a log was archived.
> if (!this.listeners.isEmpty()) {
>   for (WALActionsListener i : this.listeners) {
> i.postLogArchive(file.getPath(), p);
>   }
> }
>   }
>   LOG.debug("Moved " + files.length + " WAL file(s) to " +
> FSUtils.getPath(this.fullPathArchiveDir));
> }
> LOG.info("Closed WAL: " + toString());
>   }
> {code}
> When regionserver is stopped, the hlog will be archived, so wal/regionserver 
> is empty in hdfs.
> MasterFileSystem#252
> {code}
> if (curLogFiles == null || curLogFiles.length == 0) {
> // Empty log folder. No recovery needed
> continue;
>   }
> {code}
> The regionserver directory will be not splitted, it makes sense. But it will 
> be not deleted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-14147) REST Support for Namespaces

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell resolved HBASE-14147.

   Resolution: Fixed
Fix Version/s: 0.98.15

Pushed to 0.98, branch-1.2, branch-1, and master. Made minor fixups. New unit 
tests pass on all branches. 

> REST Support for Namespaces
> ---
>
> Key: HBASE-14147
> URL: https://issues.apache.org/jira/browse/HBASE-14147
> Project: HBase
>  Issue Type: Sub-task
>  Components: REST
>Affects Versions: 1.1.1
>Reporter: Rick Kellogg
>Assignee: Matt Warhaftig
>Priority: Minor
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.15
>
> Attachments: SuccessfulLocalBuild.txt, hbase-14147-v1.patch, 
> hbase-14147-v1.patch, hbase-14147-v1.patch
>
>
> Expand REST services to include addition features:
> * Create namespace
> * Alter namespace
> * Describe namespace
> * Drop namespace
> * List tables in a specific namespace
> * List all namespaces.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14455) Try to get rid of unused HConnection instance

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14900499#comment-14900499
 ] 

Ashish Singhi commented on HBASE-14455:
---

bq. I have a question, Why HMaster extends HRegionServer
Please refer HBASE-10569.

> Try to get rid of unused HConnection instance 
> --
>
> Key: HBASE-14455
> URL: https://issues.apache.org/jira/browse/HBASE-14455
> Project: HBase
>  Issue Type: Improvement
>Reporter: Heng Chen
>Priority: Minor
>
> After HBASE-14361,  we get rid of HConnection Instance in ReplicationSink in 
> standalone mode. But there are still three HConnection Instance, i think the 
> unused ones should be removed.
> The three instances are created below 
> {code}
> 6 2015-09-21 16:05:36,401 INFO  [10.0.3.80:60429.activeMasterManager] 
> client.ConnectionImplementation:.
> 7 java.lang.Throwable
> 8 >---at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.(ConnectionImplementation.java:217)
> 9 >---at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>10 >---at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>11 >---at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>12 >---at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
>13 >---at 
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:231)
>14 >---at 
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:118)
>15 >---at 
> org.apache.hadoop.hbase.master.ServerManager.(ServerManager.java:222)
>16 >---at 
> org.apache.hadoop.hbase.master.ServerManager.(ServerManager.java:212)
>17 >---at 
> org.apache.hadoop.hbase.master.HMaster.createServerManager(HMaster.java:853)
>18 >---at 
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:661)
>19 >---at 
> org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:180)
>20 >---at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1670)
>21 >---at java.lang.Thread.run(Thread.java:745)
>22 2015-09-21 16:05:36,401 INFO  [M:0;10.0.3.80:60429] 
> client.ConnectionImplementation:.
>23 java.lang.Throwable
>24 >---at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.(ConnectionImplementation.java:217)
>25 >---at 
> org.apache.hadoop.hbase.client.ConnectionUtils$1.(ConnectionUtils.java:128)
>26 >---at 
> org.apache.hadoop.hbase.client.ConnectionUtils.createShortCircuitConnection(ConnectionUtils.java:128)
>27 >---at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.createClusterConnection(HRegionServer.java:686)
>28 >---at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.setupClusterConnection(HRegionServer.java:717)
>29 >---at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:730)
>30 >---at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:885)
>31 >---at 
> org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.run(HMasterCommandLine.java:311)
>32 >---at java.lang.Thread.run(Thread.java:745)
>33 2015-09-21 16:05:36,401 INFO  [RS:0;10.0.3.80:60431] 
> client.ConnectionImplementation:.
>34 java.lang.Throwable
>35 >---at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.(ConnectionImplementation.java:217)
>36 >---at 
> org.apache.hadoop.hbase.client.ConnectionUtils$1.(ConnectionUtils.java:128)
>37 >---at 
> org.apache.hadoop.hbase.client.ConnectionUtils.createShortCircuitConnection(ConnectionUtils.java:128)
>38 >---at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.createClusterConnection(HRegionServer.java:686)
>39 >---at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.setupClusterConnection(HRegionServer.java:717)
>40 >---at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:730)
>41 >---at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:885)
>42 >---at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14448) Refine RegionGroupingProvider Phase-2: remove provider nesting and formalize wal group name

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14900561#comment-14900561
 ] 

Hadoop QA commented on HBASE-14448:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12761388/HBASE-14448_v3.patch
  against master branch at commit a7afc132e2db7ebbb571b43ae17161bff40b59c5.
  ATTACHMENT ID: 12761388

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
   org.apache.hadoop.hbase.mapreduce.TestImportExport
  org.apache.hadoop.hbase.client.TestFastFail
  org.apache.hadoop.hbase.util.TestProcessBasedCluster

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15649//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15649//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15649//artifact/patchprocess/checkstyle-aggregate.html

  Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15649//console

This message is automatically generated.

> Refine RegionGroupingProvider Phase-2: remove provider nesting and formalize 
> wal group name
> ---
>
> Key: HBASE-14448
> URL: https://issues.apache.org/jira/browse/HBASE-14448
> Project: HBase
>  Issue Type: Improvement
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14448.patch, HBASE-14448_v2.patch, 
> HBASE-14448_v3.patch
>
>
> Now we are nesting DefaultWALProvider inside RegionGroupingProvider, which 
> makes the logic ambiguous since a "provider" itself should provide logs. 
> Suggest to directly instantiate FSHlog in RegionGroupingProvider.
> W.r.t wal group name, now in RegionGroupingProvider it's using sth like 
> "\-null\-" which is quite long and unnecessary. 
> Suggest to directly use ".".
> For more details, please refer to the initial patch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14448) Refine RegionGroupingProvider Phase-2: remove provider nesting and formalize wal group name

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-14448:
---
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Thanks for the patch, Yu.

> Refine RegionGroupingProvider Phase-2: remove provider nesting and formalize 
> wal group name
> ---
>
> Key: HBASE-14448
> URL: https://issues.apache.org/jira/browse/HBASE-14448
> Project: HBase
>  Issue Type: Improvement
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14448.patch, HBASE-14448_v2.patch, 
> HBASE-14448_v3.patch
>
>
> Now we are nesting DefaultWALProvider inside RegionGroupingProvider, which 
> makes the logic ambiguous since a "provider" itself should provide logs. 
> Suggest to directly instantiate FSHlog in RegionGroupingProvider.
> W.r.t wal group name, now in RegionGroupingProvider it's using sth like 
> "\-null\-" which is quite long and unnecessary. 
> Suggest to directly use ".".
> For more details, please refer to the initial patch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14431) AsyncRpcClient#removeConnection() never removes connection from connections pool if server fails

[Yu Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14900725#comment-14900725
 ] 

Yu Li commented on HBASE-14431:
---

>From the HadoopQA report of HBASE-14448, I found TestFastFail failed with 
>below log:
{noformat}
2015-09-21 11:42:58,768 WARN  [AsyncRpcChannel-pool2-t17] 
logging.Slf4JLogger(151): An exception was thrown by 
org.apache.hadoop.hbase.ipc.AsyncRpcChannel$2.operationComplete()
java.lang.NullPointerException
at 
org.apache.hadoop.hbase.ipc.AsyncRpcClient.removeConnection(AsyncRpcClient.java:406)
at 
org.apache.hadoop.hbase.ipc.AsyncRpcChannel.close(AsyncRpcChannel.java:537)
at 
org.apache.hadoop.hbase.ipc.AsyncRpcChannel.retryOrClose(AsyncRpcChannel.java:300)
at 
org.apache.hadoop.hbase.ipc.AsyncRpcChannel.access$200(AsyncRpcChannel.java:82)
{noformat}

Checking line 406 of AsyncRpcClient.java, we could find below changes in this 
JIRA:
{noformat}
-int connectionHashCode = connection.getConnectionHashCode();
+int connectionHashCode = connection.hashCode();
 synchronized (connections) {
   // we use address as cache key, so we should check here to prevent 
removing the
   // wrong connection
   AsyncRpcChannel connectionInPool = 
this.connections.get(connectionHashCode);
-  if (connectionInPool == connection) {
+ if (connectionInPool.equals(connection)) {
{noformat}
And line 406 is
{code}
if (connectionInPool.equals(connection)) {
{code}

I think we lack a null pointer check here, and attached is a straight addendum.

> AsyncRpcClient#removeConnection() never removes connection from connections 
> pool if server fails
> 
>
> Key: HBASE-14431
> URL: https://issues.apache.org/jira/browse/HBASE-14431
> Project: HBase
>  Issue Type: Bug
>  Components: IPC/RPC
>Affects Versions: 2.0.0, 1.0.2, 1.1.2
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
>Priority: Critical
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>
> Attachments: HBASE-14431-v2.patch, HBASE-14431.patch
>
>
> I was playing with master branch in distributed mode (3 rs + master + 
> backup_master) and notice strange behavior when i was testing this sequence 
> of events on single rs: /kill/start/run_balancer while client was writing 
> data to cluster (LoadTestTool).
> I have notice that LTT fails with following:
> {code}
> 2015-09-09 11:05:58,364 INFO  [main] client.AsyncProcess: #2, waiting for 
> some tasks to finish. Expected max=0, tasksInProgress=35
> Exception in thread "main" 
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 
> action: BindException: 1 time, 
>   at 
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:228)
>   at 
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1800(AsyncProcess.java:208)
>   at 
> org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1697)
>   at 
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:211)
> {code}
> After some digging  and adding some more logging in code i have notice that 
> following condition in  {code}AsyncRpcClient.removeConnection(AsyncRpcChannel 
> connection) {code} is never true:
> {code}
> if (connectionInPool == connection) {
> {code} 
> causing that  {code}AsyncRpcChannel{code} connection is never removed from 
> {code}connections{code} pool in case rs fails.
> After changing this condition to:
> {code}
> if (connectionInPool.address.equals(connection.address)) {
> {code}
> issue was resolved and client was removing failed server from connections 
> pool.
> I will attach patch after running some more tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14456) Implement a namespace-based region grouping strategy for RegionGroupingProvider

[Yu Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yu Li updated HBASE-14456:
--
Fix Version/s: 1.3.0
   2.0.0
   Status: Patch Available  (was: Open)

Submit patch to see what HadoopQA will say

> Implement a namespace-based region grouping strategy for 
> RegionGroupingProvider
> ---
>
> Key: HBASE-14456
> URL: https://issues.apache.org/jira/browse/HBASE-14456
> Project: HBase
>  Issue Type: New Feature
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14456.patch, HBASE-14456_v2.patch
>
>
> In HBASE-5699 we introduced multiple wal, and after some succeeding works we 
> now have two main kinds of wal providers: DefaultWALProvider and 
> RegionGroupingProvider, while DefaultWALProvider supplies a _single_ wal per 
> regionserver, RegionGroupingProvider supplies _multiple_ wals according to 
> the RegionGroupingStrategy it's using.
> Now there're two kinds of RegionGroupingStrategy: _identity_ and _bounded_, 
> in which "identity" only for testing (trial) purpose and "bounded" for 
> randomly assigning region writes to bounded number of wals. Although the 
> "bounded" strategy is good enough for IO leverage, we may still want a more 
> "designed" way to distribute the region writes, like having one wal group per 
> business.
> Since we already have namespace for multi-tenancy, it would be good to have 
> namespace-based grouping strategy, and this is exactly what this JIRA is for.
> There might be more benefits if we have namespace-based wal group, like only 
> replicating data for the namespace in need w/o filtering anything useless, or 
> only force region flush for the heavy loaded namespace, etc. although we  
> won't include all these in this single JIRA.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14456) Implement a namespace-based region grouping strategy for RegionGroupingProvider

[Yu Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yu Li updated HBASE-14456:
--
Attachment: HBASE-14456_v2.patch

> Implement a namespace-based region grouping strategy for 
> RegionGroupingProvider
> ---
>
> Key: HBASE-14456
> URL: https://issues.apache.org/jira/browse/HBASE-14456
> Project: HBase
>  Issue Type: New Feature
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-14456.patch, HBASE-14456_v2.patch
>
>
> In HBASE-5699 we introduced multiple wal, and after some succeeding works we 
> now have two main kinds of wal providers: DefaultWALProvider and 
> RegionGroupingProvider, while DefaultWALProvider supplies a _single_ wal per 
> regionserver, RegionGroupingProvider supplies _multiple_ wals according to 
> the RegionGroupingStrategy it's using.
> Now there're two kinds of RegionGroupingStrategy: _identity_ and _bounded_, 
> in which "identity" only for testing (trial) purpose and "bounded" for 
> randomly assigning region writes to bounded number of wals. Although the 
> "bounded" strategy is good enough for IO leverage, we may still want a more 
> "designed" way to distribute the region writes, like having one wal group per 
> business.
> Since we already have namespace for multi-tenancy, it would be good to have 
> namespace-based grouping strategy, and this is exactly what this JIRA is for.
> There might be more benefits if we have namespace-based wal group, like only 
> replicating data for the namespace in need w/o filtering anything useless, or 
> only force region flush for the heavy loaded namespace, etc. although we  
> won't include all these in this single JIRA.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-13267) Deprecate or remove isFileDeletable from SnapshotHFileCleaner

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell resolved HBASE-13267.

   Resolution: Not A Problem
Fix Version/s: (was: 0.98.15)
   (was: 1.3.0)
   (was: 2.0.0)

> Deprecate or remove isFileDeletable from SnapshotHFileCleaner
> -
>
> Key: HBASE-13267
> URL: https://issues.apache.org/jira/browse/HBASE-13267
> Project: HBase
>  Issue Type: Task
>Reporter: Andrew Purtell
>Priority: Minor
>
> The isFileDeletable method in SnapshotHFileCleaner became vestigial after 
> HBASE-12627, lets remove it. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-11290) Unlock RegionStates

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-11290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-11290:
---
Fix Version/s: (was: 0.98.15)
   0.98.16

> Unlock RegionStates
> ---
>
> Key: HBASE-11290
> URL: https://issues.apache.org/jira/browse/HBASE-11290
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Francis Liu
>Assignee: Virag Kothari
> Fix For: 2.0.0, 1.3.0, 0.98.16
>
> Attachments: HBASE-11290-0.98.patch, HBASE-11290-0.98_v2.patch, 
> HBASE-11290.draft.patch
>
>
> Even though RegionStates is a highly accessed data structure in HMaster. Most 
> of it's methods are synchronized. Which limits concurrency. Even simply 
> making some of the getters non-synchronized by using concurrent data 
> structures has helped with region assignments. We can go as simple as this 
> approach or create locks per region or a bucket lock per region bucket.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12273) Generate .tabledesc file during upgrading if missing

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-12273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-12273:
---
Fix Version/s: (was: 0.98.15)
   0.98.16

> Generate .tabledesc file during upgrading if missing
> 
>
> Key: HBASE-12273
> URL: https://issues.apache.org/jira/browse/HBASE-12273
> Project: HBase
>  Issue Type: Sub-task
>  Components: Admin
>Affects Versions: 1.0.0, 0.98.7
>Reporter: Yi Deng
>Assignee: Yi Deng
>  Labels: upgrade
> Fix For: 1.0.3, 0.98.16
>
> Attachments: 
> 1.0-0001-HBASE-12273-Add-a-tool-for-fixing-missing-TableDescr.patch, 
> 1.0-0001-INTERNAL-Add-a-tool-for-fixing-missing-TableDescript.patch
>
>
> Generate .tabledesc file during upgrading if missing



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12748) RegionCoprocessorHost.execOperation creates too many iterator objects

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-12748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-12748:
---
Fix Version/s: (was: 0.98.15)
   0.98.16

> RegionCoprocessorHost.execOperation creates too many iterator objects
> -
>
> Key: HBASE-12748
> URL: https://issues.apache.org/jira/browse/HBASE-12748
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 0.94.25, 0.98.9
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0, 1.3.0, 0.98.15
>
> Attachments: HBase-12748.patch
>
>
> This is typical pattern of enhanced for ... loop usage in a hot code path. 
> For every HBase operation it instantiates iterator for coprocessor list 
> twice. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14230) replace reflection in FSHlog with HdfsDataOutputStream#getCurrentBlockReplication()

[Nick Dimiduk (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901544#comment-14901544
 ] 

Nick Dimiduk commented on HBASE-14230:
--

"ltt" == "LoadTestTool", it's a performance utility we ship with HBase.

To repro from a dev sandbox, git checkout your change and build with {{mvn 
clean package -DskipTests}}. Then launch a local "standalone" hbase (no HDFS), 
{{./bin/start-hbase.sh}}. You can tail the log in {{logs}} to watch it come up. 
Then run LoadTestTool, or any other workload. In my case, I ran {{./bin/hbase 
ltt -data_block_encoding FAST_DIFF -bloom ROWCOL -num_keys 10 -read 50 
-write 10:1000}}. Once the test starts writing data, you'll see the logs fill 
with the class cast warnings.

> replace reflection in FSHlog with 
> HdfsDataOutputStream#getCurrentBlockReplication()
> ---
>
> Key: HBASE-14230
> URL: https://issues.apache.org/jira/browse/HBASE-14230
> Project: HBase
>  Issue Type: Improvement
>  Components: wal
>Reporter: Heng Chen
>Assignee: Heng Chen
>Priority: Minor
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14230.patch
>
>
> As comment TODO said, we use 
> {{HdfsDataOutputStream#getCurrentBlockReplication}} and 
> {{DFSOutputStream.getPipeLine}} to replace reflection in FSHlog



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14147) REST Support for Namespaces

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901547#comment-14901547
 ] 

Andrew Purtell commented on HBASE-14147:


I'll take this up, and will fix up for each branch while picking back. The 
compile errors above are a known detail with test annotations. There might be 
others. I'm familiar with them. 

> REST Support for Namespaces
> ---
>
> Key: HBASE-14147
> URL: https://issues.apache.org/jira/browse/HBASE-14147
> Project: HBase
>  Issue Type: Sub-task
>  Components: REST
>Affects Versions: 1.1.1
>Reporter: Rick Kellogg
>Assignee: Matt Warhaftig
>Priority: Minor
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: SuccessfulLocalBuild.txt, hbase-14147-v1.patch, 
> hbase-14147-v1.patch, hbase-14147-v1.patch
>
>
> Expand REST services to include addition features:
> * Create namespace
> * Alter namespace
> * Describe namespace
> * Drop namespace
> * List tables in a specific namespace
> * List all namespaces.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14459) Add request and response sizes metrics

[Elliott Clark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-14459:
--
Affects Version/s: 1.2.0

> Add request and response sizes metrics
> --
>
> Key: HBASE-14459
> URL: https://issues.apache.org/jira/browse/HBASE-14459
> Project: HBase
>  Issue Type: Bug
>  Components: metrics
>Affects Versions: 1.2.0
>Reporter: Sanjeev Srivatsa
>Assignee: Sanjeev Srivatsa
>
> Adding metrics that should be useful:
> Request size
> Response size



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14459) Add request and response sizes metrics

[Elliott Clark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-14459:
--
Component/s: metrics

> Add request and response sizes metrics
> --
>
> Key: HBASE-14459
> URL: https://issues.apache.org/jira/browse/HBASE-14459
> Project: HBase
>  Issue Type: Bug
>  Components: metrics
>Affects Versions: 1.2.0
>Reporter: Sanjeev Srivatsa
>Assignee: Sanjeev Srivatsa
>
> Adding metrics that should be useful:
> Request size
> Response size



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14451) Move on to htrace-4.0.0 (from htrace-3.2.0)

[Colin Patrick McCabe (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901651#comment-14901651
 ] 

Colin Patrick McCabe commented on HBASE-14451:
--

[~stack]: right, to get "cross-cutting" tracing with HTrace you need to have 
the same major version in all your components.  You can still get tracing in 
just HBase with any version you choose.  Hadoop 2.8 will have htrace 4.0.

> Move on to htrace-4.0.0 (from htrace-3.2.0)
> ---
>
> Key: HBASE-14451
> URL: https://issues.apache.org/jira/browse/HBASE-14451
> Project: HBase
>  Issue Type: Task
>Reporter: stack
>Assignee: stack
> Attachments: 14451.txt, 14451v2.txt, 14451v3.txt, 14451v4.txt, 
> 14451v5.txt
>
>
> htrace-4.0.0 was just release with a new API. Get up on it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14391) Empty regionserver WAL will never be deleted although the coresponding regionserver has been stale

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901703#comment-14901703
 ] 

Hudson commented on HBASE-14391:


SUCCESS: Integrated in HBase-1.2-IT #160 (See 
[https://builds.apache.org/job/HBase-1.2-IT/160/])
HBASE-14391 Empty regionserver WAL will never be deleted although the 
coresponding regionserver has been stale (Qianxi Zhang) (jerryjch: rev 
7e9f7b53a2c3900ba7a305dd8effab252c55e8f4)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java


> Empty regionserver WAL will never be deleted although the coresponding 
> regionserver has been stale
> --
>
> Key: HBASE-14391
> URL: https://issues.apache.org/jira/browse/HBASE-14391
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 1.0.2
>Reporter: Qianxi Zhang
>Assignee: Qianxi Zhang
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>
> Attachments: HBASE-14391-master-v3.patch, 
> HBASE_14391_master_v4.patch, HBASE_14391_trunk_v1.patch, 
> HBASE_14391_trunk_v2.patch, WALs-leftover-dir.txt
>
>
> When I restarted the hbase cluster in which there was few data, I found there 
> are two directories for one host with different timestamp which indicates 
> that the old regionserver wal directory is not deleted.
> FHLog#989
> {code}
>  @Override
>   public void close() throws IOException {
> shutdown();
> final FileStatus[] files = getFiles();
> if (null != files && 0 != files.length) {
>   for (FileStatus file : files) {
> Path p = getWALArchivePath(this.fullPathArchiveDir, file.getPath());
> // Tell our listeners that a log is going to be archived.
> if (!this.listeners.isEmpty()) {
>   for (WALActionsListener i : this.listeners) {
> i.preLogArchive(file.getPath(), p);
>   }
> }
> if (!FSUtils.renameAndSetModifyTime(fs, file.getPath(), p)) {
>   throw new IOException("Unable to rename " + file.getPath() + " to " 
> + p);
> }
> // Tell our listeners that a log was archived.
> if (!this.listeners.isEmpty()) {
>   for (WALActionsListener i : this.listeners) {
> i.postLogArchive(file.getPath(), p);
>   }
> }
>   }
>   LOG.debug("Moved " + files.length + " WAL file(s) to " +
> FSUtils.getPath(this.fullPathArchiveDir));
> }
> LOG.info("Closed WAL: " + toString());
>   }
> {code}
> When regionserver is stopped, the hlog will be archived, so wal/regionserver 
> is empty in hdfs.
> MasterFileSystem#252
> {code}
> if (curLogFiles == null || curLogFiles.length == 0) {
> // Empty log folder. No recovery needed
> continue;
>   }
> {code}
> The regionserver directory will be not splitted, it makes sense. But it will 
> be not deleted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12748) RegionCoprocessorHost.execOperation creates too many iterator objects

[Vladimir Rodionov (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-12748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901769#comment-14901769
 ] 

Vladimir Rodionov commented on HBASE-12748:
---

Sure, go ahead [~apurtell]

> RegionCoprocessorHost.execOperation creates too many iterator objects
> -
>
> Key: HBASE-12748
> URL: https://issues.apache.org/jira/browse/HBASE-12748
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 0.94.25, 0.98.9
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0, 1.3.0, 0.98.15
>
> Attachments: HBase-12748.patch
>
>
> This is typical pattern of enhanced for ... loop usage in a hot code path. 
> For every HBase operation it instantiates iterator for coprocessor list 
> twice. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-11245) Add top N function to Aggregations Protocol

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-11245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HBASE-11245.

Resolution: Later

> Add top N function to Aggregations Protocol
> ---
>
> Key: HBASE-11245
> URL: https://issues.apache.org/jira/browse/HBASE-11245
> Project: HBase
>  Issue Type: Task
>  Components: Coprocessors
>Reporter: Ted Yu
>Priority: Minor
>
> When user table has large number of rows, one type of query is to select top 
> N rows based on matching column(s).
> Aggregations Protocol (along with AggregateImplementation and 
> AggregationClient) should support this use case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12748) RegionCoprocessorHost.execOperation creates too many iterator objects

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-12748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-12748:
---
Fix Version/s: (was: 0.98.16)
   0.98.15

> RegionCoprocessorHost.execOperation creates too many iterator objects
> -
>
> Key: HBASE-12748
> URL: https://issues.apache.org/jira/browse/HBASE-12748
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 0.94.25, 0.98.9
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0, 1.3.0, 0.98.15
>
> Attachments: HBase-12748.patch
>
>
> This is typical pattern of enhanced for ... loop usage in a hot code path. 
> For every HBase operation it instantiates iterator for coprocessor list 
> twice. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12748) RegionCoprocessorHost.execOperation creates too many iterator objects

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-12748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901539#comment-14901539
 ] 

Andrew Purtell commented on HBASE-12748:


Mind if I take this (for 0.98.15 and up) [~vrodionov]? I'll try what 
[~lhofhansl] suggests.

> RegionCoprocessorHost.execOperation creates too many iterator objects
> -
>
> Key: HBASE-12748
> URL: https://issues.apache.org/jira/browse/HBASE-12748
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 0.94.25, 0.98.9
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0, 1.3.0, 0.98.15
>
> Attachments: HBase-12748.patch
>
>
> This is typical pattern of enhanced for ... loop usage in a hot code path. 
> For every HBase operation it instantiates iterator for coprocessor list 
> twice. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12890) Provide a way to throttle the number of regions moved by the balancer

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-12890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-12890:
---
Fix Version/s: (was: 0.98.15)
   0.98.16

> Provide a way to throttle the number of regions moved by the balancer
> -
>
> Key: HBASE-12890
> URL: https://issues.apache.org/jira/browse/HBASE-12890
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.98.10
>Reporter: churro morales
>Assignee: churro morales
> Fix For: 2.0.0, 1.3.0, 0.98.16
>
> Attachments: HBASE-12890.patch
>
>
> We have a very large cluster and we frequently add remove quite a few 
> regionservers from our cluster.  Whenever we do this the balancer moves 
> thousands of regions at once.  Instead we provide a configuration parameter: 
> hbase.balancer.max.regions.  This limits the number of regions that are 
> balanced per iteration.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14391) Empty regionserver WAL will never be deleted although the coresponding regionserver has been stale

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901674#comment-14901674
 ] 

Hudson commented on HBASE-14391:


FAILURE: Integrated in HBase-1.1 #671 (See 
[https://builds.apache.org/job/HBase-1.1/671/])
HBASE-14391 Empty regionserver WAL will never be deleted although the 
coresponding regionserver has been stale (Qianxi Zhang) (jerryjch: rev 
242594d0b65ede977282520b5d4edecf3d2b9e26)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java


> Empty regionserver WAL will never be deleted although the coresponding 
> regionserver has been stale
> --
>
> Key: HBASE-14391
> URL: https://issues.apache.org/jira/browse/HBASE-14391
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 1.0.2
>Reporter: Qianxi Zhang
>Assignee: Qianxi Zhang
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>
> Attachments: HBASE-14391-master-v3.patch, 
> HBASE_14391_master_v4.patch, HBASE_14391_trunk_v1.patch, 
> HBASE_14391_trunk_v2.patch, WALs-leftover-dir.txt
>
>
> When I restarted the hbase cluster in which there was few data, I found there 
> are two directories for one host with different timestamp which indicates 
> that the old regionserver wal directory is not deleted.
> FHLog#989
> {code}
>  @Override
>   public void close() throws IOException {
> shutdown();
> final FileStatus[] files = getFiles();
> if (null != files && 0 != files.length) {
>   for (FileStatus file : files) {
> Path p = getWALArchivePath(this.fullPathArchiveDir, file.getPath());
> // Tell our listeners that a log is going to be archived.
> if (!this.listeners.isEmpty()) {
>   for (WALActionsListener i : this.listeners) {
> i.preLogArchive(file.getPath(), p);
>   }
> }
> if (!FSUtils.renameAndSetModifyTime(fs, file.getPath(), p)) {
>   throw new IOException("Unable to rename " + file.getPath() + " to " 
> + p);
> }
> // Tell our listeners that a log was archived.
> if (!this.listeners.isEmpty()) {
>   for (WALActionsListener i : this.listeners) {
> i.postLogArchive(file.getPath(), p);
>   }
> }
>   }
>   LOG.debug("Moved " + files.length + " WAL file(s) to " +
> FSUtils.getPath(this.fullPathArchiveDir));
> }
> LOG.info("Closed WAL: " + toString());
>   }
> {code}
> When regionserver is stopped, the hlog will be archived, so wal/regionserver 
> is empty in hdfs.
> MasterFileSystem#252
> {code}
> if (curLogFiles == null || curLogFiles.length == 0) {
> // Empty log folder. No recovery needed
> continue;
>   }
> {code}
> The regionserver directory will be not splitted, it makes sense. But it will 
> be not deleted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14407) NotServingRegion: hbase region closed forever

[Shuaifeng Zhou (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901759#comment-14901759
 ] 

Shuaifeng Zhou commented on HBASE-14407:


[~apurtell]

> NotServingRegion: hbase region closed forever
> -
>
> Key: HBASE-14407
> URL: https://issues.apache.org/jira/browse/HBASE-14407
> Project: HBase
>  Issue Type: Bug
>  Components: Region Assignment
>Affects Versions: 0.98.10, 1.2.0, 1.1.2, 1.3.0
>Reporter: Shuaifeng Zhou
>Assignee: Shuaifeng Zhou
>Priority: Critical
> Attachments: 14407-0.98.patch, 14407-branch-1.2.patch, 
> hbase-14407-0.98.patch, hbase-14407-1.1.patch, hbase-14407-1.2.patch, 
> hs4.log, master.log
>
>
> I found a situation may cause region closed forever, and this situation 
> happend usually on my cluster, version is 0.98.10, but 1.1.2 also have the 
> problem:
> 1, master send region open to regionserver
> 2, rs open a handler do openregion
> 3, rs return resopnse to master
> 3, master not received the response, or timeout, send open region again
> 4, rs already opened the region
> 5, master processAlreadyOpenedRegion, update regionstate open in master 
> memory
> 6, master received zk message region opened(for some reason late, eg: net 
> work), and triger update regionstate open, but find that region already 
> opened, ERROR!
> 7, master send close region, and region be closed forever.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-11290) Unlock RegionStates

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-11290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-11290:
---
Assignee: (was: Virag Kothari)

> Unlock RegionStates
> ---
>
> Key: HBASE-11290
> URL: https://issues.apache.org/jira/browse/HBASE-11290
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Francis Liu
> Fix For: 2.0.0, 1.3.0, 0.98.16
>
> Attachments: HBASE-11290-0.98.patch, HBASE-11290-0.98_v2.patch, 
> HBASE-11290.draft.patch
>
>
> Even though RegionStates is a highly accessed data structure in HMaster. Most 
> of it's methods are synchronized. Which limits concurrency. Even simply 
> making some of the getters non-synchronized by using concurrent data 
> structures has helped with region assignments. We can go as simple as this 
> approach or create locks per region or a bucket lock per region bucket.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12148) Remove TimeRangeTracker as point of contention when many threads writing a Store

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-12148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-12148:
---
Fix Version/s: (was: 0.98.15)
   0.98.16

> Remove TimeRangeTracker as point of contention when many threads writing a 
> Store
> 
>
> Key: HBASE-12148
> URL: https://issues.apache.org/jira/browse/HBASE-12148
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Affects Versions: 2.0.0, 0.99.1
>Reporter: stack
>Assignee: John Leach
> Fix For: 2.0.0, 1.3.0, 0.98.16
>
> Attachments: 
> 0001-In-AtomicUtils-change-updateMin-and-updateMax-to-ret.patch, 
> 12148.addendum.txt, 12148.txt, 12148.txt, 12148v2.txt, 12148v2.txt, 
> HBASE-12148.txt, HBASE-12148V2.txt, Screen Shot 2014-10-01 at 3.39.46 PM.png, 
> Screen Shot 2014-10-01 at 3.41.07 PM.png
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14374) Backport parent 'HBASE-14317 Stuck FSHLog' issue to 1.1 and 1.0

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-14374:
--
Attachment: 14373.branch-1.1.v7.txt

Rebase

> Backport parent 'HBASE-14317 Stuck FSHLog' issue to 1.1 and 1.0
> ---
>
> Key: HBASE-14374
> URL: https://issues.apache.org/jira/browse/HBASE-14374
> Project: HBase
>  Issue Type: Sub-task
>Reporter: stack
>Assignee: stack
> Fix For: 1.0.3, 1.1.3
>
> Attachments: 14317-branch-1.1.txt, 14317.branch-1.1.v2.txt, 
> 14317.branch-1.1.v2.txt, 14317.branch-1.1.v2.txt, 14317.branch-1.1.v6.txt, 
> 14373.branch-1.1.v7.txt, 14374.branch-1.1.v3.txt, 14374.branch-1.1.v4.txt, 
> 14374.branch-1.1.v4.txt, 14374.branch-1.1.v4.txt, 14374.branch-1.1.v4.txt, 
> 14374.branch-1.1.v4.txt, 14374.branch-1.1.v4.txt, 14374.branch-1.1.v5.txt, 
> 14374.branch-1.1.v6.txt, 14374.branch-1.v6.txt
>
>
> Backport parent issue to branch-1.1. and branch-1.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12751) Allow RowLock to be reader writer

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-12751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901693#comment-14901693
 ] 

stack commented on HBASE-12751:
---

The TestIOFencing failure is because test is parameterized and the DLR run was 
hanging. Fixed.

I removed TestVisibilityLabelsWithDistributedLogReplay because this patch 
breaks DLR and DLR is deprecated as-is.

The TestDistributedLogSplitting and TestServerCrashProcedure are both hung up 
in MVCC waiting on write number to progress. The accounting must be wrong. Let 
me see If I do the below hack, tests pass...

{code}
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
index a8ffa8d..e70e7bd 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -2099,7 +2099,7 @@ public class HRegion implements HeapSize, 
PropagatingConfigurationObserver, Regi
 writeFlushRequestMarkerToWAL(wal, writeFlushWalMarker));
 // TODO: Lets see if we hang here, if there is a scenario where an 
outstanding reader
 // with a read point is in advance of this write point.
-mvcc.completeAndWait(writeEntry);
+mvcc.complete(writeEntry);
 writeEntry = null;
 return new PrepareFlushResult(flushResult, myseqid);
   } else {
@@ -2254,7 +2254,7 @@ public class HRegion implements HeapSize, 
PropagatingConfigurationObserver, Regi
   // uncommitted transactions from being written into HFiles.
   // We have to block before we start the flush, otherwise keys that
   // were removed via a rollbackMemstore could be written to Hfiles.
-  mvcc.completeAndWait(writeEntry);
+  mvcc.complete(writeEntry);
   // set writeEntry to null to prevent mvcc.complete from being called 
again inside finally
   // block
   writeEntry = null;
@@ -3158,7 +3158,7 @@ public class HRegion implements HeapSize, 
PropagatingConfigurationObserver, Regi
   // STEP 8. Advance mvcc. This will make this put visible to scanners and 
getters.
   // --
   if (writeEntry != null) {
-mvcc.completeAndWait(writeEntry);
+mvcc.complete(writeEntry);
 writeEntry = null;
   } else if (isInReplay) {
 // ensure that the sequence id of the region is at least as big as 
orig log seq id
@@ -3203,7 +3203,7 @@ public class HRegion implements HeapSize, 
PropagatingConfigurationObserver, Regi
 }
 if (writeEntry != null) mvcc.complete(writeEntry);
   } else if (writeEntry != null) {
-mvcc.completeAndWait(writeEntry);
+mvcc.complete(writeEntry);
   }

   if (locked) {
@@ -6856,7 +6856,7 @@ public class HRegion implements HeapSize, 
PropagatingConfigurationObserver, Regi
 }
 // 13. Roll mvcc forward
 if (writeEntry != null) {
-  mvcc.completeAndWait(writeEntry);
+  mvcc.complete(writeEntry);
 }
{code}

TestReplicasClient seems broke in a different way altogether. Digging.

> Allow RowLock to be reader writer
> -
>
> Key: HBASE-12751
> URL: https://issues.apache.org/jira/browse/HBASE-12751
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.3.0
>
> Attachments: 12751.rebased.v25.txt, 12751.rebased.v26.txt, 
> 12751.rebased.v26.txt, 12751.rebased.v27.txt, 12751.rebased.v29.txt, 
> 12751.rebased.v31.txt, 12751.rebased.v32.txt, 12751.rebased.v32.txt, 
> 12751.rebased.v33.txt, 12751.rebased.v34.txt, 12751.rebased.v35.txt, 
> 12751.rebased.v35.txt, 12751.rebased.v35.txt, 12751.v37.txt, 12751.v38.txt, 
> 12751v22.txt, 12751v23.txt, 12751v23.txt, 12751v23.txt, 12751v23.txt, 
> 12751v36.txt, HBASE-12751-v1.patch, HBASE-12751-v10.patch, 
> HBASE-12751-v10.patch, HBASE-12751-v11.patch, HBASE-12751-v12.patch, 
> HBASE-12751-v13.patch, HBASE-12751-v14.patch, HBASE-12751-v15.patch, 
> HBASE-12751-v16.patch, HBASE-12751-v17.patch, HBASE-12751-v18.patch, 
> HBASE-12751-v19 (1).patch, HBASE-12751-v19.patch, HBASE-12751-v2.patch, 
> HBASE-12751-v20.patch, HBASE-12751-v20.patch, HBASE-12751-v21.patch, 
> HBASE-12751-v3.patch, HBASE-12751-v4.patch, HBASE-12751-v5.patch, 
> HBASE-12751-v6.patch, HBASE-12751-v7.patch, HBASE-12751-v8.patch, 
> HBASE-12751-v9.patch, HBASE-12751.patch
>
>
> Right now every write operation grabs a row lock. This is to prevent values 
> from changing during a read modify write operation (increment or check and 
> put). However it limits parallelism in 

[jira] [Commented] (HBASE-14391) Empty regionserver WAL will never be deleted although the coresponding regionserver has been stale

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901724#comment-14901724
 ] 

Hudson commented on HBASE-14391:


FAILURE: Integrated in HBase-TRUNK #6825 (See 
[https://builds.apache.org/job/HBase-TRUNK/6825/])
HBASE-14391 Empty regionserver WAL will never be deleted although the 
coresponding regionserver has been stale (Qianxi Zhang) (jerryjch: rev 
8765ffb0ca2b1c39d864ac495f0b3ee2abc520ed)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java


> Empty regionserver WAL will never be deleted although the coresponding 
> regionserver has been stale
> --
>
> Key: HBASE-14391
> URL: https://issues.apache.org/jira/browse/HBASE-14391
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 1.0.2
>Reporter: Qianxi Zhang
>Assignee: Qianxi Zhang
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>
> Attachments: HBASE-14391-master-v3.patch, 
> HBASE_14391_master_v4.patch, HBASE_14391_trunk_v1.patch, 
> HBASE_14391_trunk_v2.patch, WALs-leftover-dir.txt
>
>
> When I restarted the hbase cluster in which there was few data, I found there 
> are two directories for one host with different timestamp which indicates 
> that the old regionserver wal directory is not deleted.
> FHLog#989
> {code}
>  @Override
>   public void close() throws IOException {
> shutdown();
> final FileStatus[] files = getFiles();
> if (null != files && 0 != files.length) {
>   for (FileStatus file : files) {
> Path p = getWALArchivePath(this.fullPathArchiveDir, file.getPath());
> // Tell our listeners that a log is going to be archived.
> if (!this.listeners.isEmpty()) {
>   for (WALActionsListener i : this.listeners) {
> i.preLogArchive(file.getPath(), p);
>   }
> }
> if (!FSUtils.renameAndSetModifyTime(fs, file.getPath(), p)) {
>   throw new IOException("Unable to rename " + file.getPath() + " to " 
> + p);
> }
> // Tell our listeners that a log was archived.
> if (!this.listeners.isEmpty()) {
>   for (WALActionsListener i : this.listeners) {
> i.postLogArchive(file.getPath(), p);
>   }
> }
>   }
>   LOG.debug("Moved " + files.length + " WAL file(s) to " +
> FSUtils.getPath(this.fullPathArchiveDir));
> }
> LOG.info("Closed WAL: " + toString());
>   }
> {code}
> When regionserver is stopped, the hlog will be archived, so wal/regionserver 
> is empty in hdfs.
> MasterFileSystem#252
> {code}
> if (curLogFiles == null || curLogFiles.length == 0) {
> // Empty log folder. No recovery needed
> continue;
>   }
> {code}
> The regionserver directory will be not splitted, it makes sense. But it will 
> be not deleted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14451) Move on to htrace-4.0.0 (from htrace-3.2.0)

[Enis Soztutar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901767#comment-14901767
 ] 

Enis Soztutar commented on HBASE-14451:
---

Thanks for the explanation. If we can still get tracing work (without depending 
on aligning with hadoops version) and both can be on the classpath, it should 
be fine. 

> Move on to htrace-4.0.0 (from htrace-3.2.0)
> ---
>
> Key: HBASE-14451
> URL: https://issues.apache.org/jira/browse/HBASE-14451
> Project: HBase
>  Issue Type: Task
>Reporter: stack
>Assignee: stack
> Attachments: 14451.txt, 14451v2.txt, 14451v3.txt, 14451v4.txt, 
> 14451v5.txt
>
>
> htrace-4.0.0 was just release with a new API. Get up on it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13667) Backport HBASE-12975 to 1.0 and 0.98 without changing coprocessors hooks

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-13667:
---
Fix Version/s: (was: 0.98.15)
   0.98.16

> Backport HBASE-12975 to 1.0 and 0.98 without changing coprocessors hooks
> 
>
> Key: HBASE-13667
> URL: https://issues.apache.org/jira/browse/HBASE-13667
> Project: HBase
>  Issue Type: Bug
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 1.0.3, 0.98.16
>
>
> We can backport Split transaction, region merge transaction interfaces to 
> branch 1.0 and 0.98 without changing coprocessor hooks. Then it should be 
> compatible.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13053) Add support of Visibility Labels in PerformanceEvaluation

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-13053:
---
Fix Version/s: (was: 0.98.15)
   0.98.16

> Add support of Visibility Labels in PerformanceEvaluation
> -
>
> Key: HBASE-13053
> URL: https://issues.apache.org/jira/browse/HBASE-13053
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 1.0.0, 0.98.10.1
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 0.98.16
>
> Attachments: HBASE-13053-0.98.patch, HBASE-13053-master.patch
>
>
> Add support of Visibility Labels in PerformanceEvaluation:
> During write operations, support adding a visibility expression to KVs.
> During read/scan operations, support using visibility authorization.
> Here is the usage:
> {noformat}
> Options:
> ...
> visibilityExp   Writes the visibility expression along with KVs. Use for 
> write commands. Visiblity labels need to pre-exist.
> visibilityAuth  Specify the visibility auths (comma separated labels) used in 
> read or scan. Visiblity labels need to pre-exist.
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14391) Empty regionserver WAL will never be deleted although the coresponding regionserver has been stale

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901663#comment-14901663
 ] 

Hudson commented on HBASE-14391:


FAILURE: Integrated in HBase-1.3 #190 (See 
[https://builds.apache.org/job/HBase-1.3/190/])
HBASE-14391 Empty regionserver WAL will never be deleted although the 
coresponding regionserver has been stale (Qianxi Zhang) (jerryjch: rev 
9574c67610cf30eaad58de64171726eac954be9c)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java


> Empty regionserver WAL will never be deleted although the coresponding 
> regionserver has been stale
> --
>
> Key: HBASE-14391
> URL: https://issues.apache.org/jira/browse/HBASE-14391
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 1.0.2
>Reporter: Qianxi Zhang
>Assignee: Qianxi Zhang
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>
> Attachments: HBASE-14391-master-v3.patch, 
> HBASE_14391_master_v4.patch, HBASE_14391_trunk_v1.patch, 
> HBASE_14391_trunk_v2.patch, WALs-leftover-dir.txt
>
>
> When I restarted the hbase cluster in which there was few data, I found there 
> are two directories for one host with different timestamp which indicates 
> that the old regionserver wal directory is not deleted.
> FHLog#989
> {code}
>  @Override
>   public void close() throws IOException {
> shutdown();
> final FileStatus[] files = getFiles();
> if (null != files && 0 != files.length) {
>   for (FileStatus file : files) {
> Path p = getWALArchivePath(this.fullPathArchiveDir, file.getPath());
> // Tell our listeners that a log is going to be archived.
> if (!this.listeners.isEmpty()) {
>   for (WALActionsListener i : this.listeners) {
> i.preLogArchive(file.getPath(), p);
>   }
> }
> if (!FSUtils.renameAndSetModifyTime(fs, file.getPath(), p)) {
>   throw new IOException("Unable to rename " + file.getPath() + " to " 
> + p);
> }
> // Tell our listeners that a log was archived.
> if (!this.listeners.isEmpty()) {
>   for (WALActionsListener i : this.listeners) {
> i.postLogArchive(file.getPath(), p);
>   }
> }
>   }
>   LOG.debug("Moved " + files.length + " WAL file(s) to " +
> FSUtils.getPath(this.fullPathArchiveDir));
> }
> LOG.info("Closed WAL: " + toString());
>   }
> {code}
> When regionserver is stopped, the hlog will be archived, so wal/regionserver 
> is empty in hdfs.
> MasterFileSystem#252
> {code}
> if (curLogFiles == null || curLogFiles.length == 0) {
> // Empty log folder. No recovery needed
> continue;
>   }
> {code}
> The regionserver directory will be not splitted, it makes sense. But it will 
> be not deleted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14391) Empty regionserver WAL will never be deleted although the coresponding regionserver has been stale

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901708#comment-14901708
 ] 

Hudson commented on HBASE-14391:


FAILURE: Integrated in HBase-1.2 #189 (See 
[https://builds.apache.org/job/HBase-1.2/189/])
HBASE-14391 Empty regionserver WAL will never be deleted although the 
coresponding regionserver has been stale (Qianxi Zhang) (jerryjch: rev 
7e9f7b53a2c3900ba7a305dd8effab252c55e8f4)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java


> Empty regionserver WAL will never be deleted although the coresponding 
> regionserver has been stale
> --
>
> Key: HBASE-14391
> URL: https://issues.apache.org/jira/browse/HBASE-14391
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 1.0.2
>Reporter: Qianxi Zhang
>Assignee: Qianxi Zhang
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>
> Attachments: HBASE-14391-master-v3.patch, 
> HBASE_14391_master_v4.patch, HBASE_14391_trunk_v1.patch, 
> HBASE_14391_trunk_v2.patch, WALs-leftover-dir.txt
>
>
> When I restarted the hbase cluster in which there was few data, I found there 
> are two directories for one host with different timestamp which indicates 
> that the old regionserver wal directory is not deleted.
> FHLog#989
> {code}
>  @Override
>   public void close() throws IOException {
> shutdown();
> final FileStatus[] files = getFiles();
> if (null != files && 0 != files.length) {
>   for (FileStatus file : files) {
> Path p = getWALArchivePath(this.fullPathArchiveDir, file.getPath());
> // Tell our listeners that a log is going to be archived.
> if (!this.listeners.isEmpty()) {
>   for (WALActionsListener i : this.listeners) {
> i.preLogArchive(file.getPath(), p);
>   }
> }
> if (!FSUtils.renameAndSetModifyTime(fs, file.getPath(), p)) {
>   throw new IOException("Unable to rename " + file.getPath() + " to " 
> + p);
> }
> // Tell our listeners that a log was archived.
> if (!this.listeners.isEmpty()) {
>   for (WALActionsListener i : this.listeners) {
> i.postLogArchive(file.getPath(), p);
>   }
> }
>   }
>   LOG.debug("Moved " + files.length + " WAL file(s) to " +
> FSUtils.getPath(this.fullPathArchiveDir));
> }
> LOG.info("Closed WAL: " + toString());
>   }
> {code}
> When regionserver is stopped, the hlog will be archived, so wal/regionserver 
> is empty in hdfs.
> MasterFileSystem#252
> {code}
> if (curLogFiles == null || curLogFiles.length == 0) {
> // Empty log folder. No recovery needed
> continue;
>   }
> {code}
> The regionserver directory will be not splitted, it makes sense. But it will 
> be not deleted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14391) Empty regionserver WAL will never be deleted although the coresponding regionserver has been stale

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901803#comment-14901803
 ] 

Jerry He commented on HBASE-14391:
--

Some of the test case failures seem to be related to the patch.
I will take a look and revert if needed.

> Empty regionserver WAL will never be deleted although the coresponding 
> regionserver has been stale
> --
>
> Key: HBASE-14391
> URL: https://issues.apache.org/jira/browse/HBASE-14391
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 1.0.2
>Reporter: Qianxi Zhang
>Assignee: Qianxi Zhang
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>
> Attachments: HBASE-14391-master-v3.patch, 
> HBASE_14391_master_v4.patch, HBASE_14391_trunk_v1.patch, 
> HBASE_14391_trunk_v2.patch, WALs-leftover-dir.txt
>
>
> When I restarted the hbase cluster in which there was few data, I found there 
> are two directories for one host with different timestamp which indicates 
> that the old regionserver wal directory is not deleted.
> FHLog#989
> {code}
>  @Override
>   public void close() throws IOException {
> shutdown();
> final FileStatus[] files = getFiles();
> if (null != files && 0 != files.length) {
>   for (FileStatus file : files) {
> Path p = getWALArchivePath(this.fullPathArchiveDir, file.getPath());
> // Tell our listeners that a log is going to be archived.
> if (!this.listeners.isEmpty()) {
>   for (WALActionsListener i : this.listeners) {
> i.preLogArchive(file.getPath(), p);
>   }
> }
> if (!FSUtils.renameAndSetModifyTime(fs, file.getPath(), p)) {
>   throw new IOException("Unable to rename " + file.getPath() + " to " 
> + p);
> }
> // Tell our listeners that a log was archived.
> if (!this.listeners.isEmpty()) {
>   for (WALActionsListener i : this.listeners) {
> i.postLogArchive(file.getPath(), p);
>   }
> }
>   }
>   LOG.debug("Moved " + files.length + " WAL file(s) to " +
> FSUtils.getPath(this.fullPathArchiveDir));
> }
> LOG.info("Closed WAL: " + toString());
>   }
> {code}
> When regionserver is stopped, the hlog will be archived, so wal/regionserver 
> is empty in hdfs.
> MasterFileSystem#252
> {code}
> if (curLogFiles == null || curLogFiles.length == 0) {
> // Empty log folder. No recovery needed
> continue;
>   }
> {code}
> The regionserver directory will be not splitted, it makes sense. But it will 
> be not deleted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (HBASE-14391) Empty regionserver WAL will never be deleted although the coresponding regionserver has been stale

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He reopened HBASE-14391:
--

> Empty regionserver WAL will never be deleted although the coresponding 
> regionserver has been stale
> --
>
> Key: HBASE-14391
> URL: https://issues.apache.org/jira/browse/HBASE-14391
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 1.0.2
>Reporter: Qianxi Zhang
>Assignee: Qianxi Zhang
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>
> Attachments: HBASE-14391-master-v3.patch, 
> HBASE_14391_master_v4.patch, HBASE_14391_trunk_v1.patch, 
> HBASE_14391_trunk_v2.patch, WALs-leftover-dir.txt
>
>
> When I restarted the hbase cluster in which there was few data, I found there 
> are two directories for one host with different timestamp which indicates 
> that the old regionserver wal directory is not deleted.
> FHLog#989
> {code}
>  @Override
>   public void close() throws IOException {
> shutdown();
> final FileStatus[] files = getFiles();
> if (null != files && 0 != files.length) {
>   for (FileStatus file : files) {
> Path p = getWALArchivePath(this.fullPathArchiveDir, file.getPath());
> // Tell our listeners that a log is going to be archived.
> if (!this.listeners.isEmpty()) {
>   for (WALActionsListener i : this.listeners) {
> i.preLogArchive(file.getPath(), p);
>   }
> }
> if (!FSUtils.renameAndSetModifyTime(fs, file.getPath(), p)) {
>   throw new IOException("Unable to rename " + file.getPath() + " to " 
> + p);
> }
> // Tell our listeners that a log was archived.
> if (!this.listeners.isEmpty()) {
>   for (WALActionsListener i : this.listeners) {
> i.postLogArchive(file.getPath(), p);
>   }
> }
>   }
>   LOG.debug("Moved " + files.length + " WAL file(s) to " +
> FSUtils.getPath(this.fullPathArchiveDir));
> }
> LOG.info("Closed WAL: " + toString());
>   }
> {code}
> When regionserver is stopped, the hlog will be archived, so wal/regionserver 
> is empty in hdfs.
> MasterFileSystem#252
> {code}
> if (curLogFiles == null || curLogFiles.length == 0) {
> // Empty log folder. No recovery needed
> continue;
>   }
> {code}
> The regionserver directory will be not splitted, it makes sense. But it will 
> be not deleted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14391) Empty regionserver WAL will never be deleted although the coresponding regionserver has been stale

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901843#comment-14901843
 ] 

Jerry He commented on HBASE-14391:
--

Reverted from all branches.

> Empty regionserver WAL will never be deleted although the coresponding 
> regionserver has been stale
> --
>
> Key: HBASE-14391
> URL: https://issues.apache.org/jira/browse/HBASE-14391
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 1.0.2
>Reporter: Qianxi Zhang
>Assignee: Qianxi Zhang
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>
> Attachments: HBASE-14391-master-v3.patch, 
> HBASE_14391_master_v4.patch, HBASE_14391_trunk_v1.patch, 
> HBASE_14391_trunk_v2.patch, WALs-leftover-dir.txt
>
>
> When I restarted the hbase cluster in which there was few data, I found there 
> are two directories for one host with different timestamp which indicates 
> that the old regionserver wal directory is not deleted.
> FHLog#989
> {code}
>  @Override
>   public void close() throws IOException {
> shutdown();
> final FileStatus[] files = getFiles();
> if (null != files && 0 != files.length) {
>   for (FileStatus file : files) {
> Path p = getWALArchivePath(this.fullPathArchiveDir, file.getPath());
> // Tell our listeners that a log is going to be archived.
> if (!this.listeners.isEmpty()) {
>   for (WALActionsListener i : this.listeners) {
> i.preLogArchive(file.getPath(), p);
>   }
> }
> if (!FSUtils.renameAndSetModifyTime(fs, file.getPath(), p)) {
>   throw new IOException("Unable to rename " + file.getPath() + " to " 
> + p);
> }
> // Tell our listeners that a log was archived.
> if (!this.listeners.isEmpty()) {
>   for (WALActionsListener i : this.listeners) {
> i.postLogArchive(file.getPath(), p);
>   }
> }
>   }
>   LOG.debug("Moved " + files.length + " WAL file(s) to " +
> FSUtils.getPath(this.fullPathArchiveDir));
> }
> LOG.info("Closed WAL: " + toString());
>   }
> {code}
> When regionserver is stopped, the hlog will be archived, so wal/regionserver 
> is empty in hdfs.
> MasterFileSystem#252
> {code}
> if (curLogFiles == null || curLogFiles.length == 0) {
> // Empty log folder. No recovery needed
> continue;
>   }
> {code}
> The regionserver directory will be not splitted, it makes sense. But it will 
> be not deleted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13268) Backport the HBASE-7781 security test updates to use the MiniKDC

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-13268:
---
Fix Version/s: (was: 0.98.15)
   0.98.16

> Backport the HBASE-7781 security test updates to use the MiniKDC
> 
>
> Key: HBASE-13268
> URL: https://issues.apache.org/jira/browse/HBASE-13268
> Project: HBase
>  Issue Type: Task
>Reporter: Andrew Purtell
> Fix For: 0.98.16
>
>
> Consider backport of the security test updates to use the MiniKDC that are 
> subtasks of HBASE-7781. Would be good to improve test coverage of security 
> code in 0.98 branch, as long as neither:
> - The changes are a PITA to backport
> - The changes break a compatibility requirement
> - The changes introduce test instability
>  
> Investigate



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14459) Add request and response sizes metrics

[Sanjeev Srivatsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sanjeev Srivatsa updated HBASE-14459:
-
Status: Patch Available  (was: Open)

> Add request and response sizes metrics
> --
>
> Key: HBASE-14459
> URL: https://issues.apache.org/jira/browse/HBASE-14459
> Project: HBase
>  Issue Type: Bug
>  Components: metrics
>Affects Versions: 1.2.0
>Reporter: Sanjeev Srivatsa
>Assignee: Sanjeev Srivatsa
>
> Adding metrics that should be useful:
> Request size
> Response size



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14459) Add request and response sizes metrics

[Sanjeev Srivatsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sanjeev Srivatsa updated HBASE-14459:
-
Attachment: HBASE-14459-v1.patch

> Add request and response sizes metrics
> --
>
> Key: HBASE-14459
> URL: https://issues.apache.org/jira/browse/HBASE-14459
> Project: HBase
>  Issue Type: Bug
>  Components: metrics
>Affects Versions: 1.2.0
>Reporter: Sanjeev Srivatsa
>Assignee: Sanjeev Srivatsa
> Attachments: HBASE-14459-v1.patch
>
>
> Adding metrics that should be useful:
> Request size
> Response size



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13770) Programmatic JAAS configuration option for secure zookeeper may be broken

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-13770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901490#comment-14901490
 ] 

Andrew Purtell commented on HBASE-13770:


Looks good. We only have a patch for 0.98 here? Will need one for master, then 
we should be good to go. 

> Programmatic JAAS configuration option for secure zookeeper may be broken
> -
>
> Key: HBASE-13770
> URL: https://issues.apache.org/jira/browse/HBASE-13770
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.0.1, 1.1.0, 0.98.13, 1.2.0
>Reporter: Andrew Purtell
>Assignee: Maddineni Sukumar
> Fix For: 0.98.13
>
> Attachments: HBASE-13770-0.98.patch, HBASE-13770-0.98.patch, 
> HBASE-13770-v1.patch, HBASE-13770-v2-0.98.patch, HBASE-13770-v2.patch, 
> HBASE-13770-v3-0.98.patch, HBASE-13770-v4-0.98.patch
>
>
> While verifying the patch fix for HBASE-13768 we were unable to successfully 
> test the programmatic JAAS configuration option for secure ZooKeeper 
> integration. Unclear if that was due to a bug or incorrect test configuration.
> Update the security section of the online book with clear instructions for 
> setting up the programmatic JAAS configuration option for secure ZooKeeper 
> integration.
> Verify it works.
> Fix as necessary.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14443) Add request parameter to the TooSlow/TooLarge warn message of RpcServer

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901494#comment-14901494
 ] 

Andrew Purtell commented on HBASE-14443:


What about when table schema details might be sensitive? Row key might contain 
sensitive information? Wouldn't want those details going to the log in that 
case. 

> Add request parameter to the TooSlow/TooLarge warn message of RpcServer
> ---
>
> Key: HBASE-14443
> URL: https://issues.apache.org/jira/browse/HBASE-14443
> Project: HBase
>  Issue Type: Improvement
>  Components: Operability, rpc
>Affects Versions: 1.2.1
>Reporter: Jianwei Cui
>Assignee: Jianwei Cui
>Priority: Minor
> Attachments: HBASE-14443-trunk-v1.patch
>
>
> The RpcServer will log a warn message for TooSlow or TooLarge request as:
> {code}
> logResponse(new Object[]{param},
> md.getName(), md.getName() + "(" + param.getClass().getName() + 
> ")",
> (tooLarge ? "TooLarge" : "TooSlow"),
> status.getClient(), startTime, processingTime, qTime,
> responseSize);
> {code}
> The RpcServer#logResponse will create the warn message as:
> {code}
> if (params.length == 2 && server instanceof HRegionServer &&
> params[0] instanceof byte[] &&
> params[1] instanceof Operation) {
>   ...
>   responseInfo.putAll(((Operation) params[1]).toMap());
>   ...
> } else if (params.length == 1 && server instanceof HRegionServer &&
> params[0] instanceof Operation) {
>   ...
>   responseInfo.putAll(((Operation) params[0]).toMap());
>   ...
> } else {
>   ...
> }
> {code}
> Because the parameter is always a protobuf message, not an instance of 
> Operation, the request parameter will not be added into the warn message. The 
> parameter is helpful to find out the problem, for example, knowing the 
> startRow/endRow is useful for a TooSlow scan. To improve the warn message, we 
> can transform the protobuf request message to corresponding Operation 
> subclass object by ProtobufUtil, so that it can be added the warn message. 
> Suggestion and discussion are welcomed.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14391) Empty regionserver WAL will never be deleted although the coresponding regionserver has been stale

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901513#comment-14901513
 ] 

Jerry He commented on HBASE-14391:
--

I ran the patch on a cluster. It works well.
Committed to branch 1.1+

Thanks for the patch [~qianxiZhang]. Thanks for the review [~stack].

> Empty regionserver WAL will never be deleted although the coresponding 
> regionserver has been stale
> --
>
> Key: HBASE-14391
> URL: https://issues.apache.org/jira/browse/HBASE-14391
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 1.0.2
>Reporter: Qianxi Zhang
>Assignee: Qianxi Zhang
> Attachments: HBASE-14391-master-v3.patch, 
> HBASE_14391_master_v4.patch, HBASE_14391_trunk_v1.patch, 
> HBASE_14391_trunk_v2.patch, WALs-leftover-dir.txt
>
>
> When I restarted the hbase cluster in which there was few data, I found there 
> are two directories for one host with different timestamp which indicates 
> that the old regionserver wal directory is not deleted.
> FHLog#989
> {code}
>  @Override
>   public void close() throws IOException {
> shutdown();
> final FileStatus[] files = getFiles();
> if (null != files && 0 != files.length) {
>   for (FileStatus file : files) {
> Path p = getWALArchivePath(this.fullPathArchiveDir, file.getPath());
> // Tell our listeners that a log is going to be archived.
> if (!this.listeners.isEmpty()) {
>   for (WALActionsListener i : this.listeners) {
> i.preLogArchive(file.getPath(), p);
>   }
> }
> if (!FSUtils.renameAndSetModifyTime(fs, file.getPath(), p)) {
>   throw new IOException("Unable to rename " + file.getPath() + " to " 
> + p);
> }
> // Tell our listeners that a log was archived.
> if (!this.listeners.isEmpty()) {
>   for (WALActionsListener i : this.listeners) {
> i.postLogArchive(file.getPath(), p);
>   }
> }
>   }
>   LOG.debug("Moved " + files.length + " WAL file(s) to " +
> FSUtils.getPath(this.fullPathArchiveDir));
> }
> LOG.info("Closed WAL: " + toString());
>   }
> {code}
> When regionserver is stopped, the hlog will be archived, so wal/regionserver 
> is empty in hdfs.
> MasterFileSystem#252
> {code}
> if (curLogFiles == null || curLogFiles.length == 0) {
> // Empty log folder. No recovery needed
> continue;
>   }
> {code}
> The regionserver directory will be not splitted, it makes sense. But it will 
> be not deleted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14370) Use separate thread for calling ZKPermissionWatcher#refreshNodes()

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-14370:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

This has been committed, the issue should be resolved as Fixed.

> Use separate thread for calling ZKPermissionWatcher#refreshNodes()
> --
>
> Key: HBASE-14370
> URL: https://issues.apache.org/jira/browse/HBASE-14370
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.15, 1.0.3, 1.1.3
>
> Attachments: 14370-0.98-v10.txt, 14370-branch-1-v10.txt, 
> 14370-branch-1-v10.txt, 14370-v1.txt, 14370-v10.txt, 14370-v3.txt, 
> 14370-v5.txt, 14370-v7.txt, 14370-v8.txt, 14370-wait-nofity-v2.txt, 
> 14370-wait-nofity.txt, hbase-14370_v4.patch, test-acl3-branch-1.stack
>
>
> I came off a support case (0.98.0) where main zk thread was seen doing the 
> following:
> {code}
>   at 
> org.apache.hadoop.hbase.security.access.ZKPermissionWatcher.refreshAuthManager(ZKPermissionWatcher.java:152)
>   at 
> org.apache.hadoop.hbase.security.access.ZKPermissionWatcher.refreshNodes(ZKPermissionWatcher.java:135)
>   at 
> org.apache.hadoop.hbase.security.access.ZKPermissionWatcher.nodeChildrenChanged(ZKPermissionWatcher.java:121)
>   at 
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:348)
>   at 
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
>   at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
> {code}
> There were 62000 nodes under /acl due to lack of fix from HBASE-12635, 
> leading to slowness in table creation because zk notification for region 
> offline was blocked by the above.
> The attached patch separates refreshNodes() call into its own thread.
> Thanks to Enis and Devaraj for offline discussion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14451) Move on to htrace-4.0.0 (from htrace-3.2.0)

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901518#comment-14901518
 ] 

Hadoop QA commented on HBASE-14451:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12761498/14451v5.txt
  against master branch at commit 8765ffb0ca2b1c39d864ac495f0b3ee2abc520ed.
  ATTACHMENT ID: 12761498

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 29 new 
or modified tests.

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15655//console

This message is automatically generated.

> Move on to htrace-4.0.0 (from htrace-3.2.0)
> ---
>
> Key: HBASE-14451
> URL: https://issues.apache.org/jira/browse/HBASE-14451
> Project: HBase
>  Issue Type: Task
>Reporter: stack
>Assignee: stack
> Attachments: 14451.txt, 14451v2.txt, 14451v3.txt, 14451v4.txt, 
> 14451v5.txt
>
>
> htrace-4.0.0 was just release with a new API. Get up on it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14370) Use separate thread for calling ZKPermissionWatcher#refreshNodes()

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-14370:
---
Fix Version/s: (was: 1.1.3)
   (was: 1.0.3)
   (was: 1.3.0)
   (was: 1.2.0)

Update fix versions when committed to other branches

> Use separate thread for calling ZKPermissionWatcher#refreshNodes()
> --
>
> Key: HBASE-14370
> URL: https://issues.apache.org/jira/browse/HBASE-14370
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0, 0.98.15
>
> Attachments: 14370-0.98-v10.txt, 14370-branch-1-v10.txt, 
> 14370-branch-1-v10.txt, 14370-v1.txt, 14370-v10.txt, 14370-v3.txt, 
> 14370-v5.txt, 14370-v7.txt, 14370-v8.txt, 14370-wait-nofity-v2.txt, 
> 14370-wait-nofity.txt, hbase-14370_v4.patch, test-acl3-branch-1.stack
>
>
> I came off a support case (0.98.0) where main zk thread was seen doing the 
> following:
> {code}
>   at 
> org.apache.hadoop.hbase.security.access.ZKPermissionWatcher.refreshAuthManager(ZKPermissionWatcher.java:152)
>   at 
> org.apache.hadoop.hbase.security.access.ZKPermissionWatcher.refreshNodes(ZKPermissionWatcher.java:135)
>   at 
> org.apache.hadoop.hbase.security.access.ZKPermissionWatcher.nodeChildrenChanged(ZKPermissionWatcher.java:121)
>   at 
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:348)
>   at 
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
>   at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
> {code}
> There were 62000 nodes under /acl due to lack of fix from HBASE-12635, 
> leading to slowness in table creation because zk notification for region 
> offline was blocked by the above.
> The attached patch separates refreshNodes() call into its own thread.
> Thanks to Enis and Devaraj for offline discussion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14391) Empty regionserver WAL will never be deleted although the coresponding regionserver has been stale

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-14391:
-
 Hadoop Flags: Reviewed
Fix Version/s: 1.1.3
   1.3.0
   1.2.0
   2.0.0

> Empty regionserver WAL will never be deleted although the coresponding 
> regionserver has been stale
> --
>
> Key: HBASE-14391
> URL: https://issues.apache.org/jira/browse/HBASE-14391
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 1.0.2
>Reporter: Qianxi Zhang
>Assignee: Qianxi Zhang
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>
> Attachments: HBASE-14391-master-v3.patch, 
> HBASE_14391_master_v4.patch, HBASE_14391_trunk_v1.patch, 
> HBASE_14391_trunk_v2.patch, WALs-leftover-dir.txt
>
>
> When I restarted the hbase cluster in which there was few data, I found there 
> are two directories for one host with different timestamp which indicates 
> that the old regionserver wal directory is not deleted.
> FHLog#989
> {code}
>  @Override
>   public void close() throws IOException {
> shutdown();
> final FileStatus[] files = getFiles();
> if (null != files && 0 != files.length) {
>   for (FileStatus file : files) {
> Path p = getWALArchivePath(this.fullPathArchiveDir, file.getPath());
> // Tell our listeners that a log is going to be archived.
> if (!this.listeners.isEmpty()) {
>   for (WALActionsListener i : this.listeners) {
> i.preLogArchive(file.getPath(), p);
>   }
> }
> if (!FSUtils.renameAndSetModifyTime(fs, file.getPath(), p)) {
>   throw new IOException("Unable to rename " + file.getPath() + " to " 
> + p);
> }
> // Tell our listeners that a log was archived.
> if (!this.listeners.isEmpty()) {
>   for (WALActionsListener i : this.listeners) {
> i.postLogArchive(file.getPath(), p);
>   }
> }
>   }
>   LOG.debug("Moved " + files.length + " WAL file(s) to " +
> FSUtils.getPath(this.fullPathArchiveDir));
> }
> LOG.info("Closed WAL: " + toString());
>   }
> {code}
> When regionserver is stopped, the hlog will be archived, so wal/regionserver 
> is empty in hdfs.
> MasterFileSystem#252
> {code}
> if (curLogFiles == null || curLogFiles.length == 0) {
> // Empty log folder. No recovery needed
> continue;
>   }
> {code}
> The regionserver directory will be not splitted, it makes sense. But it will 
> be not deleted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14391) Empty regionserver WAL will never be deleted although the coresponding regionserver has been stale

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-14391:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Empty regionserver WAL will never be deleted although the coresponding 
> regionserver has been stale
> --
>
> Key: HBASE-14391
> URL: https://issues.apache.org/jira/browse/HBASE-14391
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 1.0.2
>Reporter: Qianxi Zhang
>Assignee: Qianxi Zhang
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>
> Attachments: HBASE-14391-master-v3.patch, 
> HBASE_14391_master_v4.patch, HBASE_14391_trunk_v1.patch, 
> HBASE_14391_trunk_v2.patch, WALs-leftover-dir.txt
>
>
> When I restarted the hbase cluster in which there was few data, I found there 
> are two directories for one host with different timestamp which indicates 
> that the old regionserver wal directory is not deleted.
> FHLog#989
> {code}
>  @Override
>   public void close() throws IOException {
> shutdown();
> final FileStatus[] files = getFiles();
> if (null != files && 0 != files.length) {
>   for (FileStatus file : files) {
> Path p = getWALArchivePath(this.fullPathArchiveDir, file.getPath());
> // Tell our listeners that a log is going to be archived.
> if (!this.listeners.isEmpty()) {
>   for (WALActionsListener i : this.listeners) {
> i.preLogArchive(file.getPath(), p);
>   }
> }
> if (!FSUtils.renameAndSetModifyTime(fs, file.getPath(), p)) {
>   throw new IOException("Unable to rename " + file.getPath() + " to " 
> + p);
> }
> // Tell our listeners that a log was archived.
> if (!this.listeners.isEmpty()) {
>   for (WALActionsListener i : this.listeners) {
> i.postLogArchive(file.getPath(), p);
>   }
> }
>   }
>   LOG.debug("Moved " + files.length + " WAL file(s) to " +
> FSUtils.getPath(this.fullPathArchiveDir));
> }
> LOG.info("Closed WAL: " + toString());
>   }
> {code}
> When regionserver is stopped, the hlog will be archived, so wal/regionserver 
> is empty in hdfs.
> MasterFileSystem#252
> {code}
> if (curLogFiles == null || curLogFiles.length == 0) {
> // Empty log folder. No recovery needed
> continue;
>   }
> {code}
> The regionserver directory will be not splitted, it makes sense. But it will 
> be not deleted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14355) Scan different TimeRange for each column family

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-14355:
---
Fix Version/s: (was: 0.98.15)
   0.98.16

> Scan different TimeRange for each column family
> ---
>
> Key: HBASE-14355
> URL: https://issues.apache.org/jira/browse/HBASE-14355
> Project: HBase
>  Issue Type: New Feature
>  Components: Client, regionserver, Scanners
>Reporter: Dave Latham
> Fix For: 2.0.0, 1.3.0, 0.98.16
>
>
> At present the Scan API supports only table level time range. We have 
> specific use cases that will benefit from per column family time range. (See 
> background discussion at 
> https://mail-archives.apache.org/mod_mbox/hbase-user/201508.mbox/%3ccaa4mzom00ef5eoxstk0hetxeby8mqss61gbvgttgpaspmhq...@mail.gmail.com%3E)
> There are a couple of choices that would be good to validate.  First - how to 
> update the Scan API to support family and table level updates.  One proposal 
> would be to add Scan.setTimeRange(byte family, long minTime, long maxTime), 
> then store it in a Map.  When executing the scan, if a 
> family has a specified TimeRange, then use it, otherwise fall back to using 
> the table level TimeRange.  Clients using the new API against old region 
> servers would not get the families correctly filterd.  Old clients sending 
> scans to new region servers would work correctly.
> The other question is how to get StoreFileScanner.shouldUseScanner to match 
> up the proper family and time range.  It has the Scan available but doesn't 
> currently have available which family it is a part of.  One option would be 
> to try to pass down the column family in each constructor path.  Another 
> would be to instead alter shouldUseScanner to pass down the specific 
> TimeRange to use (similar to how it currently passes down the columns to use 
> which also appears to be a workaround for not having the family available). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14129) If any regionserver gets shutdown uncleanly during full cluster restart, locality looks to be lost

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-14129:
---
Fix Version/s: (was: 0.98.15)
   0.98.16

> If any regionserver gets shutdown uncleanly during full cluster restart, 
> locality looks to be lost
> --
>
> Key: HBASE-14129
> URL: https://issues.apache.org/jira/browse/HBASE-14129
> Project: HBase
>  Issue Type: Bug
>Reporter: churro morales
> Fix For: 2.0.0, 1.3.0, 0.98.16
>
> Attachments: HBASE-14129.patch
>
>
> We were doing a cluster restart the other day.  Some regionservers did not 
> shut down cleanly.  Upon restart our locality went from 99% to 5%.  Upon 
> looking at the AssignmentManager.joinCluster() code it calls 
> AssignmentManager.processDeadServersAndRegionsInTransition().
> If the failover flag gets set for any reason it seems we don't call 
> assignAllUserRegions().  Then it looks like the balancer does the work in 
> assigning those regions, we don't use a locality aware balancer and we lost 
> our region locality.
> I don't have a solid grasp on the reasoning for these checks but there could 
> be some potential workarounds here.
> 1. After shutting down your cluster, move your WALs aside (replay later).  
> 2. Clean up your zNodes 
> That seems to work, but requires a lot of manual labor.  Another solution 
> which I prefer would be to have a flag for ./start-hbase.sh --clean 
> If we start master with that flag then we do a check in 
> AssignmentManager.processDeadServersAndRegionsInTransition()  thus if this 
> flag is set we call: assignAllUserRegions() regardless of the failover state.
> I have a patch for the later solution, that is if I am understanding the 
> logic correctly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14289) Backport HBASE-13965 'Stochastic Load Balancer JMX Metrics' to 0.98

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-14289:
---
Fix Version/s: (was: 0.98.15)
   0.98.16

> Backport HBASE-13965 'Stochastic Load Balancer JMX Metrics' to 0.98
> ---
>
> Key: HBASE-14289
> URL: https://issues.apache.org/jira/browse/HBASE-14289
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 0.98.16
>
> Attachments: 14289-0.98-v2.txt, 14289-0.98-v3.txt, 14289-0.98-v4.txt, 
> 14289-0.98-v5.txt
>
>
> The default HBase load balancer (the Stochastic load balancer) is cost 
> function based. The cost function weights are tunable but no visibility into 
> those cost function results is directly provided.
> This issue backports HBASE-13965 to 0.98 branch to provide visibility via JMX 
> into each cost function of the stochastic load balancer, as well as the 
> overall cost of the balancing plan.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14276) NPE when setting up stub for connection to master if secure connect is refused

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-14276:
---
Fix Version/s: (was: 0.98.15)
   0.98.16

> NPE when setting up stub for connection to master if secure connect is refused
> --
>
> Key: HBASE-14276
> URL: https://issues.apache.org/jira/browse/HBASE-14276
> Project: HBase
>  Issue Type: Bug
>Reporter: Andrew Purtell
>Priority: Minor
> Fix For: 0.98.16
>
>
> If an insecure client contacts a secure cluster we can get an NPE when 
> setting up the stub for the master connection. We should throw an IOException 
> with a clear message, and not retry if possible to distinguish this case.
> Found in 0.98 but might be relevant for later branches
> {noformat}
> Aug 20, 2015 12:04:31 AM 
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper 
> INFO: Process identifier=hconnection-0x3c1e23ff connecting to ZooKeeper 
> ensemble=x.x.x.x:2181,x.x.x.x:2181,x.x.x.x:2181
> Aug 20, 2015 12:04:31 AM 
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation 
> makeStub
> INFO: getMaster attempt 1 of 35 failed; retrying after sleep of 100, 
> exception=com.google.protobuf.ServiceException: java.lang.NullPointerException
> Aug 20, 2015 12:04:31 AM 
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation 
> makeStub
> INFO: getMaster attempt 2 of 35 failed; retrying after sleep of 200, 
> exception=com.google.protobuf.ServiceException: java.io.IOException: Call to 
> /x.x.x.x:6 failed on local exception: java.io.EOFException
> Aug 20, 2015 12:04:31 AM 
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation 
> makeStub
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14374) Backport parent 'HBASE-14317 Stuck FSHLog' issue to 1.1 and 1.0

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901531#comment-14901531
 ] 

Hadoop QA commented on HBASE-14374:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12761499/14374.branch-1.1.v6.txt
  against branch-1.1 branch at commit 8765ffb0ca2b1c39d864ac495f0b3ee2abc520ed.
  ATTACHMENT ID: 12761499

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 6 new 
or modified tests.

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15656//console

This message is automatically generated.

> Backport parent 'HBASE-14317 Stuck FSHLog' issue to 1.1 and 1.0
> ---
>
> Key: HBASE-14374
> URL: https://issues.apache.org/jira/browse/HBASE-14374
> Project: HBase
>  Issue Type: Sub-task
>Reporter: stack
>Assignee: stack
> Fix For: 1.0.3, 1.1.3
>
> Attachments: 14317-branch-1.1.txt, 14317.branch-1.1.v2.txt, 
> 14317.branch-1.1.v2.txt, 14317.branch-1.1.v2.txt, 14317.branch-1.1.v6.txt, 
> 14374.branch-1.1.v3.txt, 14374.branch-1.1.v4.txt, 14374.branch-1.1.v4.txt, 
> 14374.branch-1.1.v4.txt, 14374.branch-1.1.v4.txt, 14374.branch-1.1.v4.txt, 
> 14374.branch-1.1.v4.txt, 14374.branch-1.1.v5.txt, 14374.branch-1.1.v6.txt, 
> 14374.branch-1.v6.txt
>
>
> Backport parent issue to branch-1.1. and branch-1.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-14079) improve error message when Master fails to connect to Hadoop-auth

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell resolved HBASE-14079.

   Resolution: Duplicate
 Assignee: (was: Matt Warhaftig)
Fix Version/s: (was: 1.1.3)
   (was: 1.0.3)
   (was: 1.2.1)
   (was: 0.98.15)
   (was: 1.3.0)
   (was: 0.94.28)
   (was: 2.0.0)

Resolving as dup. Please reopen if I'm mistaken

> improve error message when Master fails to connect to Hadoop-auth
> -
>
> Key: HBASE-14079
> URL: https://issues.apache.org/jira/browse/HBASE-14079
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 0.94.0, 0.98.0, 1.0.0, 1.1.0
>Reporter: Sean Busbey
>
> Current error message at INFO level doesn't give any hint about what keytab 
> and principle are in use
> {quote}
> 2015-07-14 13:32:48,514 INFO  [main] impl.MetricsConfig: loaded properties 
> from hadoop-metrics2-hbase.properties
> 2015-07-14 13:32:48,640 INFO  [main] impl.MetricsSystemImpl: Scheduled 
> snapshot period at 10 second(s).
> 2015-07-14 13:32:48,640 INFO  [main] impl.MetricsSystemImpl: HBase metrics 
> system started
> 2015-07-14 13:32:48,776 ERROR [main] master.HMasterCommandLine: Master exiting
> java.lang.RuntimeException: Failed construction of Master: class 
> org.apache.hadoop.hbase.master.HMaster
> at 
> org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2258)
> at 
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:234)
> at 
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at 
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2272)
> Caused by: javax.security.auth.login.LoginException: Unable to obtain 
> password from user
> at 
> com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:856)
> at 
> com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:719)
> at 
> com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:584)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
> at 
> javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
> at java.security.AccessController.doPrivileged(Native Method)
> at 
> javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
> at javax.security.auth.login.LoginContext.login(LoginContext.java:595)
> at 
> org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:912)
> at 
> org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:242)
> at 
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:385)
> at org.apache.hadoop.hbase.security.User.login(User.java:252)
> at 
> org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:115)
> at org.apache.hadoop.hbase.master.HMaster.login(HMaster.java:464)
> at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:553)
> at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:351)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at 
> org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2253)
> ... 5 more
> {quote}
> increasing to DEBUG also doesn't help.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14067) bundle ruby files for hbase shell into a jar.

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-14067:
---
Fix Version/s: (was: 0.98.15)
   0.98.16

> bundle ruby files for hbase shell into a jar.
> -
>
> Key: HBASE-14067
> URL: https://issues.apache.org/jira/browse/HBASE-14067
> Project: HBase
>  Issue Type: Improvement
>  Components: shell
>Reporter: Sean Busbey
> Fix For: 2.0.0, 1.3.0, 0.98.16
>
>
> We currently package all the ruby scripts for the hbase shell by placing them 
> in a directory within lib/. We should be able to put these in a jar file 
> since we rely on jruby.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12751) Allow RowLock to be reader writer

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-12751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-12751:
--
Hadoop Flags: Incompatible change,Reviewed
Release Note: 
Locks on row are now reader/writer rather than exclusive.

Moves sequenceid out of HRegion and into MVCC class; MVCC is now in charge. A 
WAL append is still stamped in same way (we pass MVCC context in a few places 
where we previously we did not).

MVCC methods cleaned up. Make a bit more sense now. Less of them.

Simplifies our update of MemStore/WAL. Now we update memstore AFTER we add to 
WAL (but before we sync). This fixes possible dataloss when two edits came in 
with same coordinates; we could order the edits in memstore differently to how 
they arrived in the WAL.

Marked as an incompatible change because it breaks Distributed Log Replay, a 
feature we'd determined already was unreliable and to be removed (See 
http://search-hadoop.com/m/YGbbhTJpoal8GD1).

> Allow RowLock to be reader writer
> -
>
> Key: HBASE-12751
> URL: https://issues.apache.org/jira/browse/HBASE-12751
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.3.0
>
> Attachments: 12751.rebased.v25.txt, 12751.rebased.v26.txt, 
> 12751.rebased.v26.txt, 12751.rebased.v27.txt, 12751.rebased.v29.txt, 
> 12751.rebased.v31.txt, 12751.rebased.v32.txt, 12751.rebased.v32.txt, 
> 12751.rebased.v33.txt, 12751.rebased.v34.txt, 12751.rebased.v35.txt, 
> 12751.rebased.v35.txt, 12751.rebased.v35.txt, 12751.v37.txt, 12751.v38.txt, 
> 12751.v39.txt, 12751v22.txt, 12751v23.txt, 12751v23.txt, 12751v23.txt, 
> 12751v23.txt, 12751v36.txt, HBASE-12751-v1.patch, HBASE-12751-v10.patch, 
> HBASE-12751-v10.patch, HBASE-12751-v11.patch, HBASE-12751-v12.patch, 
> HBASE-12751-v13.patch, HBASE-12751-v14.patch, HBASE-12751-v15.patch, 
> HBASE-12751-v16.patch, HBASE-12751-v17.patch, HBASE-12751-v18.patch, 
> HBASE-12751-v19 (1).patch, HBASE-12751-v19.patch, HBASE-12751-v2.patch, 
> HBASE-12751-v20.patch, HBASE-12751-v20.patch, HBASE-12751-v21.patch, 
> HBASE-12751-v3.patch, HBASE-12751-v4.patch, HBASE-12751-v5.patch, 
> HBASE-12751-v6.patch, HBASE-12751-v7.patch, HBASE-12751-v8.patch, 
> HBASE-12751-v9.patch, HBASE-12751.patch
>
>
> Right now every write operation grabs a row lock. This is to prevent values 
> from changing during a read modify write operation (increment or check and 
> put). However it limits parallelism in several different scenarios.
> If there are several puts to the same row but different columns or stores 
> then this is very limiting.
> If there are puts to the same column then mvcc number should ensure a 
> consistent ordering. So locking is not needed.
> However locking for check and put or increment is still needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14147) REST Support for Namespaces

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901941#comment-14901941
 ] 

Hudson commented on HBASE-14147:


FAILURE: Integrated in HBase-1.3 #191 (See 
[https://builds.apache.org/job/HBase-1.3/191/])
HBASE-14147 Add namespace CRUD functionality to REST (apurtell: rev 
05bd89b9b781d7c108858d883aae7f08fe1fce32)
* 
hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/provider/JAXBContextResolver.java
* 
hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/model/NamespacesInstanceModel.java
* 
hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/protobuf/generated/NamespacesMessage.java
* 
hbase-rest/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/NamespacePropertiesMessage.proto
* 
hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/protobuf/generated/NamespacePropertiesMessage.java
* 
hbase-rest/src/main/resources/org/apache/hadoop/hbase/rest/protobuf/NamespacesMessage.proto
* 
hbase-rest/src/test/java/org/apache/hadoop/hbase/rest/model/TestNamespacesModel.java
* 
hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/NamespacesInstanceResource.java
* 
hbase-rest/src/test/java/org/apache/hadoop/hbase/rest/TestNamespacesInstanceResource.java
* 
hbase-rest/src/test/java/org/apache/hadoop/hbase/rest/model/TestNamespacesInstanceModel.java
* hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/RootResource.java
* hbase-rest/src/main/resources/org/apache/hadoop/hbase/rest/XMLSchema.xsd
* hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/NamespacesResource.java
* 
hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/model/NamespacesModel.java
* 
hbase-rest/src/test/java/org/apache/hadoop/hbase/rest/TestNamespacesResource.java
* hbase-rest/pom.xml


> REST Support for Namespaces
> ---
>
> Key: HBASE-14147
> URL: https://issues.apache.org/jira/browse/HBASE-14147
> Project: HBase
>  Issue Type: Sub-task
>  Components: REST
>Affects Versions: 1.1.1
>Reporter: Rick Kellogg
>Assignee: Matt Warhaftig
>Priority: Minor
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.15
>
> Attachments: SuccessfulLocalBuild.txt, hbase-14147-v1.patch, 
> hbase-14147-v1.patch, hbase-14147-v1.patch
>
>
> Expand REST services to include addition features:
> * Create namespace
> * Alter namespace
> * Describe namespace
> * Drop namespace
> * List tables in a specific namespace
> * List all namespaces.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)