[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707954#comment-13707954
 ] 

Ted Yu commented on HBASE-8884:
---

I don't have other comments.

> Pluggable RpcScheduler
> --
>
> Key: HBASE-8884
> URL: https://issues.apache.org/jira/browse/HBASE-8884
> Project: HBase
>  Issue Type: Improvement
>  Components: IPC/RPC
>Reporter: Chao Shi
>Assignee: Chao Shi
> Attachments: hbase-8884.patch, hbase-8884-v2.patch, 
> hbase-8884-v3.patch, hbase-8884-v4.patch, hbase-8884-v5.patch, 
> hbase-8884-v6.patch, hbase-8884-v7.patch
>
>
> Today, the RPC scheduling mechanism is pretty simple: it execute requests in 
> isolated thread-pools based on their priority. In the current implementation, 
> all normal get/put requests are using the same pool. We'd like to add some 
> per-user or per-region level isolation, so that a misbehaved user/region will 
> not saturate the thread-pool and cause DoS to others easily. The idea is 
> similar to FairScheduler in MR. The current scheduling code is not standalone 
> and is mixed with others (Connection#processRequest). The issue is the first 
> step to extract it to an interface, so that people are free to write and test 
> their own implementations.
> This patch doesn't make it completely pluggable yet, as some parameters are 
> pass from constructor. This is because HMaster and HRegionServer both use 
> RpcServer and they have different thread-pool size config. Let me know if you 
> have a solution to this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707951#comment-13707951
 ] 

Hadoop QA commented on HBASE-8884:
--

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12592181/hbase-8884-v7.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 12 new 
or modified tests.

{color:green}+1 hadoop1.0{color}.  The patch compiles against the hadoop 
1.0 profile.

{color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 
2.0 profile.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6328//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6328//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6328//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6328//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6328//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6328//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6328//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6328//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6328//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6328//console

This message is automatically generated.

> Pluggable RpcScheduler
> --
>
> Key: HBASE-8884
> URL: https://issues.apache.org/jira/browse/HBASE-8884
> Project: HBase
>  Issue Type: Improvement
>  Components: IPC/RPC
>Reporter: Chao Shi
>Assignee: Chao Shi
> Attachments: hbase-8884.patch, hbase-8884-v2.patch, 
> hbase-8884-v3.patch, hbase-8884-v4.patch, hbase-8884-v5.patch, 
> hbase-8884-v6.patch, hbase-8884-v7.patch
>
>
> Today, the RPC scheduling mechanism is pretty simple: it execute requests in 
> isolated thread-pools based on their priority. In the current implementation, 
> all normal get/put requests are using the same pool. We'd like to add some 
> per-user or per-region level isolation, so that a misbehaved user/region will 
> not saturate the thread-pool and cause DoS to others easily. The idea is 
> similar to FairScheduler in MR. The current scheduling code is not standalone 
> and is mixed with others (Connection#processRequest). The issue is the first 
> step to extract it to an interface, so that people are free to write and test 
> their own implementations.
> This patch doesn't make it completely pluggable yet, as some parameters are 
> pass from constructor. This is because HMaster and HRegionServer both use 
> RpcServer and they have different thread-pool size config. Let me know if you 
> have a solution to this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8884) Pluggable RpcScheduler

[Chao Shi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chao Shi updated HBASE-8884:


Attachment: hbase-8884-v7.patch

Thanks for review. Here is v7 patch.

- make RpcServer.CallRunner and RpcScheduler package private
- fix comments in RpcScheduler and SimpleRpcScheduler

> Pluggable RpcScheduler
> --
>
> Key: HBASE-8884
> URL: https://issues.apache.org/jira/browse/HBASE-8884
> Project: HBase
>  Issue Type: Improvement
>  Components: IPC/RPC
>Reporter: Chao Shi
>Assignee: Chao Shi
> Attachments: hbase-8884.patch, hbase-8884-v2.patch, 
> hbase-8884-v3.patch, hbase-8884-v4.patch, hbase-8884-v5.patch, 
> hbase-8884-v6.patch, hbase-8884-v7.patch
>
>
> Today, the RPC scheduling mechanism is pretty simple: it execute requests in 
> isolated thread-pools based on their priority. In the current implementation, 
> all normal get/put requests are using the same pool. We'd like to add some 
> per-user or per-region level isolation, so that a misbehaved user/region will 
> not saturate the thread-pool and cause DoS to others easily. The idea is 
> similar to FairScheduler in MR. The current scheduling code is not standalone 
> and is mixed with others (Connection#processRequest). The issue is the first 
> step to extract it to an interface, so that people are free to write and test 
> their own implementations.
> This patch doesn't make it completely pluggable yet, as some parameters are 
> pass from constructor. This is because HMaster and HRegionServer both use 
> RpcServer and they have different thread-pool size config. Let me know if you 
> have a solution to this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8873) minor concurrent issue about filesCompacting

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707935#comment-13707935
 ] 

Ted Yu commented on HBASE-8873:
---

Have you run patch through 0.94 test suite ?

Please post results here.

> minor concurrent issue about filesCompacting
> 
>
> Key: HBASE-8873
> URL: https://issues.apache.org/jira/browse/HBASE-8873
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.95.1, 0.94.9
>Reporter: Liang Xie
>Assignee: Liang Xie
> Attachments: HBase-8873-0.94.txt
>
>
> i am reading compaction code, seems there's a minor thread-safe issue on 
> filesCompacting in both 0.94 and trunk, we guard it with synchronized/lock, 
> except needsCompaction() function.  and the fix should sth like this:
> synchronized (filesCompacting) {
> ...
> }

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8648) Call to equals() comparing different types in HRegionServer.handleReportForDutyResponse()

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-8648:
--

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Looks like HBASE-8696 has fixed this.

> Call to equals() comparing different types in 
> HRegionServer.handleReportForDutyResponse()
> -
>
> Key: HBASE-8648
> URL: https://issues.apache.org/jira/browse/HBASE-8648
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Attachments: 8648-v1.txt, 8648-v2.txt
>
>
> Here is related code:
> {code}
>   if (!this.serverNameFromMasterPOV.equals(this.isa.getHostName())) {
> {code}
> serverNameFromMasterPOV is of type ServerName but this.isa.getHostName() 
> returns String

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8930) Filter evaluates KVs outside requested columns

[Lars Hofhansl (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707920#comment-13707920
 ] 

Lars Hofhansl commented on HBASE-8930:
--

This is (somewhat) by design. Filters are evaluated before the column trackers. 
The column trackers are responsible for the version counting and TTL processing 
(that part is by design - it used to be the other way round, but was changed 
because that caused more problems).
As a side-effect the columns are also processed after the filters have been 
evaluated.

Disentangling that is tedious and would lead to perf issues in other cases.
The place to change this is ExplicitColumnTracker.java. There would need to be 
a method that can check a column ahead of the filtering, but the tracker still 
needs to do versions/TTL after the filters ran.
Whatever the refactoring... if columns need to be compare twice that would be 
an unacceptable performance hit.

At some point I had experimented with letting a filter decide whether it is 
evaluated before or after the column trackers. Maybe I should restart that. See 
HBASE-5257.

This seems at odds with HBASE-4364.

In the interim, the filters should do their own column filtering.


> Filter evaluates KVs outside requested columns
> --
>
> Key: HBASE-8930
> URL: https://issues.apache.org/jira/browse/HBASE-8930
> Project: HBase
>  Issue Type: Bug
>  Components: Filters
>Affects Versions: 0.94.7
>Reporter: Federico Gaule
>Priority: Critical
>  Labels: filters, hbase, keyvalue
>
> 1- Fill row with some columns
> 2- Get row with some columns less than universe - Use filter to print kvs
> 3- Filter prints not requested columns
> Filter (AllwaysNextColFilter) always return ReturnCode.INCLUDE_AND_NEXT_COL 
> and prints KV's qualifier
> SUFFIX_0 = 0
> SUFFIX_1 = 1
> SUFFIX_4 = 4
> SUFFIX_6 = 6
> P= Persisted
> R= Requested
> E= Evaluated
> X= Returned
> | 5580 | 5581 | 5584 | 5586 | 5590 | 5591 | 5594 | 5596 | 5600 | 5601 | 5604 
> | 5606 |... 
> |  |  P   |   P  |  |  |  P   |   P  |  |  |  P   |   P  
> |  |...
> |  |  R   |   R  |   R  |  |  R   |   R  |   R  |  |  |  
> |  |...
> |  |  E   |   E  |  |  |  E   |   E  |  |  |  
> {color:red}E{color}   |  |  |...
> |  |  X   |   X  |  |  |  X   |   X  |  |  |  |  
> |  |
> {code:title=ExtraColumnTest.java|borderStyle=solid}
> @Test
> public void testFilter() throws Exception {
> Configuration config = HBaseConfiguration.create();
> config.set("hbase.zookeeper.quorum", "myZK");
> HTable hTable = new HTable(config, "testTable");
> byte[] cf = Bytes.toBytes("cf");
> byte[] row = Bytes.toBytes("row");
> byte[] col1 = new QualifierConverter().objectToByteArray(new 
> Qualifier((short) 558, (byte) SUFFIX_1));
> byte[] col2 = new QualifierConverter().objectToByteArray(new 
> Qualifier((short) 559, (byte) SUFFIX_1));
> byte[] col3 = new QualifierConverter().objectToByteArray(new 
> Qualifier((short) 560, (byte) SUFFIX_1));
> byte[] col4 = new QualifierConverter().objectToByteArray(new 
> Qualifier((short) 561, (byte) SUFFIX_1));
> byte[] col5 = new QualifierConverter().objectToByteArray(new 
> Qualifier((short) 562, (byte) SUFFIX_1));
> byte[] col6 = new QualifierConverter().objectToByteArray(new 
> Qualifier((short) 563, (byte) SUFFIX_1));
> byte[] col1g = new QualifierConverter().objectToByteArray(new 
> Qualifier((short) 558, (byte) SUFFIX_6));
> byte[] col2g = new QualifierConverter().objectToByteArray(new 
> Qualifier((short) 559, (byte) SUFFIX_6));
> byte[] col1v = new QualifierConverter().objectToByteArray(new 
> Qualifier((short) 558, (byte) SUFFIX_4));
> byte[] col2v = new QualifierConverter().objectToByteArray(new 
> Qualifier((short) 559, (byte) SUFFIX_4));
> byte[] col3v = new QualifierConverter().objectToByteArray(new 
> Qualifier((short) 560, (byte) SUFFIX_4));
> byte[] col4v = new QualifierConverter().objectToByteArray(new 
> Qualifier((short) 561, (byte) SUFFIX_4));
> byte[] col5v = new QualifierConverter().objectToByteArray(new 
> Qualifier((short) 562, (byte) SUFFIX_4));
> byte[] col6v = new QualifierConverter().objectToByteArray(new 
> Qualifier((short) 563, (byte) SUFFIX_4));
> // === INSERTION =//
> Put put = new Put(row);
> put.add(cf, col1, Bytes.toBytes((short) 1));
> put.add(cf, col2, Bytes.toBytes((short) 1));
> put.add(cf, col3, Bytes.toBytes((short) 3));
> put.add(cf, col4, Bytes.toBytes((short) 3));
> put.add(cf, col5, Bytes.toBytes((short) 3));
> put.

[jira] [Commented] (HBASE-8927) Use nano time instead of mili time everywhere

[Jean-Marc Spaggiari (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707916#comment-13707916
 ] 

Jean-Marc Spaggiari commented on HBASE-8927:


I'm not getting the points against this change.

We will not be the first one to move from Milli to Nano, no?

https://issues.apache.org/jira/browse/CASSANDRA-733

The issues we might have we nano, we already have them with milli?

> Use nano time instead of mili time everywhere
> -
>
> Key: HBASE-8927
> URL: https://issues.apache.org/jira/browse/HBASE-8927
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
> Attachments: 8927.txt
>
>
> Less collisions and we are paying the price of a long anyways so might as 
> well fill it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8806) Row locks are acquired repeatedly in HRegion.doMiniBatchMutation for duplicate rows.

[Dave Latham (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707917#comment-13707917
 ] 

Dave Latham commented on HBASE-8806:


[~lhofhansl] Agreed, don't see a reason it should be much slower in that case.  
Also, you can see the stddev of the measurements is high there, so my guess is 
it's related to not so good benchmarking (not enough burn in time).

However, please check out the work in HBASE-8877 as there's an alternate patch 
there for 0.94 as well.

> Row locks are acquired repeatedly in HRegion.doMiniBatchMutation for 
> duplicate rows.
> 
>
> Key: HBASE-8806
> URL: https://issues.apache.org/jira/browse/HBASE-8806
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.94.5
>Reporter: rahul gidwani
>Priority: Critical
> Fix For: 0.98.0, 0.95.2, 0.94.10
>
> Attachments: 8806-0.94-v4.txt, 8806-0.94-v5.txt, 8806-0.94-v6.txt, 
> HBASE-8806-0.94.10.patch, HBASE-8806-0.94.10-v2.patch, 
> HBASE-8806-0.94.10-v3.patch, HBASE-8806.patch, 
> HBASE-8806-threadBasedRowLocks.patch, 
> HBASE-8806-threadBasedRowLocks-v2.patch, row_lock_perf_results.txt
>
>
> If we already have the lock in the doMiniBatchMutation we don't need to 
> re-acquire it. The solution would be to keep a cache of the rowKeys already 
> locked for a miniBatchMutation and If we already have the 
> rowKey in the cache, we don't repeatedly try and acquire the lock.  A fix to 
> this problem would be to keep a set of rows we already locked and not try to 
> acquire the lock for these rows.  
> We have tested this fix in our production environment and has improved 
> replication performance quite a bit.  We saw a replication batch go from 3+ 
> minutes to less than 10 seconds for batches with duplicate row keys.
> {code}
> static final int ACQUIRE_LOCK_COUNT = 0;
>   @Test
>   public void testRedundantRowKeys() throws Exception {
> final int batchSize = 10;
> 
> String tableName = getClass().getSimpleName();
> Configuration conf = HBaseConfiguration.create();
> conf.setClass(HConstants.REGION_IMPL, MockHRegion.class, HeapSize.class);
> MockHRegion region = (MockHRegion) 
> TestHRegion.initHRegion(Bytes.toBytes(tableName), tableName, conf, 
> Bytes.toBytes("a"));
> List> someBatch = Lists.newArrayList();
> int i = 0;
> while (i < batchSize) {
>   if (i % 2 == 0) {
> someBatch.add(new Pair(new Put(Bytes.toBytes(0)), 
> null));
>   } else {
> someBatch.add(new Pair(new Put(Bytes.toBytes(1)), 
> null));
>   }
>   i++;
> }
> long startTime = System.currentTimeMillis();
> region.batchMutate(someBatch.toArray(new Pair[0]));
> long endTime = System.currentTimeMillis();
> long duration = endTime - startTime;
> System.out.println("duration: " + duration + " ms");
> assertEquals(2, ACQUIRE_LOCK_COUNT);
>   }
>   @Override
>   public Integer getLock(Integer lockid, byte[] row, boolean waitForLock) 
> throws IOException {
> ACQUIRE_LOCK_COUNT++;
> return super.getLock(lockid, row, waitForLock);
>   }
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8873) minor concurrent issue about filesCompacting

[Liang Xie (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707915#comment-13707915
 ] 

Liang Xie commented on HBASE-8873:
--

hi, [~yuzhih...@gmail.com], it's just a 0.94 issue, in my opinion.

> minor concurrent issue about filesCompacting
> 
>
> Key: HBASE-8873
> URL: https://issues.apache.org/jira/browse/HBASE-8873
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.95.1, 0.94.9
>Reporter: Liang Xie
>Assignee: Liang Xie
> Attachments: HBase-8873-0.94.txt
>
>
> i am reading compaction code, seems there's a minor thread-safe issue on 
> filesCompacting in both 0.94 and trunk, we guard it with synchronized/lock, 
> except needsCompaction() function.  and the fix should sth like this:
> synchronized (filesCompacting) {
> ...
> }

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8648) Call to equals() comparing different types in HRegionServer.handleReportForDutyResponse()

[chunhui shen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707914#comment-13707914
 ] 

chunhui shen commented on HBASE-8648:
-

{code}
this.serverNameFromMasterPOV = new ServerName(hostnameFromMasterPOV,
this.isa.getPort(), this.startcode);
-  if (!hostnameFromMasterPOV.equals(this.isa.getHostName())) {
+  if (!this.serverNameFromMasterPOV.getHostname()
+  .equals(this.isa.getHostName())) {

{code}
>From the patch,  the description code {code}
if (!this.serverNameFromMasterPOV.equals(this.isa.getHostName())) {
{code} 
seems not matched where the fact code is {code}if 
(!hostnameFromMasterPOV.equals(this.isa.getHostName())) {
{code}

> Call to equals() comparing different types in 
> HRegionServer.handleReportForDutyResponse()
> -
>
> Key: HBASE-8648
> URL: https://issues.apache.org/jira/browse/HBASE-8648
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Attachments: 8648-v1.txt, 8648-v2.txt
>
>
> Here is related code:
> {code}
>   if (!this.serverNameFromMasterPOV.equals(this.isa.getHostName())) {
> {code}
> serverNameFromMasterPOV is of type ServerName but this.isa.getHostName() 
> returns String

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8945) Remove ColumnTracker.update() method

[Lars Hofhansl (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-8945:
-

Attachment: 8945-0.94.txt

Trivial patch (0.94 version).

> Remove ColumnTracker.update() method
> 
>
> Key: HBASE-8945
> URL: https://issues.apache.org/jira/browse/HBASE-8945
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
> Fix For: 0.98.0, 0.95.2, 0.94.10
>
> Attachments: 8945-0.94.txt
>
>
> ColumnTracker.update() is no longer called from anywhere but a test; we 
> should remove it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-8945) Remove ColumnTracker.update() method

[Lars Hofhansl (JIRA)]

Lars Hofhansl created HBASE-8945:


 Summary: Remove ColumnTracker.update() method
 Key: HBASE-8945
 URL: https://issues.apache.org/jira/browse/HBASE-8945
 Project: HBase
  Issue Type: Bug
Reporter: Lars Hofhansl
 Fix For: 0.98.0, 0.95.2, 0.94.10
 Attachments: 8945-0.94.txt

ColumnTracker.update() is no longer called from anywhere but a test; we should 
remove it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8806) Row locks are acquired repeatedly in HRegion.doMiniBatchMutation for duplicate rows.

[Lars Hofhansl (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707897#comment-13707897
 ] 

Lars Hofhansl commented on HBASE-8806:
--

Looking at the numbers again this looks like a measurement glitch. Only the 
case with no duplicates is slow, and there is no reason why it would be slower 
all the cases with duplicates.


> Row locks are acquired repeatedly in HRegion.doMiniBatchMutation for 
> duplicate rows.
> 
>
> Key: HBASE-8806
> URL: https://issues.apache.org/jira/browse/HBASE-8806
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.94.5
>Reporter: rahul gidwani
>Priority: Critical
> Fix For: 0.98.0, 0.95.2, 0.94.10
>
> Attachments: 8806-0.94-v4.txt, 8806-0.94-v5.txt, 8806-0.94-v6.txt, 
> HBASE-8806-0.94.10.patch, HBASE-8806-0.94.10-v2.patch, 
> HBASE-8806-0.94.10-v3.patch, HBASE-8806.patch, 
> HBASE-8806-threadBasedRowLocks.patch, 
> HBASE-8806-threadBasedRowLocks-v2.patch, row_lock_perf_results.txt
>
>
> If we already have the lock in the doMiniBatchMutation we don't need to 
> re-acquire it. The solution would be to keep a cache of the rowKeys already 
> locked for a miniBatchMutation and If we already have the 
> rowKey in the cache, we don't repeatedly try and acquire the lock.  A fix to 
> this problem would be to keep a set of rows we already locked and not try to 
> acquire the lock for these rows.  
> We have tested this fix in our production environment and has improved 
> replication performance quite a bit.  We saw a replication batch go from 3+ 
> minutes to less than 10 seconds for batches with duplicate row keys.
> {code}
> static final int ACQUIRE_LOCK_COUNT = 0;
>   @Test
>   public void testRedundantRowKeys() throws Exception {
> final int batchSize = 10;
> 
> String tableName = getClass().getSimpleName();
> Configuration conf = HBaseConfiguration.create();
> conf.setClass(HConstants.REGION_IMPL, MockHRegion.class, HeapSize.class);
> MockHRegion region = (MockHRegion) 
> TestHRegion.initHRegion(Bytes.toBytes(tableName), tableName, conf, 
> Bytes.toBytes("a"));
> List> someBatch = Lists.newArrayList();
> int i = 0;
> while (i < batchSize) {
>   if (i % 2 == 0) {
> someBatch.add(new Pair(new Put(Bytes.toBytes(0)), 
> null));
>   } else {
> someBatch.add(new Pair(new Put(Bytes.toBytes(1)), 
> null));
>   }
>   i++;
> }
> long startTime = System.currentTimeMillis();
> region.batchMutate(someBatch.toArray(new Pair[0]));
> long endTime = System.currentTimeMillis();
> long duration = endTime - startTime;
> System.out.println("duration: " + duration + " ms");
> assertEquals(2, ACQUIRE_LOCK_COUNT);
>   }
>   @Override
>   public Integer getLock(Integer lockid, byte[] row, boolean waitForLock) 
> throws IOException {
> ACQUIRE_LOCK_COUNT++;
> return super.getLock(lockid, row, waitForLock);
>   }
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8806) Row locks are acquired repeatedly in HRegion.doMiniBatchMutation for duplicate rows.

[Lars Hofhansl (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707896#comment-13707896
 ] 

Lars Hofhansl commented on HBASE-8806:
--

[~davelatham] and [~churromorales], is my reading of the numbers correct?
This cannot be committed if it slows down the "normal" case by almost 100%.

Also, did you test this exact patch (with the slight interface change that 
avoid creating HashedBytes multiple times)?


> Row locks are acquired repeatedly in HRegion.doMiniBatchMutation for 
> duplicate rows.
> 
>
> Key: HBASE-8806
> URL: https://issues.apache.org/jira/browse/HBASE-8806
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.94.5
>Reporter: rahul gidwani
>Priority: Critical
> Fix For: 0.98.0, 0.95.2, 0.94.10
>
> Attachments: 8806-0.94-v4.txt, 8806-0.94-v5.txt, 8806-0.94-v6.txt, 
> HBASE-8806-0.94.10.patch, HBASE-8806-0.94.10-v2.patch, 
> HBASE-8806-0.94.10-v3.patch, HBASE-8806.patch, 
> HBASE-8806-threadBasedRowLocks.patch, 
> HBASE-8806-threadBasedRowLocks-v2.patch, row_lock_perf_results.txt
>
>
> If we already have the lock in the doMiniBatchMutation we don't need to 
> re-acquire it. The solution would be to keep a cache of the rowKeys already 
> locked for a miniBatchMutation and If we already have the 
> rowKey in the cache, we don't repeatedly try and acquire the lock.  A fix to 
> this problem would be to keep a set of rows we already locked and not try to 
> acquire the lock for these rows.  
> We have tested this fix in our production environment and has improved 
> replication performance quite a bit.  We saw a replication batch go from 3+ 
> minutes to less than 10 seconds for batches with duplicate row keys.
> {code}
> static final int ACQUIRE_LOCK_COUNT = 0;
>   @Test
>   public void testRedundantRowKeys() throws Exception {
> final int batchSize = 10;
> 
> String tableName = getClass().getSimpleName();
> Configuration conf = HBaseConfiguration.create();
> conf.setClass(HConstants.REGION_IMPL, MockHRegion.class, HeapSize.class);
> MockHRegion region = (MockHRegion) 
> TestHRegion.initHRegion(Bytes.toBytes(tableName), tableName, conf, 
> Bytes.toBytes("a"));
> List> someBatch = Lists.newArrayList();
> int i = 0;
> while (i < batchSize) {
>   if (i % 2 == 0) {
> someBatch.add(new Pair(new Put(Bytes.toBytes(0)), 
> null));
>   } else {
> someBatch.add(new Pair(new Put(Bytes.toBytes(1)), 
> null));
>   }
>   i++;
> }
> long startTime = System.currentTimeMillis();
> region.batchMutate(someBatch.toArray(new Pair[0]));
> long endTime = System.currentTimeMillis();
> long duration = endTime - startTime;
> System.out.println("duration: " + duration + " ms");
> assertEquals(2, ACQUIRE_LOCK_COUNT);
>   }
>   @Override
>   public Integer getLock(Integer lockid, byte[] row, boolean waitForLock) 
> throws IOException {
> ACQUIRE_LOCK_COUNT++;
> return super.getLock(lockid, row, waitForLock);
>   }
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8803) region_mover.rb should move multiple regions at a time

[Jean-Marc Spaggiari (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707891#comment-13707891
 ] 

Jean-Marc Spaggiari commented on HBASE-8803:


That's weird. This "8" is not coming from anywhere. Not the default, not the 
parameter...

I stopped my cluster because we have a heatwave and it's not more an house that 
I have, it's a sauna ;) I will restart that tomorrow morning and retry. I will 
keep you posted.

> region_mover.rb should move multiple regions at a time
> --
>
> Key: HBASE-8803
> URL: https://issues.apache.org/jira/browse/HBASE-8803
> Project: HBase
>  Issue Type: Bug
>  Components: Usability
>Affects Versions: 0.98.0, 0.94.8, 0.95.1
>Reporter: Jean-Marc Spaggiari
>Assignee: Jean-Marc Spaggiari
> Fix For: 0.98.0, 0.95.2, 0.94.10
>
> Attachments: HBASE-8803-v0-trunk.patch, HBASE-8803-v1-0.94.patch, 
> HBASE-8803-v1-trunk.patch, HBASE-8803-v2-0.94.patch, HBASE-8803-v2-0.94.patch
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> When there is many regions in a cluster, rolling_restart can take hours 
> because region_mover is moving the regions one by one.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8938) Thrift2 does not close scanner instance

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707888#comment-13707888
 ] 

Hudson commented on HBASE-8938:
---

FAILURE: Integrated in hbase-0.95-on-hadoop2 #178 (See 
[https://builds.apache.org/job/hbase-0.95-on-hadoop2/178/])
HBASE-8938 Thrift2 does not close scanner instance (Hamed Madani) (larsgeorge: 
rev 1502797)
* 
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/thrift2/ThriftHBaseServiceHandler.java


> Thrift2 does not close scanner instance
> ---
>
> Key: HBASE-8938
> URL: https://issues.apache.org/jira/browse/HBASE-8938
> Project: HBase
>  Issue Type: Sub-task
>  Components: Thrift
>Affects Versions: 0.94.0
>Reporter: Hamed Madani
>Assignee: Hamed Madani
>  Labels: thrift2
> Fix For: 0.98.0, 0.95.2, 0.94.10
>
> Attachments: HBASE-8938-0.94.patch, HBASE-8938.patch
>
>
> closeScanner at *ThriftHBaseServiceHandler* simply remove scannerId from 
> internal map, but fails to close the scanner instance. 
> {code}
>   public void closeScanner(int scannerId) throws TIOError, TIllegalArgument, 
> TException {
> if (removeScanner(scannerId) == null) {
>   TIllegalArgument ex = new TIllegalArgument();
>   ex.setMessage("Invalid scanner Id");
>   throw ex;
> }
>   }
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8845) Add integration test for split, online merge, and compaction

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707879#comment-13707879
 ] 

Ted Yu commented on HBASE-8845:
---

Should taking snapshot be added as one ChaosMonkey action ?

> Add integration test for split, online merge, and compaction
> 
>
> Key: HBASE-8845
> URL: https://issues.apache.org/jira/browse/HBASE-8845
> Project: HBase
>  Issue Type: Test
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Fix For: 0.95.2
>
> Attachments: trunk-8845.patch, trunk-8845-v2.patch
>
>
> We should add an integration test for online merge to exercise online merging 
> while chaos money kills/restarts region servers, moves regions around.  We 
> need to make sure there is no data loss no matter online merge succeeds/fails.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8798) Fix a minor bug in shell command with clone_snapshot table error

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-8798:
--

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Fix a minor bug in shell command with clone_snapshot table error
> 
>
> Key: HBASE-8798
> URL: https://issues.apache.org/jira/browse/HBASE-8798
> Project: HBase
>  Issue Type: Bug
>  Components: shell, snapshots
>Affects Versions: 0.94.8, 0.95.1
>Reporter: Jerry He
>Assignee: Ted Yu
>Priority: Minor
> Fix For: 0.98.0
>
> Attachments: 8798-trunk-v2.txt, 8798-v3.txt, HBASE-8798-trunk.patch
>
>
> In HBase shell, the syntax for clone_snapshot is:
>   hbase> clone_snapshot 'snapshotName', 'tableName'
> If the target table already exists, we'll get an error.
> For example:
> --
> hbase(main):011:0> clone_snapshot 'mysnapshot1', 'TestTable'
> ERROR: Table already exists: mysnapshot1!
> Here is some help for this command:
> Create a new table by cloning the snapshot content.
> There're no copies of data involved.
> And writing on the newly created table will not influence the snapshot data.
> Examples:
>   hbase> clone_snapshot 'snapshotName', 'tableName'
> --
> The bug is in the ERROR message:
> *ERROR: Table already exists: mysnapshot1!*
> We should output the table name, not the snapshot name.
> Currently, in command.rb, we have the output fixed as args.first for 
> TableExistsException:
> {code}
>   def translate_hbase_exceptions(*args)
> yield
>   rescue org.apache.hadoop.hbase.exceptions.TableNotFoundException
> raise "Unknown table #{args.first}!"
>   rescue org.apache.hadoop.hbase.exceptions.NoSuchColumnFamilyException
> valid_cols = table(args.first).get_all_columns.map { |c| c + '*' }
> raise "Unknown column family! Valid column names: 
> #{valid_cols.join(", ")}"
>   rescue org.apache.hadoop.hbase.exceptions.TableExistsException
> raise "Table already exists: #{args.first}!"
>   end
> {code}
> This is fine with commands like 'create tableName ...' but not 
> 'clone_snapshot snapshotName tableName'.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8847) Filter.transform() always applies unconditionally, even when combined in a FilterList

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-8847:
--

Status: Open  (was: Patch Available)

> Filter.transform() always applies unconditionally, even when combined in a 
> FilterList
> -
>
> Key: HBASE-8847
> URL: https://issues.apache.org/jira/browse/HBASE-8847
> Project: HBase
>  Issue Type: Bug
>  Components: Filters
>Affects Versions: 0.94.2
>Reporter: Christophe Taton
>Assignee: Christophe Taton
>Priority: Minor
> Fix For: 0.98.0
>
> Attachments: HBASE-8847.base=trunk.diff, HBASE-8847.patch
>
>
> FilterList.transform() applies all transformation from all filters 
> unconditionally.
> This leads to surprising results, eg. with combinations such as
>   (family=X and column=Y and KeyOnlyFilter) or ...
> The KeyOnlyFilter will strip the values from all KeyValues.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8847) Filter.transform() always applies unconditionally, even when combined in a FilterList

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707877#comment-13707877
 ] 

Ted Yu commented on HBASE-8847:
---

For 0.94 backport, the guideline is that the patch should allow rolling-restart 
to work.
Meaning, the patch should be API-compatible.

> Filter.transform() always applies unconditionally, even when combined in a 
> FilterList
> -
>
> Key: HBASE-8847
> URL: https://issues.apache.org/jira/browse/HBASE-8847
> Project: HBase
>  Issue Type: Bug
>  Components: Filters
>Affects Versions: 0.94.2
>Reporter: Christophe Taton
>Assignee: Christophe Taton
>Priority: Minor
> Fix For: 0.98.0
>
> Attachments: HBASE-8847.base=trunk.diff, HBASE-8847.patch
>
>
> FilterList.transform() applies all transformation from all filters 
> unconditionally.
> This leads to surprising results, eg. with combinations such as
>   (family=X and column=Y and KeyOnlyFilter) or ...
> The KeyOnlyFilter will strip the values from all KeyValues.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8873) minor concurrent issue about filesCompacting

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707874#comment-13707874
 ] 

Ted Yu commented on HBASE-8873:
---

Patch makes sense.

Mind attaching patch for trunk ?

> minor concurrent issue about filesCompacting
> 
>
> Key: HBASE-8873
> URL: https://issues.apache.org/jira/browse/HBASE-8873
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.95.1, 0.94.9
>Reporter: Liang Xie
>Assignee: Liang Xie
> Attachments: HBase-8873-0.94.txt
>
>
> i am reading compaction code, seems there's a minor thread-safe issue on 
> filesCompacting in both 0.94 and trunk, we guard it with synchronized/lock, 
> except needsCompaction() function.  and the fix should sth like this:
> synchronized (filesCompacting) {
> ...
> }

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8875) incorrect javadoc for EXCLUDE_FROM_MINOR_COMPACTION

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707873#comment-13707873
 ] 

Ted Yu commented on HBASE-8875:
---

+1 on patch.

> incorrect javadoc for EXCLUDE_FROM_MINOR_COMPACTION
> ---
>
> Key: HBASE-8875
> URL: https://issues.apache.org/jira/browse/HBASE-8875
> Project: HBase
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Liang Xie
>Assignee: Liang Xie
>Priority: Trivial
> Attachments: HBase-8875.txt
>
>
> -  /** Major compaction flag in FileInfo */
> +  /** Minor compaction flag in FileInfo */
>public static final byte[] EXCLUDE_FROM_MINOR_COMPACTION_KEY =
>Bytes.toBytes("EXCLUDE_FROM_MINOR_COMPACTION");

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707871#comment-13707871
 ] 

Ted Yu commented on HBASE-8884:
---

Patch v6 looks good. Minor comments below.

For RpcScheduler interface, please add javadoc for the method parameters.

For SimpleRpcScheduler:
{code}
+ * A scheduler that maintains isolated handler pools for high-priority and 
replication
+ * requests.
{code}
There are actually 3 queues. Better make the above comment clearer.

RpcServer.CallRunner can be package private.

> Pluggable RpcScheduler
> --
>
> Key: HBASE-8884
> URL: https://issues.apache.org/jira/browse/HBASE-8884
> Project: HBase
>  Issue Type: Improvement
>  Components: IPC/RPC
>Reporter: Chao Shi
>Assignee: Chao Shi
> Attachments: hbase-8884.patch, hbase-8884-v2.patch, 
> hbase-8884-v3.patch, hbase-8884-v4.patch, hbase-8884-v5.patch, 
> hbase-8884-v6.patch
>
>
> Today, the RPC scheduling mechanism is pretty simple: it execute requests in 
> isolated thread-pools based on their priority. In the current implementation, 
> all normal get/put requests are using the same pool. We'd like to add some 
> per-user or per-region level isolation, so that a misbehaved user/region will 
> not saturate the thread-pool and cause DoS to others easily. The idea is 
> similar to FairScheduler in MR. The current scheduling code is not standalone 
> and is mixed with others (Connection#processRequest). The issue is the first 
> step to extract it to an interface, so that people are free to write and test 
> their own implementations.
> This patch doesn't make it completely pluggable yet, as some parameters are 
> pass from constructor. This is because HMaster and HRegionServer both use 
> RpcServer and they have different thread-pool size config. Let me know if you 
> have a solution to this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8940) TestRegionMergeTransactionOnCluster#testWholesomeMerge may fail due to race in opening region

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707861#comment-13707861
 ] 

stack commented on HBASE-8940:
--

Chatting w/ Jimmy friday afternoon, there is a little hole where we have 
published the region as available on regionserver X -- because it has been set 
in .META. and master has been informed the region has been opened via a 
transitioning of znode -- but the region may not be added to the regionserver 
online regions set just yet (it is done after we move region from OPENING to 
OPENED up in zk).  If a client comes in in the meantime, which is possible when 
all threads are running in the one jvm as we have in unit tests, client will 
get NotServingRegionException (or RegionOpeningException which is a subclass in 
this case)... which is the 'truth' in that we have not yet put up the region 
online.  The client will retry usually but in this case, in merge, there is no 
retry since it is the regionserver itself making a call on itself; there is no 
client.  Adding retries inside the regionserver seems wrong.  Regionserver 
knows its own state.

Talking w/ Jimmy, changing the order in which we online the region in the 
regionserver so we do it before we 'publish' via znode could open us up to 
races where region could be open in more than one place so it is safer to leave 
things as they are regards region onlining and instead just fix tests, or 
better, HBaseTestingUtility; i.e. not start the merge until for sure the 
regions are online.

Let me put up a patch.


> TestRegionMergeTransactionOnCluster#testWholesomeMerge may fail due to race 
> in opening region
> -
>
> Key: HBASE-8940
> URL: https://issues.apache.org/jira/browse/HBASE-8940
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 8940-v1.txt
>
>
> From 
> http://54.241.6.143/job/HBase-TRUNK-Hadoop-2/org.apache.hbase$hbase-server/395/testReport/org.apache.hadoop.hbase.regionserver/TestRegionMergeTransactionOnCluster/testWholesomeMerge/
>  :
> {code}
> 013-07-11 09:33:44,154 INFO  [AM.ZK.Worker-pool-2-thread-2] 
> master.RegionStates(309): Offlined 3ffefd878a234031675de6b2c70b2ead from 
> ip-10-174-118-204.us-west-1.compute.internal,60498,1373535184820
> 2013-07-11 09:33:44,154 INFO  [AM.ZK.Worker-pool-2-thread-2] 
> master.AssignmentManager$4(1223): The master has opened 
> testWholesomeMerge,testRow0020,1373535210125.3ffefd878a234031675de6b2c70b2ead.
>  that was online on 
> ip-10-174-118-204.us-west-1.compute.internal,59210,1373535184884
> 2013-07-11 09:33:44,182 DEBUG [RS_OPEN_REGION-ip-10-174-118-204:59210-1] 
> zookeeper.ZKAssign(862): regionserver:59210-0x13fcd13a20c0002 Successfully 
> transitioned node 3ffefd878a234031675de6b2c70b2ead from RS_ZK_REGION_OPENING 
> to RS_ZK_REGION_OPENED
> 2013-07-11 09:33:44,182 INFO  
> [MASTER_TABLE_OPERATIONS-ip-10-174-118-204:39405-0] 
> handler.DispatchMergingRegionHandler(154): Failed send MERGE REGIONS RPC to 
> server ip-10-174-118-204.us-west-1.compute.internal,59210,1373535184884 for 
> region 
> testWholesomeMerge,,1373535210124.efcb10dcfa250e31bfd50dc6c7049f32.,testWholesomeMerge,testRow0020,1373535210125.3ffefd878a234031675de6b2c70b2ead.,
>  focible=false, org.apache.hadoop.hbase.exceptions.RegionOpeningException: 
> Region is being opened: 3ffefd878a234031675de6b2c70b2ead
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2566)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3862)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.mergeRegions(HRegionServer.java:3649)
>   at 
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:14400)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2124)
>   at 
> org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1831)
> 2013-07-11 09:33:44,182 DEBUG [RS_OPEN_REGION-ip-10-174-118-204:59210-1] 
> handler.OpenRegionHandler(373): region transitioned to opened in zookeeper: 
> {ENCODED => 3ffefd878a234031675de6b2c70b2ead, NAME => 
> 'testWholesomeMerge,testRow0020,1373535210125.3ffefd878a234031675de6b2c70b2ead.',
>  STARTKEY => 'testRow0020', ENDKEY => 'testRow0040'}, server: 
> ip-10-174-118-204.us-west-1.compute.internal,59210,1373535184884
> 2013-07-11 09:33:44,183 DEBUG [RS_OPEN_REGION-ip-10-174-118-204:59210-1] 
> handler.OpenRegionHandler(186): Opened 
> testWholesomeMerge,testRow0020,1373535210125.3ffefd878a234031675de6b2c70b2ead.
>  on server:ip-10-174-118-204.us-west-1.compute.internal,59210,1373535184884
> {code}
> We can see that MASTER_TABLE_OPERATIONS thread couldn't get region 
> 3f

[jira] [Commented] (HBASE-8941) TestAccessController.testGlobalPermissionList failed with IndexOutOfBoundsException

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707857#comment-13707857
 ] 

stack commented on HBASE-8941:
--

Hmm...  I don't think this patch right.   Let me look at it again.

> TestAccessController.testGlobalPermissionList failed with 
> IndexOutOfBoundsException
> ---
>
> Key: HBASE-8941
> URL: https://issues.apache.org/jira/browse/HBASE-8941
> Project: HBase
>  Issue Type: Test
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Attachments: 8941.txt
>
>
> https://builds.apache.org/job/HBase-TRUNK/4246/testReport/junit/org.apache.hadoop.hbase.security.access/TestAccessController/testGlobalPermissionList/
> {noformat}
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
>   at java.util.ArrayList.RangeCheck(ArrayList.java:547)
>   at java.util.ArrayList.get(ArrayList.java:322)
>   at 
> org.apache.hadoop.hbase.security.access.TestAccessController.setUp(TestAccessController.java:188)
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8941) TestAccessController.testGlobalPermissionList failed with IndexOutOfBoundsException

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-8941:
-

Attachment: 8941.txt

Here is an addition that does a get on each region in the list of online 
regions.  It will retry until region is fully online.

It is crass.  May be should be done in //.

> TestAccessController.testGlobalPermissionList failed with 
> IndexOutOfBoundsException
> ---
>
> Key: HBASE-8941
> URL: https://issues.apache.org/jira/browse/HBASE-8941
> Project: HBase
>  Issue Type: Test
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Attachments: 8941.txt
>
>
> https://builds.apache.org/job/HBase-TRUNK/4246/testReport/junit/org.apache.hadoop.hbase.security.access/TestAccessController/testGlobalPermissionList/
> {noformat}
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
>   at java.util.ArrayList.RangeCheck(ArrayList.java:547)
>   at java.util.ArrayList.get(ArrayList.java:322)
>   at 
> org.apache.hadoop.hbase.security.access.TestAccessController.setUp(TestAccessController.java:188)
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8909) HBASE-6170 broke "hbase.regionserver.lease.period" config, we should support it for BC

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707839#comment-13707839
 ] 

stack commented on HBASE-8909:
--

+1

Make sure above test failure is transient before commit.

> HBASE-6170 broke "hbase.regionserver.lease.period" config, we should support 
> it for BC
> --
>
> Key: HBASE-8909
> URL: https://issues.apache.org/jira/browse/HBASE-8909
> Project: HBase
>  Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Critical
> Fix For: 0.98.0, 0.95.2
>
> Attachments: hbase-8909_v1.patch, hbase-8909_v2.patch
>
>
> HBASE-6170 replaces "hbase.regionserver.lease.period" with the new 
> configuration "hbase.client.scanner.timeout.period". We should support the 
> old configuration at least in 0.96. We can put a WARN and deprecate the old 
> value. Remove it in 1.0. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8943) Split Thrift2's ThriftServer into separate classes for easier testing and modularization

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707838#comment-13707838
 ] 

stack commented on HBASE-8943:
--

Can they share code?

> Split Thrift2's ThriftServer into separate classes for easier testing and 
> modularization
> 
>
> Key: HBASE-8943
> URL: https://issues.apache.org/jira/browse/HBASE-8943
> Project: HBase
>  Issue Type: Sub-task
>  Components: Thrift
>Reporter: Lars George
>Assignee: Lars George
>  Labels: thrift2
>
> Currently the ThriftServer class in Thrift 2 sets up and starts the actual 
> server. Better follow a similar pattern to Thrift 1 where there is some 
> factory setting up the server, and a separate start section. That way it is 
> easier to test if the setup of the server is picking up everything it needs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7035) thrift server closes HTable of open Scanners

[Lars George (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707828#comment-13707828
 ] 

Lars George commented on HBASE-7035:


Connections are reference counted, see 
[HConnectionManager|https://github.com/apache/hbase/blob/0.94/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java#L296]
 for reference. So given the issues we had with connection identities in there 
are fixed, we should be OK to close a table as the connection will be counted 
down? But then the scanner is a non-counted reference, but only when all tables 
are closed and the reference count ends up being zero it gets really closed. By 
that time scanners should be all done?

The other option we discussed is to hold on to the table for the scanner, but 
then we have one for every instance, which is way too much under load. We only 
need to hold on to one per table, therefore keeping the umbilical cord alive 
(so to speak).

> thrift server closes HTable of open Scanners
> 
>
> Key: HBASE-7035
> URL: https://issues.apache.org/jira/browse/HBASE-7035
> Project: HBase
>  Issue Type: Sub-task
>  Components: Thrift
>Affects Versions: 0.94.4
>Reporter: Devin Bayer
>  Labels: thrift2
> Attachments: old-hbase-thrift-v1.patch
>
>   Original Estimate: 4h
>  Remaining Estimate: 4h
>
> ThriftHBaseServiceHandler.openScanner() does this:
> 1. table = pool.getTable()
> 2. scanner = table.getScanner()
> 3. table.close()
> 4. return scanner
> While back porting the thrift server to 0.92.6, I found that table.close() 
> calls connection.close(). Further calls to scanner.next() raise a 
> ConnectionClosed exception. The unit tests do not catch this since they reuse 
> an open HConnection instance.
> This might work on trunk, but depends on the implementations of HTablePool, 
> HTable and HConnectionManager. Even with the pool wrapper, if the pool is 
> full, table.close() may be called, which may invalidate the table. Also,  
> HTable is not thread-safe, but they are being reused since they go back in 
> the pool.
> I suggest storing the table handle along with the scanner.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8648) Call to equals() comparing different types in HRegionServer.handleReportForDutyResponse()

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707818#comment-13707818
 ] 

Hadoop QA commented on HBASE-8648:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12592158/8648-v2.txt
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 hadoop1.0{color}.  The patch compiles against the hadoop 
1.0 profile.

{color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 
2.0 profile.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6327//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6327//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6327//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6327//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6327//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6327//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6327//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6327//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6327//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6327//console

This message is automatically generated.

> Call to equals() comparing different types in 
> HRegionServer.handleReportForDutyResponse()
> -
>
> Key: HBASE-8648
> URL: https://issues.apache.org/jira/browse/HBASE-8648
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Attachments: 8648-v1.txt, 8648-v2.txt
>
>
> Here is related code:
> {code}
>   if (!this.serverNameFromMasterPOV.equals(this.isa.getHostName())) {
> {code}
> serverNameFromMasterPOV is of type ServerName but this.isa.getHostName() 
> returns String

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7035) thrift server closes HTable of open Scanners

[Lars George (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707816#comment-13707816
 ] 

Lars George commented on HBASE-7035:


[~madani], ah sorry, you meant the PooledHTable wrapper handed out by the pool. 
Makes sense now. [~akvadrako] is right though, this has changed over time, but 
what we are concerned about today is all sharing the same wrapper, i.e. 
[HTablePool|https://github.com/apache/hbase/blob/0.94/src/main/java/org/apache/hadoop/hbase/client/HTablePool.java#L177]
 in 0.94's branch head.

So we are facing the above question then really, should we hold on to the table 
instance or not. I am looking it that now, but please let me know your thoughts.

> thrift server closes HTable of open Scanners
> 
>
> Key: HBASE-7035
> URL: https://issues.apache.org/jira/browse/HBASE-7035
> Project: HBase
>  Issue Type: Sub-task
>  Components: Thrift
>Affects Versions: 0.94.4
>Reporter: Devin Bayer
>  Labels: thrift2
> Attachments: old-hbase-thrift-v1.patch
>
>   Original Estimate: 4h
>  Remaining Estimate: 4h
>
> ThriftHBaseServiceHandler.openScanner() does this:
> 1. table = pool.getTable()
> 2. scanner = table.getScanner()
> 3. table.close()
> 4. return scanner
> While back porting the thrift server to 0.92.6, I found that table.close() 
> calls connection.close(). Further calls to scanner.next() raise a 
> ConnectionClosed exception. The unit tests do not catch this since they reuse 
> an open HConnection instance.
> This might work on trunk, but depends on the implementations of HTablePool, 
> HTable and HConnectionManager. Even with the pool wrapper, if the pool is 
> full, table.close() may be called, which may invalidate the table. Also,  
> HTable is not thread-safe, but they are being reused since they go back in 
> the pool.
> I suggest storing the table handle along with the scanner.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8938) Thrift2 does not close scanner instance

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707814#comment-13707814
 ] 

Hudson commented on HBASE-8938:
---

SUCCESS: Integrated in HBase-TRUNK #4251 (See 
[https://builds.apache.org/job/HBase-TRUNK/4251/])
HBASE-8938 Thrift2 does not close scanner instance (Hamed Madani) (larsgeorge: 
rev 1502798)
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/thrift2/ThriftHBaseServiceHandler.java


> Thrift2 does not close scanner instance
> ---
>
> Key: HBASE-8938
> URL: https://issues.apache.org/jira/browse/HBASE-8938
> Project: HBase
>  Issue Type: Sub-task
>  Components: Thrift
>Affects Versions: 0.94.0
>Reporter: Hamed Madani
>Assignee: Hamed Madani
>  Labels: thrift2
> Fix For: 0.98.0, 0.95.2, 0.94.10
>
> Attachments: HBASE-8938-0.94.patch, HBASE-8938.patch
>
>
> closeScanner at *ThriftHBaseServiceHandler* simply remove scannerId from 
> internal map, but fails to close the scanner instance. 
> {code}
>   public void closeScanner(int scannerId) throws TIOError, TIllegalArgument, 
> TException {
> if (removeScanner(scannerId) == null) {
>   TIllegalArgument ex = new TIllegalArgument();
>   ex.setMessage("Invalid scanner Id");
>   throw ex;
> }
>   }
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7035) thrift server closes HTable of open Scanners

[Lars George (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707811#comment-13707811
 ] 

Lars George commented on HBASE-7035:


[~madani], I am not sure what you are looking at. HTable has no reference to 
any HTablePool instance, so calling HTable.close does not return it to the 
pool. There is an internal pool of worker threads handled in HTable.close() but 
that is unrelated.

[~akvadrako], I see what you are saying - and of course we need to hold on to 
the table instances - either using HTablePool or as Thrift 1 does with our own 
list (which you do of sorts). 

Your patch does change the behaviour so that when a scanner is opened, the 
HTable instance is *not* returned to the pool, but only when you close the 
scanner. 

The first issue you are describing is that when you return the table, you might 
run into the issue that the table is reused although they are not thread safe. 
That should not be an issue since you are not using the table anymore, but a 
scanner instance. I think the second issue you describe is the problem, i.e. 
when a table is closed, the underlying connection is closed possibly, and 
therefore leaves the scanner dangling.

I need to look into 0.94, 0.95 and trunk/0.98 to see what the status (as you 
and Hamed touch upon above). If we decide to hang on, we could wrap this into 
HBASE-3852 which creates a Scanner wrapper, holding a last-used time and the 
actual result scanner. We could easily add the table instance there and hold on 
to it in one map. Same thing as yours, just a merge.



> thrift server closes HTable of open Scanners
> 
>
> Key: HBASE-7035
> URL: https://issues.apache.org/jira/browse/HBASE-7035
> Project: HBase
>  Issue Type: Sub-task
>  Components: Thrift
>Affects Versions: 0.94.4
>Reporter: Devin Bayer
>  Labels: thrift2
> Attachments: old-hbase-thrift-v1.patch
>
>   Original Estimate: 4h
>  Remaining Estimate: 4h
>
> ThriftHBaseServiceHandler.openScanner() does this:
> 1. table = pool.getTable()
> 2. scanner = table.getScanner()
> 3. table.close()
> 4. return scanner
> While back porting the thrift server to 0.92.6, I found that table.close() 
> calls connection.close(). Further calls to scanner.next() raise a 
> ConnectionClosed exception. The unit tests do not catch this since they reuse 
> an open HConnection instance.
> This might work on trunk, but depends on the implementations of HTablePool, 
> HTable and HConnectionManager. Even with the pool wrapper, if the pool is 
> full, table.close() may be called, which may invalidate the table. Also,  
> HTable is not thread-safe, but they are being reused since they go back in 
> the pool.
> I suggest storing the table handle along with the scanner.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8944) Runtime NullPointerException in MiniDFSCluster on CentOS and other Linux

[David Williams (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Williams updated HBASE-8944:
--

Description: 
The HBaseTestingUtility appears to have an unhandled NullPointerException in 
certain environments.   It works on Apache HBase 0.94.6.1 on Mac OSX but not in 
certain Linux environments such as CentOS. 

In Java:
{code}
HBaseTestingUtility testingUtility = new HBaseTestingUtility();
testingUtility.startMiniCluster(1);  // blows up on CentOS Linux but not Mac OSX
{code}

Creates this exception:
{code}
ERROR in (create-table) (MiniDFSCluster.java:426)
Uncaught exception, not in assertion.
expected: nil
  actual: java.lang.NullPointerException: null
 at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes 
(MiniDFSCluster.java:426)
org.apache.hadoop.hdfs.MiniDFSCluster. (MiniDFSCluster.java:284)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster 
(HBaseTestingUtility.java:444)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster 
(HBaseTestingUtility.java:612)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster 
(HBaseTestingUtility.java:568)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster 
(HBaseTestingUtility.java:555)
{code}

I also created a stack overflow ticket here, with more info:

http://stackoverflow.com/questions/17625938/hbase-minidfscluster-java-fails-in-certain-environments



  was:
The HBaseTestingUtility appears to have an unhandled NullPointerException in 
certain environments.   It works on Apache HBase 0.94.6.1 on Mac OSX but not in 
certain Linux environments such as CentOS. 

In Java:
{code}
HBaseTestingUtility testingUtility = new HBaseTestingUtility();
testingUtility.startMiniCluster(1);  // blows up on Linux but not Mac OSX
{code}

Creates this exception:
{code}
ERROR in (create-table) (MiniDFSCluster.java:426)
Uncaught exception, not in assertion.
expected: nil
  actual: java.lang.NullPointerException: null
 at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes 
(MiniDFSCluster.java:426)
org.apache.hadoop.hdfs.MiniDFSCluster. (MiniDFSCluster.java:284)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster 
(HBaseTestingUtility.java:444)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster 
(HBaseTestingUtility.java:612)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster 
(HBaseTestingUtility.java:568)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster 
(HBaseTestingUtility.java:555)
{code}

I also created a stack overflow ticket here, with more info:

http://stackoverflow.com/questions/17625938/hbase-minidfscluster-java-fails-in-certain-environments




> Runtime NullPointerException in MiniDFSCluster on CentOS and other Linux
> 
>
> Key: HBASE-8944
> URL: https://issues.apache.org/jira/browse/HBASE-8944
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.94.6, 0.94.6.1
>Reporter: David Williams
>
> The HBaseTestingUtility appears to have an unhandled NullPointerException in 
> certain environments.   It works on Apache HBase 0.94.6.1 on Mac OSX but not 
> in certain Linux environments such as CentOS. 
> In Java:
> {code}
> HBaseTestingUtility testingUtility = new HBaseTestingUtility();
> testingUtility.startMiniCluster(1);  // blows up on CentOS Linux but not Mac 
> OSX
> {code}
> Creates this exception:
> {code}
> ERROR in (create-table) (MiniDFSCluster.java:426)
> Uncaught exception, not in assertion.
> expected: nil
>   actual: java.lang.NullPointerException: null
>  at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes 
> (MiniDFSCluster.java:426)
> org.apache.hadoop.hdfs.MiniDFSCluster. (MiniDFSCluster.java:284)
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster 
> (HBaseTestingUtility.java:444)
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster 
> (HBaseTestingUtility.java:612)
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster 
> (HBaseTestingUtility.java:568)
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster 
> (HBaseTestingUtility.java:555)
> {code}
> I also created a stack overflow ticket here, with more info:
> http://stackoverflow.com/questions/17625938/hbase-minidfscluster-java-fails-in-certain-environments

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-8944) Runtime NullPointerException in MiniDFSCluster on CentOS and other Linux

[David Williams (JIRA)]

David Williams created HBASE-8944:
-

 Summary: Runtime NullPointerException in MiniDFSCluster on CentOS 
and other Linux
 Key: HBASE-8944
 URL: https://issues.apache.org/jira/browse/HBASE-8944
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.6.1, 0.94.6
Reporter: David Williams


The HBaseTestingUtility appears to have an unhandled NullPointerException in 
certain environments.   It works on Apache HBase 0.94.6.1 on Mac OSX but not in 
certain Linux environments such as CentOS. 

In Java:
{code}
HBaseTestingUtility testingUtility = new HBaseTestingUtility();
testingUtility.startMiniCluster(1);  // blows up on Linux but not Mac OSX
{code}

Creates this exception:
{code}
ERROR in (create-table) (MiniDFSCluster.java:426)
Uncaught exception, not in assertion.
expected: nil
  actual: java.lang.NullPointerException: null
 at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes 
(MiniDFSCluster.java:426)
org.apache.hadoop.hdfs.MiniDFSCluster. (MiniDFSCluster.java:284)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster 
(HBaseTestingUtility.java:444)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster 
(HBaseTestingUtility.java:612)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster 
(HBaseTestingUtility.java:568)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster 
(HBaseTestingUtility.java:555)
{code}

I also created a stack overflow ticket here, with more info:

http://stackoverflow.com/questions/17625938/hbase-minidfscluster-java-fails-in-certain-environments



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8938) Thrift2 does not close scanner instance

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707810#comment-13707810
 ] 

Hudson commented on HBASE-8938:
---

FAILURE: Integrated in hbase-0.95 #323 (See 
[https://builds.apache.org/job/hbase-0.95/323/])
HBASE-8938 Thrift2 does not close scanner instance (Hamed Madani) (larsgeorge: 
rev 1502797)
* 
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/thrift2/ThriftHBaseServiceHandler.java


> Thrift2 does not close scanner instance
> ---
>
> Key: HBASE-8938
> URL: https://issues.apache.org/jira/browse/HBASE-8938
> Project: HBase
>  Issue Type: Sub-task
>  Components: Thrift
>Affects Versions: 0.94.0
>Reporter: Hamed Madani
>Assignee: Hamed Madani
>  Labels: thrift2
> Fix For: 0.98.0, 0.95.2, 0.94.10
>
> Attachments: HBASE-8938-0.94.patch, HBASE-8938.patch
>
>
> closeScanner at *ThriftHBaseServiceHandler* simply remove scannerId from 
> internal map, but fails to close the scanner instance. 
> {code}
>   public void closeScanner(int scannerId) throws TIOError, TIllegalArgument, 
> TException {
> if (removeScanner(scannerId) == null) {
>   TIllegalArgument ex = new TIllegalArgument();
>   ex.setMessage("Invalid scanner Id");
>   throw ex;
> }
>   }
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8877) Reentrant row locks

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707809#comment-13707809
 ] 

Ted Yu commented on HBASE-8877:
---

+1 on patch v6.

> Reentrant row locks
> ---
>
> Key: HBASE-8877
> URL: https://issues.apache.org/jira/browse/HBASE-8877
> Project: HBase
>  Issue Type: Bug
>  Components: Coprocessors, regionserver
>Reporter: Dave Latham
>Assignee: Dave Latham
> Fix For: 0.95.2
>
> Attachments: hbase-8877-0.94-microbenchmark.txt, 
> HBASE-8877-0.94.patch, HBASE-8877.patch, HBASE-8877-v2.patch, 
> HBASE-8877-v3.patch, hbase-8877-v4-microbenchmark.txt, HBASE-8877-v4.patch, 
> HBASE-8877-v5.patch, HBASE-8877-v6.patch
>
>
> HBASE-8806 revealed performance problems with batch mutations failing to 
> reacquire the same row locks.  It looks like HBASE-8806 will use a less 
> intrusive change for 0.94 to have batch mutations track their own row locks 
> and not attempt to reacquire them.  Another approach will be to support 
> reentrant row locks directly.  This allows simplifying a great deal of 
> calling code to no longer track and pass around lock ids.
> One affect this change will have is changing the RegionObserver coprocessor's 
> methods preBatchMutate and postBatchMutate from taking a 
> {{MiniBatchOperationInProgress> miniBatchOp}} to 
> taking a {{MiniBatchOperationInProgress miniBatchOp}}.  I don't 
> believe CPs should be relying on these lock ids, but that's a potential 
> incompatibility.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8877) Reentrant row locks

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707808#comment-13707808
 ] 

Hadoop QA commented on HBASE-8877:
--

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12592154/HBASE-8877-v6.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 9 new 
or modified tests.

{color:green}+1 hadoop1.0{color}.  The patch compiles against the hadoop 
1.0 profile.

{color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 
2.0 profile.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6326//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6326//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6326//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6326//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6326//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6326//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6326//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6326//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6326//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6326//console

This message is automatically generated.

> Reentrant row locks
> ---
>
> Key: HBASE-8877
> URL: https://issues.apache.org/jira/browse/HBASE-8877
> Project: HBase
>  Issue Type: Bug
>  Components: Coprocessors, regionserver
>Reporter: Dave Latham
>Assignee: Dave Latham
> Fix For: 0.95.2
>
> Attachments: hbase-8877-0.94-microbenchmark.txt, 
> HBASE-8877-0.94.patch, HBASE-8877.patch, HBASE-8877-v2.patch, 
> HBASE-8877-v3.patch, hbase-8877-v4-microbenchmark.txt, HBASE-8877-v4.patch, 
> HBASE-8877-v5.patch, HBASE-8877-v6.patch
>
>
> HBASE-8806 revealed performance problems with batch mutations failing to 
> reacquire the same row locks.  It looks like HBASE-8806 will use a less 
> intrusive change for 0.94 to have batch mutations track their own row locks 
> and not attempt to reacquire them.  Another approach will be to support 
> reentrant row locks directly.  This allows simplifying a great deal of 
> calling code to no longer track and pass around lock ids.
> One affect this change will have is changing the RegionObserver coprocessor's 
> methods preBatchMutate and postBatchMutate from taking a 
> {{MiniBatchOperationInProgress> miniBatchOp}} to 
> taking a {{MiniBatchOperationInProgress miniBatchOp}}.  I don't 
> believe CPs should be relying on these lock ids, but that's a potential 
> incompatibility.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8648) Call to equals() comparing different types in HRegionServer.handleReportForDutyResponse()

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-8648:
--

Attachment: 8648-v2.txt

> Call to equals() comparing different types in 
> HRegionServer.handleReportForDutyResponse()
> -
>
> Key: HBASE-8648
> URL: https://issues.apache.org/jira/browse/HBASE-8648
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Attachments: 8648-v1.txt, 8648-v2.txt
>
>
> Here is related code:
> {code}
>   if (!this.serverNameFromMasterPOV.equals(this.isa.getHostName())) {
> {code}
> serverNameFromMasterPOV is of type ServerName but this.isa.getHostName() 
> returns String

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8648) Call to equals() comparing different types in HRegionServer.handleReportForDutyResponse()

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-8648:
--

Attachment: (was: 8648-v1.txt)

> Call to equals() comparing different types in 
> HRegionServer.handleReportForDutyResponse()
> -
>
> Key: HBASE-8648
> URL: https://issues.apache.org/jira/browse/HBASE-8648
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Attachments: 8648-v1.txt
>
>
> Here is related code:
> {code}
>   if (!this.serverNameFromMasterPOV.equals(this.isa.getHostName())) {
> {code}
> serverNameFromMasterPOV is of type ServerName but this.isa.getHostName() 
> returns String

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8938) Thrift2 does not close scanner instance

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707791#comment-13707791
 ] 

Hudson commented on HBASE-8938:
---

FAILURE: Integrated in HBase-0.94 #1056 (See 
[https://builds.apache.org/job/HBase-0.94/1056/])
HBASE-8938 Thrift2 does not close scanner instance (Hamed Madani) (larsgeorge: 
rev 1502799)
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/thrift2/ThriftHBaseServiceHandler.java


> Thrift2 does not close scanner instance
> ---
>
> Key: HBASE-8938
> URL: https://issues.apache.org/jira/browse/HBASE-8938
> Project: HBase
>  Issue Type: Sub-task
>  Components: Thrift
>Affects Versions: 0.94.0
>Reporter: Hamed Madani
>Assignee: Hamed Madani
>  Labels: thrift2
> Fix For: 0.98.0, 0.95.2, 0.94.10
>
> Attachments: HBASE-8938-0.94.patch, HBASE-8938.patch
>
>
> closeScanner at *ThriftHBaseServiceHandler* simply remove scannerId from 
> internal map, but fails to close the scanner instance. 
> {code}
>   public void closeScanner(int scannerId) throws TIOError, TIllegalArgument, 
> TException {
> if (removeScanner(scannerId) == null) {
>   TIllegalArgument ex = new TIllegalArgument();
>   ex.setMessage("Invalid scanner Id");
>   throw ex;
> }
>   }
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8877) Reentrant row locks

[Dave Latham (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707787#comment-13707787
 ] 

Dave Latham commented on HBASE-8877:


Note the above comment is referencing the HRegion class.

> Reentrant row locks
> ---
>
> Key: HBASE-8877
> URL: https://issues.apache.org/jira/browse/HBASE-8877
> Project: HBase
>  Issue Type: Bug
>  Components: Coprocessors, regionserver
>Reporter: Dave Latham
>Assignee: Dave Latham
> Fix For: 0.95.2
>
> Attachments: hbase-8877-0.94-microbenchmark.txt, 
> HBASE-8877-0.94.patch, HBASE-8877.patch, HBASE-8877-v2.patch, 
> HBASE-8877-v3.patch, hbase-8877-v4-microbenchmark.txt, HBASE-8877-v4.patch, 
> HBASE-8877-v5.patch, HBASE-8877-v6.patch
>
>
> HBASE-8806 revealed performance problems with batch mutations failing to 
> reacquire the same row locks.  It looks like HBASE-8806 will use a less 
> intrusive change for 0.94 to have batch mutations track their own row locks 
> and not attempt to reacquire them.  Another approach will be to support 
> reentrant row locks directly.  This allows simplifying a great deal of 
> calling code to no longer track and pass around lock ids.
> One affect this change will have is changing the RegionObserver coprocessor's 
> methods preBatchMutate and postBatchMutate from taking a 
> {{MiniBatchOperationInProgress> miniBatchOp}} to 
> taking a {{MiniBatchOperationInProgress miniBatchOp}}.  I don't 
> believe CPs should be relying on these lock ids, but that's a potential 
> incompatibility.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8877) Reentrant row locks

[Dave Latham (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dave Latham updated HBASE-8877:
---

Attachment: HBASE-8877-v6.patch

Last patch failed on TestHeapSize.  This surprised me since it had passed for 
me locally.  Turns out locally I was on a 64-bit VM and Jenkins ran it out a 
32-bit.  It also turns out the current trunk had the FIXED_OVERHEAD off by +1 
long, -1 reference and -2 booleans which just happened to pass the tests due to 
alignment but after this change began to fail.  Digging further it's obvious 
the DEEP_OVERHEAD is missing many fields as is the heapSize() method.  Looks 
like we may not be even trying any more.  Has there already been any discussion 
about this?

Attaching v6 patch.  In this patch I've corrected FIXED_OVERHEAD which gets the 
test passing.  I also added a couple missing AtomicIntegers to DEEP_OVERHEAD as 
well as a comment specifying all the other missing things which I think is too 
great a scope for me to try to fix in this patch, but at least serves as a 
warning to others. 

> Reentrant row locks
> ---
>
> Key: HBASE-8877
> URL: https://issues.apache.org/jira/browse/HBASE-8877
> Project: HBase
>  Issue Type: Bug
>  Components: Coprocessors, regionserver
>Reporter: Dave Latham
>Assignee: Dave Latham
> Fix For: 0.95.2
>
> Attachments: hbase-8877-0.94-microbenchmark.txt, 
> HBASE-8877-0.94.patch, HBASE-8877.patch, HBASE-8877-v2.patch, 
> HBASE-8877-v3.patch, hbase-8877-v4-microbenchmark.txt, HBASE-8877-v4.patch, 
> HBASE-8877-v5.patch, HBASE-8877-v6.patch
>
>
> HBASE-8806 revealed performance problems with batch mutations failing to 
> reacquire the same row locks.  It looks like HBASE-8806 will use a less 
> intrusive change for 0.94 to have batch mutations track their own row locks 
> and not attempt to reacquire them.  Another approach will be to support 
> reentrant row locks directly.  This allows simplifying a great deal of 
> calling code to no longer track and pass around lock ids.
> One affect this change will have is changing the RegionObserver coprocessor's 
> methods preBatchMutate and postBatchMutate from taking a 
> {{MiniBatchOperationInProgress> miniBatchOp}} to 
> taking a {{MiniBatchOperationInProgress miniBatchOp}}.  I don't 
> believe CPs should be relying on these lock ids, but that's a potential 
> incompatibility.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7035) thrift server closes HTable of open Scanners

[Devin Bayer (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707783#comment-13707783
 ] 

Devin Bayer commented on HBASE-7035:


Lars: That's a mistake - we should use HTablePool to avoid recreating the 
HTable instances.

Hamed: It depends on the HBase version. As this was about backporting to 
0.94.4, putTable() was needed.

> thrift server closes HTable of open Scanners
> 
>
> Key: HBASE-7035
> URL: https://issues.apache.org/jira/browse/HBASE-7035
> Project: HBase
>  Issue Type: Sub-task
>  Components: Thrift
>Affects Versions: 0.94.4
>Reporter: Devin Bayer
>  Labels: thrift2
> Attachments: old-hbase-thrift-v1.patch
>
>   Original Estimate: 4h
>  Remaining Estimate: 4h
>
> ThriftHBaseServiceHandler.openScanner() does this:
> 1. table = pool.getTable()
> 2. scanner = table.getScanner()
> 3. table.close()
> 4. return scanner
> While back porting the thrift server to 0.92.6, I found that table.close() 
> calls connection.close(). Further calls to scanner.next() raise a 
> ConnectionClosed exception. The unit tests do not catch this since they reuse 
> an open HConnection instance.
> This might work on trunk, but depends on the implementations of HTablePool, 
> HTable and HConnectionManager. Even with the pool wrapper, if the pool is 
> full, table.close() may be called, which may invalidate the table. Also,  
> HTable is not thread-safe, but they are being reused since they go back in 
> the pool.
> I suggest storing the table handle along with the scanner.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8938) Thrift2 does not close scanner instance

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707781#comment-13707781
 ] 

Hudson commented on HBASE-8938:
---

SUCCESS: Integrated in HBase-0.94-security #210 (See 
[https://builds.apache.org/job/HBase-0.94-security/210/])
HBASE-8938 Thrift2 does not close scanner instance (Hamed Madani) (larsgeorge: 
rev 1502799)
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/thrift2/ThriftHBaseServiceHandler.java


> Thrift2 does not close scanner instance
> ---
>
> Key: HBASE-8938
> URL: https://issues.apache.org/jira/browse/HBASE-8938
> Project: HBase
>  Issue Type: Sub-task
>  Components: Thrift
>Affects Versions: 0.94.0
>Reporter: Hamed Madani
>Assignee: Hamed Madani
>  Labels: thrift2
> Fix For: 0.98.0, 0.95.2, 0.94.10
>
> Attachments: HBASE-8938-0.94.patch, HBASE-8938.patch
>
>
> closeScanner at *ThriftHBaseServiceHandler* simply remove scannerId from 
> internal map, but fails to close the scanner instance. 
> {code}
>   public void closeScanner(int scannerId) throws TIOError, TIllegalArgument, 
> TException {
> if (removeScanner(scannerId) == null) {
>   TIllegalArgument ex = new TIllegalArgument();
>   ex.setMessage("Invalid scanner Id");
>   throw ex;
> }
>   }
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8877) Reentrant row locks

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707776#comment-13707776
 ] 

Hadoop QA commented on HBASE-8877:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12592151/HBASE-8877-v5.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 9 new 
or modified tests.

{color:green}+1 hadoop1.0{color}.  The patch compiles against the hadoop 
1.0 profile.

{color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 
2.0 profile.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
   org.apache.hadoop.hbase.io.TestHeapSize

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6325//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6325//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6325//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6325//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6325//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6325//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6325//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6325//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6325//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6325//console

This message is automatically generated.

> Reentrant row locks
> ---
>
> Key: HBASE-8877
> URL: https://issues.apache.org/jira/browse/HBASE-8877
> Project: HBase
>  Issue Type: Bug
>  Components: Coprocessors, regionserver
>Reporter: Dave Latham
>Assignee: Dave Latham
> Fix For: 0.95.2
>
> Attachments: hbase-8877-0.94-microbenchmark.txt, 
> HBASE-8877-0.94.patch, HBASE-8877.patch, HBASE-8877-v2.patch, 
> HBASE-8877-v3.patch, hbase-8877-v4-microbenchmark.txt, HBASE-8877-v4.patch, 
> HBASE-8877-v5.patch
>
>
> HBASE-8806 revealed performance problems with batch mutations failing to 
> reacquire the same row locks.  It looks like HBASE-8806 will use a less 
> intrusive change for 0.94 to have batch mutations track their own row locks 
> and not attempt to reacquire them.  Another approach will be to support 
> reentrant row locks directly.  This allows simplifying a great deal of 
> calling code to no longer track and pass around lock ids.
> One affect this change will have is changing the RegionObserver coprocessor's 
> methods preBatchMutate and postBatchMutate from taking a 
> {{MiniBatchOperationInProgress> miniBatchOp}} to 
> taking a {{MiniBatchOperationInProgress miniBatchOp}}.  I don't 
> believe CPs should be relying on these lock ids, but that's a potential 
> incompatibility.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8877) Reentrant row locks

[Dave Latham (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dave Latham updated HBASE-8877:
---

Attachment: HBASE-8877-v5.patch

Here's a v5 patch which merges in changes from trunk since v4 could no longer 
apply.

> Reentrant row locks
> ---
>
> Key: HBASE-8877
> URL: https://issues.apache.org/jira/browse/HBASE-8877
> Project: HBase
>  Issue Type: Bug
>  Components: Coprocessors, regionserver
>Reporter: Dave Latham
>Assignee: Dave Latham
> Fix For: 0.95.2
>
> Attachments: hbase-8877-0.94-microbenchmark.txt, 
> HBASE-8877-0.94.patch, HBASE-8877.patch, HBASE-8877-v2.patch, 
> HBASE-8877-v3.patch, hbase-8877-v4-microbenchmark.txt, HBASE-8877-v4.patch, 
> HBASE-8877-v5.patch
>
>
> HBASE-8806 revealed performance problems with batch mutations failing to 
> reacquire the same row locks.  It looks like HBASE-8806 will use a less 
> intrusive change for 0.94 to have batch mutations track their own row locks 
> and not attempt to reacquire them.  Another approach will be to support 
> reentrant row locks directly.  This allows simplifying a great deal of 
> calling code to no longer track and pass around lock ids.
> One affect this change will have is changing the RegionObserver coprocessor's 
> methods preBatchMutate and postBatchMutate from taking a 
> {{MiniBatchOperationInProgress> miniBatchOp}} to 
> taking a {{MiniBatchOperationInProgress miniBatchOp}}.  I don't 
> believe CPs should be relying on these lock ids, but that's a potential 
> incompatibility.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-8943) Split Thrift2's ThriftServer into separate classes for easier testing and modularization

[Lars George (JIRA)]

Lars George created HBASE-8943:
--

 Summary: Split Thrift2's ThriftServer into separate classes for 
easier testing and modularization
 Key: HBASE-8943
 URL: https://issues.apache.org/jira/browse/HBASE-8943
 Project: HBase
  Issue Type: Sub-task
  Components: Thrift
Reporter: Lars George
Assignee: Lars George


Currently the ThriftServer class in Thrift 2 sets up and starts the actual 
server. Better follow a similar pattern to Thrift 1 where there is some factory 
setting up the server, and a separate start section. That way it is easier to 
test if the setup of the server is picking up everything it needs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8938) Thrift2 does not close scanner instance

[Lars George (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars George updated HBASE-8938:
---

   Resolution: Fixed
Fix Version/s: 0.94.10
   0.95.2
   0.98.0
 Assignee: Hamed Madani
   Status: Resolved  (was: Patch Available)

Applied to all major branches, thanks Hamed!

> Thrift2 does not close scanner instance
> ---
>
> Key: HBASE-8938
> URL: https://issues.apache.org/jira/browse/HBASE-8938
> Project: HBase
>  Issue Type: Sub-task
>  Components: Thrift
>Affects Versions: 0.94.0
>Reporter: Hamed Madani
>Assignee: Hamed Madani
>  Labels: thrift2
> Fix For: 0.98.0, 0.95.2, 0.94.10
>
> Attachments: HBASE-8938-0.94.patch, HBASE-8938.patch
>
>
> closeScanner at *ThriftHBaseServiceHandler* simply remove scannerId from 
> internal map, but fails to close the scanner instance. 
> {code}
>   public void closeScanner(int scannerId) throws TIOError, TIllegalArgument, 
> TException {
> if (removeScanner(scannerId) == null) {
>   TIllegalArgument ex = new TIllegalArgument();
>   ex.setMessage("Invalid scanner Id");
>   throw ex;
> }
>   }
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8877) Reentrant row locks

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707750#comment-13707750
 ] 

Hadoop QA commented on HBASE-8877:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12592150/HBASE-8877-v4.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 9 new 
or modified tests.

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6324//console

This message is automatically generated.

> Reentrant row locks
> ---
>
> Key: HBASE-8877
> URL: https://issues.apache.org/jira/browse/HBASE-8877
> Project: HBase
>  Issue Type: Bug
>  Components: Coprocessors, regionserver
>Reporter: Dave Latham
>Assignee: Dave Latham
> Fix For: 0.95.2
>
> Attachments: hbase-8877-0.94-microbenchmark.txt, 
> HBASE-8877-0.94.patch, HBASE-8877.patch, HBASE-8877-v2.patch, 
> HBASE-8877-v3.patch, hbase-8877-v4-microbenchmark.txt, HBASE-8877-v4.patch
>
>
> HBASE-8806 revealed performance problems with batch mutations failing to 
> reacquire the same row locks.  It looks like HBASE-8806 will use a less 
> intrusive change for 0.94 to have batch mutations track their own row locks 
> and not attempt to reacquire them.  Another approach will be to support 
> reentrant row locks directly.  This allows simplifying a great deal of 
> calling code to no longer track and pass around lock ids.
> One affect this change will have is changing the RegionObserver coprocessor's 
> methods preBatchMutate and postBatchMutate from taking a 
> {{MiniBatchOperationInProgress> miniBatchOp}} to 
> taking a {{MiniBatchOperationInProgress miniBatchOp}}.  I don't 
> believe CPs should be relying on these lock ids, but that's a potential 
> incompatibility.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8877) Reentrant row locks

[Dave Latham (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707747#comment-13707747
 ] 

Dave Latham commented on HBASE-8877:


Reattached HBASE-8877-v4.patch for Hadoop QA to notice.

> Reentrant row locks
> ---
>
> Key: HBASE-8877
> URL: https://issues.apache.org/jira/browse/HBASE-8877
> Project: HBase
>  Issue Type: Bug
>  Components: Coprocessors, regionserver
>Reporter: Dave Latham
>Assignee: Dave Latham
> Fix For: 0.95.2
>
> Attachments: hbase-8877-0.94-microbenchmark.txt, 
> HBASE-8877-0.94.patch, HBASE-8877.patch, HBASE-8877-v2.patch, 
> HBASE-8877-v3.patch, hbase-8877-v4-microbenchmark.txt, HBASE-8877-v4.patch
>
>
> HBASE-8806 revealed performance problems with batch mutations failing to 
> reacquire the same row locks.  It looks like HBASE-8806 will use a less 
> intrusive change for 0.94 to have batch mutations track their own row locks 
> and not attempt to reacquire them.  Another approach will be to support 
> reentrant row locks directly.  This allows simplifying a great deal of 
> calling code to no longer track and pass around lock ids.
> One affect this change will have is changing the RegionObserver coprocessor's 
> methods preBatchMutate and postBatchMutate from taking a 
> {{MiniBatchOperationInProgress> miniBatchOp}} to 
> taking a {{MiniBatchOperationInProgress miniBatchOp}}.  I don't 
> believe CPs should be relying on these lock ids, but that's a potential 
> incompatibility.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8877) Reentrant row locks

[Dave Latham (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dave Latham updated HBASE-8877:
---

Attachment: HBASE-8877-v4.patch

> Reentrant row locks
> ---
>
> Key: HBASE-8877
> URL: https://issues.apache.org/jira/browse/HBASE-8877
> Project: HBase
>  Issue Type: Bug
>  Components: Coprocessors, regionserver
>Reporter: Dave Latham
>Assignee: Dave Latham
> Fix For: 0.95.2
>
> Attachments: hbase-8877-0.94-microbenchmark.txt, 
> HBASE-8877-0.94.patch, HBASE-8877.patch, HBASE-8877-v2.patch, 
> HBASE-8877-v3.patch, hbase-8877-v4-microbenchmark.txt, HBASE-8877-v4.patch
>
>
> HBASE-8806 revealed performance problems with batch mutations failing to 
> reacquire the same row locks.  It looks like HBASE-8806 will use a less 
> intrusive change for 0.94 to have batch mutations track their own row locks 
> and not attempt to reacquire them.  Another approach will be to support 
> reentrant row locks directly.  This allows simplifying a great deal of 
> calling code to no longer track and pass around lock ids.
> One affect this change will have is changing the RegionObserver coprocessor's 
> methods preBatchMutate and postBatchMutate from taking a 
> {{MiniBatchOperationInProgress> miniBatchOp}} to 
> taking a {{MiniBatchOperationInProgress miniBatchOp}}.  I don't 
> believe CPs should be relying on these lock ids, but that's a potential 
> incompatibility.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8877) Reentrant row locks

[Dave Latham (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dave Latham updated HBASE-8877:
---

Attachment: (was: HBASE-8877-v4.patch)

> Reentrant row locks
> ---
>
> Key: HBASE-8877
> URL: https://issues.apache.org/jira/browse/HBASE-8877
> Project: HBase
>  Issue Type: Bug
>  Components: Coprocessors, regionserver
>Reporter: Dave Latham
>Assignee: Dave Latham
> Fix For: 0.95.2
>
> Attachments: hbase-8877-0.94-microbenchmark.txt, 
> HBASE-8877-0.94.patch, HBASE-8877.patch, HBASE-8877-v2.patch, 
> HBASE-8877-v3.patch, hbase-8877-v4-microbenchmark.txt
>
>
> HBASE-8806 revealed performance problems with batch mutations failing to 
> reacquire the same row locks.  It looks like HBASE-8806 will use a less 
> intrusive change for 0.94 to have batch mutations track their own row locks 
> and not attempt to reacquire them.  Another approach will be to support 
> reentrant row locks directly.  This allows simplifying a great deal of 
> calling code to no longer track and pass around lock ids.
> One affect this change will have is changing the RegionObserver coprocessor's 
> methods preBatchMutate and postBatchMutate from taking a 
> {{MiniBatchOperationInProgress> miniBatchOp}} to 
> taking a {{MiniBatchOperationInProgress miniBatchOp}}.  I don't 
> believe CPs should be relying on these lock ids, but that's a potential 
> incompatibility.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8935) IntegrationTestBigLinkedList fails under load on 0.94 due to some scan issues

[Lars Hofhansl (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707742#comment-13707742
 ] 

Lars Hofhansl commented on HBASE-8935:
--

+1 (trunk and 0.94)

> IntegrationTestBigLinkedList fails under load on 0.94 due to some scan issues
> -
>
> Key: HBASE-8935
> URL: https://issues.apache.org/jira/browse/HBASE-8935
> Project: HBase
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HBASE-8935-v0-94-logging.patch, HBASE-8935-v0.patch
>
>
> We observe occasional failures (4-5k undefined and unreferenced nodes in the 
> list, running) when the cluster is stressed while running 
> IntegrationTestBigLinkedList on 0.94. Unfortunately debug logging is disabled.
> If Verify is rerun after the fact, the data is there, so it's not data loss. 
> All the missing keys come from very narrow range from the very beginning of  
> one region; most of this region is not affected.
> In the case I'm looking at, region range is
> {code}\xD0\x0C`=%\xA2\xD3\xA8\x0A\xF8\x917\x05\x94\x1D> .. 
> \xD4\x0Bx\xAF\x88\x0C\xF4"7\x9A\x9F{\xCE\x0E\x8A{code}
> and problematic renage
> {code}\xD0\x0C`QLD\xED2\xD5c\x8D\xDB5\x01\xD2H] ... 
> \xD0\x0D\x89)\x11\x0E8\xC5\x08o\xD7\xE3$\xB3\xAAu{code}
> There are no splits and compactions on the region during MR job; there are no 
> compactions after that could have affected the data, although there is one 
> flush that happened long after, and region was moved and reopened (before I 
> ran verify manually that showed that data is in HBase).
> One thing that happened was that the scanners used by all the map tasks had 
> lease expirations during the MR job that had missing data, some of them twice.
> First scanner expiration for each task I looked at was exactly 1 minute after 
> "Processing split" log message, when the scanner is opened.
> Only the tasks where the scanners have expired twice (2 of them) logged any 
> errors; presumably one expiration in each task happened after the reading was 
> finished, because everything was slow and scanner wasn't closed in time - no 
> errors or exceptions are logged in the tasks where scanner only expired once.
> The job I ran afterwards had no scanner expirations so it does close them 
> correctly in normal case...
> The task that was reading the problematic range had one scanner expiration 
> and didn't log any errors.
> It's a little bit special (or it may be a total red herring) in that in other 
> tasks, scanner expired while the task was splitting partial outputs (which 
> may mean end of input reading?), whereas in the task that lost data spilling 
> started long (~40s) after the scanner expired.
> However there's one another such task (spilling started 25s after scanner 
> expired) and it didn't log any errors and didn't lose data.
> At this point I am not sure about the root cause but I suspect it might be 
> related to scanner expiration handling, given the patterns.
> One thing for sure is that there isn't enough logging... so I will start with 
> adding that.
>  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8938) Thrift2 does not close scanner instance

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707697#comment-13707697
 ] 

Hadoop QA commented on HBASE-8938:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12592002/HBASE-8938-0.94.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/6323//console

This message is automatically generated.

> Thrift2 does not close scanner instance
> ---
>
> Key: HBASE-8938
> URL: https://issues.apache.org/jira/browse/HBASE-8938
> Project: HBase
>  Issue Type: Sub-task
>  Components: Thrift
>Affects Versions: 0.94.0
>Reporter: Hamed Madani
>  Labels: thrift2
> Attachments: HBASE-8938-0.94.patch, HBASE-8938.patch
>
>
> closeScanner at *ThriftHBaseServiceHandler* simply remove scannerId from 
> internal map, but fails to close the scanner instance. 
> {code}
>   public void closeScanner(int scannerId) throws TIOError, TIllegalArgument, 
> TException {
> if (removeScanner(scannerId) == null) {
>   TIllegalArgument ex = new TIllegalArgument();
>   ex.setMessage("Invalid scanner Id");
>   throw ex;
> }
>   }
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-8938) Thrift2 does not close scanner instance

[Hamed Madani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-8938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hamed Madani updated HBASE-8938:


Status: Patch Available  (was: Open)

> Thrift2 does not close scanner instance
> ---
>
> Key: HBASE-8938
> URL: https://issues.apache.org/jira/browse/HBASE-8938
> Project: HBase
>  Issue Type: Sub-task
>  Components: Thrift
>Affects Versions: 0.94.0
>Reporter: Hamed Madani
>  Labels: thrift2
> Attachments: HBASE-8938-0.94.patch, HBASE-8938.patch
>
>
> closeScanner at *ThriftHBaseServiceHandler* simply remove scannerId from 
> internal map, but fails to close the scanner instance. 
> {code}
>   public void closeScanner(int scannerId) throws TIOError, TIllegalArgument, 
> TException {
> if (removeScanner(scannerId) == null) {
>   TIllegalArgument ex = new TIllegalArgument();
>   ex.setMessage("Invalid scanner Id");
>   throw ex;
> }
>   }
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7035) thrift server closes HTable of open Scanners

[Hamed Madani (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707696#comment-13707696
 ] 

Hamed Madani commented on HBASE-7035:
-

looking at *htablePool.java*, I don't think it's necessary to change 
closeTable() function. Both table.close() and htablePool.putTable(table) are 
calling returnTable(). from what I see putTable() is deprecated. 

> thrift server closes HTable of open Scanners
> 
>
> Key: HBASE-7035
> URL: https://issues.apache.org/jira/browse/HBASE-7035
> Project: HBase
>  Issue Type: Sub-task
>  Components: Thrift
>Affects Versions: 0.94.4
>Reporter: Devin Bayer
>  Labels: thrift2
> Attachments: old-hbase-thrift-v1.patch
>
>   Original Estimate: 4h
>  Remaining Estimate: 4h
>
> ThriftHBaseServiceHandler.openScanner() does this:
> 1. table = pool.getTable()
> 2. scanner = table.getScanner()
> 3. table.close()
> 4. return scanner
> While back porting the thrift server to 0.92.6, I found that table.close() 
> calls connection.close(). Further calls to scanner.next() raise a 
> ConnectionClosed exception. The unit tests do not catch this since they reuse 
> an open HConnection instance.
> This might work on trunk, but depends on the implementations of HTablePool, 
> HTable and HConnectionManager. Even with the pool wrapper, if the pool is 
> full, table.close() may be called, which may invalidate the table. Also,  
> HTable is not thread-safe, but they are being reused since they go back in 
> the pool.
> I suggest storing the table handle along with the scanner.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7035) thrift server closes HTable of open Scanners

[Lars George (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707687#comment-13707687
 ] 

Lars George commented on HBASE-7035:


Hey [~akvadrako], so this is more like Thrift 1, which stores the tables in a 
ThreadLocal instance to keep it open, not using the HTablePool at all? In other 
words, you remove the pool here as well, keeping the tables for each scanner 
separately? Sounds reasonable to me.

> thrift server closes HTable of open Scanners
> 
>
> Key: HBASE-7035
> URL: https://issues.apache.org/jira/browse/HBASE-7035
> Project: HBase
>  Issue Type: Sub-task
>  Components: Thrift
>Affects Versions: 0.94.4
>Reporter: Devin Bayer
>  Labels: thrift2
> Attachments: old-hbase-thrift-v1.patch
>
>   Original Estimate: 4h
>  Remaining Estimate: 4h
>
> ThriftHBaseServiceHandler.openScanner() does this:
> 1. table = pool.getTable()
> 2. scanner = table.getScanner()
> 3. table.close()
> 4. return scanner
> While back porting the thrift server to 0.92.6, I found that table.close() 
> calls connection.close(). Further calls to scanner.next() raise a 
> ConnectionClosed exception. The unit tests do not catch this since they reuse 
> an open HConnection instance.
> This might work on trunk, but depends on the implementations of HTablePool, 
> HTable and HConnectionManager. Even with the pool wrapper, if the pool is 
> full, table.close() may be called, which may invalidate the table. Also,  
> HTable is not thread-safe, but they are being reused since they go back in 
> the pool.
> I suggest storing the table handle along with the scanner.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-8938) Thrift2 does not close scanner instance

[Lars George (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707681#comment-13707681
 ] 

Lars George commented on HBASE-8938:


Ah, you are right, Thrift 1 closes the scanner. I was mistaken there. Trying 
the patches now and will commit.

> Thrift2 does not close scanner instance
> ---
>
> Key: HBASE-8938
> URL: https://issues.apache.org/jira/browse/HBASE-8938
> Project: HBase
>  Issue Type: Sub-task
>  Components: Thrift
>Affects Versions: 0.94.0
>Reporter: Hamed Madani
>  Labels: thrift2
> Attachments: HBASE-8938-0.94.patch, HBASE-8938.patch
>
>
> closeScanner at *ThriftHBaseServiceHandler* simply remove scannerId from 
> internal map, but fails to close the scanner instance. 
> {code}
>   public void closeScanner(int scannerId) throws TIOError, TIllegalArgument, 
> TException {
> if (removeScanner(scannerId) == null) {
>   TIllegalArgument ex = new TIllegalArgument();
>   ex.setMessage("Invalid scanner Id");
>   throw ex;
> }
>   }
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira