date:20120323


[ 
https://issues.apache.org/jira/browse/HBASE-5573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236375#comment-13236375
 ] 

stack commented on HBASE-5573:
--

This is radical in the ReplicationAdmin:

{code}
+System.exit(1);
{code}

This is a client only?  Maybe get the Abortable the this.connection is using?  
Would that make sense?

Hmm... you do it in hbasefsck too.

Why not add a create method to ZooKeeperWatcher that takes a name, conf, and 
Abortable?  Or is that a ZKW Constructor altogether?

Creating the ZKW each time is probably expensive, takes time?  But its ok in 
ReplicationAdmin and in HBaseFSCK I would say?

In testing, do we want to rethrow what caused an abort?  Perhaps rethrow as a 
RuntimeException?

{code}
+@Override public void abort(String why, Throwable e) {}
{code}

N, can you explain more about what is going on here.  How is it that we are not 
taking a Watcher when we are creating a ZKW?   Because we don't call start?  
(If so, that'd be 'elegant' solution)




 Replace client ZooKeeper watchers by simple ZooKeeper reads
 ---

 Key: HBASE-5573
 URL: https://issues.apache.org/jira/browse/HBASE-5573
 Project: HBase
  Issue Type: Improvement
  Components: client, zookeeper
Affects Versions: 0.96.0
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
 Attachments: 5573.v1.patch, 5573.v2.patch, 5573.v4.patch, 
 5573.v6.patch


 Some code in the package needs to read data in ZK. This could be done by a 
 simple read, but is actually implemented with a watcher. This holds ZK 
 resources.
 Fixing this could also be an opportunity to remove the need for the client to 
 provide the master address and port.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5591) ThiftServerRunner.HBaseHandler.toBytes() is identical to Bytes.getBytes()

2012-03-23 Thread Phabricator (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236380#comment-13236380
 ] 

Phabricator commented on HBASE-5591:


stack has commented on the revision HBASE-5591 [jira] 
ThiftServerRunner.HBaseHandler.toBytes() is identical to Bytes.getBytes().

INLINE COMMENTS
  src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java:611 
Funny.  If it was added by you, then I'd say its your prerogative to remove it! 
 Good stuff.

REVISION DETAIL
  https://reviews.facebook.net/D2355

BRANCH
  getbytes


 ThiftServerRunner.HBaseHandler.toBytes() is identical to Bytes.getBytes()
 -

 Key: HBASE-5591
 URL: https://issues.apache.org/jira/browse/HBASE-5591
 Project: HBase
  Issue Type: Improvement
Reporter: Scott Chen
Assignee: Scott Chen
Priority: Trivial
 Fix For: 0.96.0

 Attachments: HBASE-5591.D2355.1.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5623) Race condition when rolling the HLog and hlogFlush


[ 
https://issues.apache.org/jira/browse/HBASE-5623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236379#comment-13236379
 ] 

Hadoop QA commented on HBASE-5623:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12519585/5623v2.txt
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 2 new or modified tests.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 1 new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.mapreduce.TestImportTsv
  org.apache.hadoop.hbase.mapred.TestTableMapReduce
  org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1272//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1272//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1272//console

This message is automatically generated.

 Race condition when rolling the HLog and hlogFlush
 --

 Key: HBASE-5623
 URL: https://issues.apache.org/jira/browse/HBASE-5623
 Project: HBase
  Issue Type: Bug
  Components: wal
Affects Versions: 0.94.0
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Priority: Critical
 Attachments: 5623v2.txt, HBASE-5623_v0.patch


 When doing a ycsb test with a large number of handlers 
 (regionserver.handler.count=60), I get the following exceptions:
 {code}
 Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: 
 java.lang.NullPointerException
   at 
 org.apache.hadoop.io.SequenceFile$Writer.getLength(SequenceFile.java:1099)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.getLength(SequenceFileLogWriter.java:314)
   at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1291)
   at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1388)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:2192)
   at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1985)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3400)
   at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:366)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1351)
   at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:920)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:152)
   at $Proxy1.multi(Unknown Source)
   at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1691)
   at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1689)
   at 
 org.apache.hadoop.hbase.client.ServerCallable.withoutRetries(ServerCallable.java:214)
 {code}
 and 
 {code}
   java.lang.NullPointerException
   at 
 org.apache.hadoop.io.SequenceFile$Writer.checkAndWriteSync(SequenceFile.java:1026)
   at 
 org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1068)
   at 
 org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1035)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.append(SequenceFileLogWriter.java:279)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.hlogFlush(HLog.java:1237)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1271)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1391)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:2192)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1985)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3400)
   at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source)

[jira] [Commented] (HBASE-5624) Aborting regionserver when splitting region, may cause daughter region not assigned by ServerShutdownHandler.

2012-03-23 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236384#comment-13236384
 ] 

Hadoop QA commented on HBASE-5624:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12519584/HBASE-5624.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 1 new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.util.TestFSUtils

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1273//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1273//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1273//console

This message is automatically generated.

 Aborting regionserver when splitting region, may cause daughter region not 
 assigned by ServerShutdownHandler.
 -

 Key: HBASE-5624
 URL: https://issues.apache.org/jira/browse/HBASE-5624
 Project: HBase
  Issue Type: Bug
Reporter: chunhui shen
Assignee: chunhui shen
 Attachments: HBASE-5624.patch


 If one region is splitting when regionserver is stoping.
 The following code may executed in SplitTransaction#openDaughters.
 {code}
 // TODO: Is this check needed here?
 if (stopped || stopping) {
   // add 2nd daughter first (see HBASE-4335)
   MetaEditor.addDaughter(server.getCatalogTracker(),
   b.getRegionInfo(), null);
   MetaEditor.addDaughter(server.getCatalogTracker(),
   a.getRegionInfo(), null);
   LOG.info(Not opening daughters  +
   b.getRegionInfo().getRegionNameAsString() +
and  +
   a.getRegionInfo().getRegionNameAsString() +
because stopping= + stopping + , stopped= + stopped);
 } 
 {code}
 So, for the two daughter regions, their location are both null in .META.
 When ServerShutdownHandler process the dead server, it will not assign these 
 two daughter regions since their location(info:server) are null in .META. by 
 MetaReader.getServerUserRegions().

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5624) Aborting regionserver when splitting region, may cause daughter region not assigned by ServerShutdownHandler.

2012-03-23 Thread chunhui shen (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236393#comment-13236393
 ] 

chunhui shen commented on HBASE-5624:
-

I run org.apache.hadoop.hbase.util.TestFSUtils on local PC with the patch and 
returns OK.

 Aborting regionserver when splitting region, may cause daughter region not 
 assigned by ServerShutdownHandler.
 -

 Key: HBASE-5624
 URL: https://issues.apache.org/jira/browse/HBASE-5624
 Project: HBase
  Issue Type: Bug
Reporter: chunhui shen
Assignee: chunhui shen
 Attachments: HBASE-5624.patch


 If one region is splitting when regionserver is stoping.
 The following code may executed in SplitTransaction#openDaughters.
 {code}
 // TODO: Is this check needed here?
 if (stopped || stopping) {
   // add 2nd daughter first (see HBASE-4335)
   MetaEditor.addDaughter(server.getCatalogTracker(),
   b.getRegionInfo(), null);
   MetaEditor.addDaughter(server.getCatalogTracker(),
   a.getRegionInfo(), null);
   LOG.info(Not opening daughters  +
   b.getRegionInfo().getRegionNameAsString() +
and  +
   a.getRegionInfo().getRegionNameAsString() +
because stopping= + stopping + , stopped= + stopped);
 } 
 {code}
 So, for the two daughter regions, their location are both null in .META.
 When ServerShutdownHandler process the dead server, it will not assign these 
 two daughter regions since their location(info:server) are null in .META. by 
 MetaReader.getServerUserRegions().

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5564) Bulkload is discarding duplicate records

2012-03-23 Thread Anoop Sam John (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236441#comment-13236441
 ] 

Anoop Sam John commented on HBASE-5564:
---

{quote}
In bulkload, if multiple records are having same timestamp, then the last KV 
entry processed by reducer only will be persisted (TreeSet in Reducer)
{quote}

The 1st KV processed by the Reducer right...

Yes agree with you which one is the latest might not be possible to be 
predicted in the reducer side...

 Bulkload is discarding duplicate records
 

 Key: HBASE-5564
 URL: https://issues.apache.org/jira/browse/HBASE-5564
 Project: HBase
  Issue Type: Bug
  Components: mapreduce
Affects Versions: 0.90.7, 0.92.2, 0.94.0, 0.96.0
 Environment: HBase 0.92
Reporter: Laxman
Assignee: Laxman
  Labels: bulkloader
 Fix For: 0.96.0

 Attachments: 5564.lint, HBASE-5564_trunk.1.patch, 
 HBASE-5564_trunk.1.patch, HBASE-5564_trunk.patch


 Duplicate records are getting discarded when duplicate records exists in same 
 input file and more specifically if they exists in same split.
 Duplicate records are considered if the records are from diffrent different 
 splits.
 Version under test: HBase 0.92

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5591) ThiftServerRunner.HBaseHandler.toBytes() is identical to Bytes.getBytes()

2012-03-23 Thread Phabricator (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236444#comment-13236444
 ] 

Phabricator commented on HBASE-5591:


sc has commented on the revision HBASE-5591 [jira] 
ThiftServerRunner.HBaseHandler.toBytes() is identical to Bytes.getBytes().

INLINE COMMENTS
  src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java:611 
Sorry for the trouble. Thanks for bearing with me :)

REVISION DETAIL
  https://reviews.facebook.net/D2355

BRANCH
  getbytes


 ThiftServerRunner.HBaseHandler.toBytes() is identical to Bytes.getBytes()
 -

 Key: HBASE-5591
 URL: https://issues.apache.org/jira/browse/HBASE-5591
 Project: HBase
  Issue Type: Improvement
Reporter: Scott Chen
Assignee: Scott Chen
Priority: Trivial
 Fix For: 0.96.0

 Attachments: HBASE-5591.D2355.1.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3134) [replication] Add the ability to enable/disable streams


[ 
https://issues.apache.org/jira/browse/HBASE-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236450#comment-13236450
 ] 

jirapos...@reviews.apache.org commented on HBASE-3134:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3686/
---

(Updated 2012-03-23 08:16:47.333041)


Review request for hbase.


Changes
---

This update addresses the issue found by J-D.
A new test which fails without the modification to the shipEdits() is added to 
the TestReplication.


Summary
---

1. The patch introduces a znode which represents whether a peer is enabled or 
not. ReplicationSource skips sending entries if the replication is disabled.
2. The list_peers shows state of each peer and support for 
{enable|disable}Peer is added.


This addresses bug HBASE-3134.
https://issues.apache.org/jira/browse/HBASE-3134


Diffs (updated)
-

  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java
 1304172 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeer.java
 1304190 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
 1304190 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
 1304190 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java
 1304190 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
 1304190 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/ruby/hbase/replication_admin.rb
 1304172 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/ruby/shell/commands/disable_peer.rb
 1304172 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/ruby/shell/commands/enable_peer.rb
 1304172 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/ruby/shell/commands/list_peers.rb
 1304172 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/ReplicationSourceDummy.java
 1304172 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestReplication.java
 1304172 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java
 1304172 

Diff: https://reviews.apache.org/r/3686/diff


Testing
---

1. TestReplication passed.
2. Checked on a cluster with some updates (puts) from shell.


Thanks,

Teruyoshi



 [replication] Add the ability to enable/disable streams
 ---

 Key: HBASE-3134
 URL: https://issues.apache.org/jira/browse/HBASE-3134
 Project: HBase
  Issue Type: New Feature
  Components: replication
Reporter: Jean-Daniel Cryans
Assignee: Teruyoshi Zenmyo
Priority: Minor
  Labels: replication
 Fix For: 0.94.1

 Attachments: 3134-v2.txt, 3134-v3.txt, 3134.txt, HBASE-3134.patch, 
 HBASE-3134.patch, HBASE-3134.patch, HBASE-3134.patch


 This jira was initially in the scope of HBASE-2201, but was pushed out since 
 it has low value compared to the required effort (and when want to ship 
 0.90.0 rather soonish).
 We need to design a way to enable/disable replication streams in a 
 determinate fashion.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-5625) Avoid byte buffer allocations when reading a value from a Result object

2012-03-23 Thread Tudor Scurtu (Created) (JIRA)

Avoid byte buffer allocations when reading a value from a Result object
---

 Key: HBASE-5625
 URL: https://issues.apache.org/jira/browse/HBASE-5625
 Project: HBase
  Issue Type: Improvement
  Components: client
Affects Versions: 0.92.1
Reporter: Tudor Scurtu


When calling Result.getValue(), an extra dummy KeyValue and its associated 
underlying byte array are allocated, as well as a persistent buffer that will 
contain the returned value.

These can be avoided by reusing a static array for the dummy object and by 
passing a ByteBuffer object as a value destination buffer to the read method.

The current functionality is maintained, and we have added a separate method 
call stack that employs the described changes.

On our use case - with large map-reduce jobs - we have noticed a reduction of 
read time of up to 40%.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5625) Avoid byte buffer allocations when reading a value from a Result object

2012-03-23 Thread Tudor Scurtu (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tudor Scurtu updated HBASE-5625:


Description: 
When calling Result.getValue(), an extra dummy KeyValue and its associated 
underlying byte array are allocated, as well as a persistent buffer that will 
contain the returned value.

These can be avoided by reusing a static array for the dummy object and by 
passing a ByteBuffer object as a value destination buffer to the read method.

The current functionality is maintained, and we have added a separate method 
call stack that employs the described changes. I will provide more details with 
the patch.

Running tests with a profiler, the reduction of read time seems to be of up to 
40%.

  was:
When calling Result.getValue(), an extra dummy KeyValue and its associated 
underlying byte array are allocated, as well as a persistent buffer that will 
contain the returned value.

These can be avoided by reusing a static array for the dummy object and by 
passing a ByteBuffer object as a value destination buffer to the read method.

The current functionality is maintained, and we have added a separate method 
call stack that employs the described changes.

On our use case - with large map-reduce jobs - we have noticed a reduction of 
read time of up to 40%.


 Avoid byte buffer allocations when reading a value from a Result object
 ---

 Key: HBASE-5625
 URL: https://issues.apache.org/jira/browse/HBASE-5625
 Project: HBase
  Issue Type: Improvement
  Components: client
Affects Versions: 0.92.1
Reporter: Tudor Scurtu
  Labels: patch

 When calling Result.getValue(), an extra dummy KeyValue and its associated 
 underlying byte array are allocated, as well as a persistent buffer that will 
 contain the returned value.
 These can be avoided by reusing a static array for the dummy object and by 
 passing a ByteBuffer object as a value destination buffer to the read method.
 The current functionality is maintained, and we have added a separate method 
 call stack that employs the described changes. I will provide more details 
 with the patch.
 Running tests with a profiler, the reduction of read time seems to be of up 
 to 40%.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5625) Avoid byte buffer allocations when reading a value from a Result object

2012-03-23 Thread Tudor Scurtu (Updated) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tudor Scurtu updated HBASE-5625:

Attachment: 5625.txt

The current patch introduces a method (Result.loadValue()) that uses
pre-allocated buffers. Momentarily the new methods duplicate code from the
original ones so as not to impact the current architecture.

Avoid byte buffer allocations when reading a value from a Result object
---

Key: HBASE-5625
URL: https://issues.apache.org/jira/browse/HBASE-5625
Project: HBase
Issue Type: Improvement
Components: client
Affects Versions: 0.92.1
Reporter: Tudor Scurtu
Labels: patch
Attachments: 5625.txt

When calling Result.getValue(), an extra dummy KeyValue and its associated
underlying byte array are allocated, as well as a persistent buffer that will
contain the returned value.
These can be avoided by reusing a static array for the dummy object and by
passing a ByteBuffer object as a value destination buffer to the read method.
The current functionality is maintained, and we have added a separate method
call stack that employs the described changes. I will provide more details
with the patch.
Running tests with a profiler, the reduction of read time seems to be of up
to 40%.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5615) the master never do balance becauseof balance the parent region

2012-03-23 Thread xufeng (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236464#comment-13236464
 ] 

xufeng commented on HBASE-5615:
---

reproduce this issue by 0.90
In this issue,META should hold parent region info for long time.So before 
test,I delete those code in regionserver class:
{noformat}
  public void postOpenDeployTasks(final HRegion r, final CatalogTracker ct,
  final boolean daughter)
  throws KeeperException, IOException {
// Do checks to see if we need to compact (references or too many files)
/*if (r.hasReferences() || r.hasTooManyStoreFiles()) {
  getCompactionRequester().requestCompaction(r,
r.hasReferences()? Region has references on open :
  Region has too many store files);
}*/
{noformat}

step1:start cluster that has two master and one regionerver process.
step2:create a table and input some data in it.
step3:split the table by shell.
step4:kill the active master.
step5:after backup master become active one,start another regionserver process.
result:the issue happen


I also test my patch many times and it can work.

 the master never do balance becauseof  balance the parent region
 

 Key: HBASE-5615
 URL: https://issues.apache.org/jira/browse/HBASE-5615
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.7
Reporter: xufeng
Assignee: xufeng
Priority: Critical
 Attachments: HBASE-5615.patch


 the master never do balance becauseof when master do rebuildUserRegions()，it 
 will add the parent region into  AssignmentManager#servers,
 if balancer let the parent region to move,the parent will in RIT forever.thus 
 balance will never be executed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5615) the master never do balance becauseof balance the parent region

2012-03-23 Thread xufeng (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

xufeng updated HBASE-5615:
--

Attachment: HBASE-5615-90.patch

@Zhihong Yu
updated the patch.
this patch for 0.90 version

@Rama
I will check the TRUNK and 0.92 version.

 the master never do balance becauseof  balance the parent region
 

 Key: HBASE-5615
 URL: https://issues.apache.org/jira/browse/HBASE-5615
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.7
Reporter: xufeng
Assignee: xufeng
Priority: Critical
 Attachments: HBASE-5615-90.patch, HBASE-5615.patch


 the master never do balance becauseof when master do rebuildUserRegions()，it 
 will add the parent region into  AssignmentManager#servers,
 if balancer let the parent region to move,the parent will in RIT forever.thus 
 balance will never be executed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5128) [uber hbck] Enable hbck to automatically repair table integrity problems as well as region consistency problems while online.

2012-03-23 Thread fulin wang (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236488#comment-13236488
]

fulin wang commented on HBASE-5128:
---

1.NOT_IN_META_OR_DEPLOYED
handler.handleHoleInRegionChain(key, holeStopKey);

NOT_IN_META
HBaseFsckRepair.fixMetaHoleOnline(conf, hbi.getHdfsHRI());

I think that you should check the region file of table the hole and the
region in the hole, you can create region for this hole.
otherwise you should not create region.
There is scenarios you shou think: the region of table is good or this region
is junk file.

2.FIRST_REGION_STARTKEY_NOT_EMPTY and HOLE_IN_REGION_CHAIN
I think when there is a type of error you can create empty region for this
hole.
if there is another error, another error you handle the first.

[uber hbck] Enable hbck to automatically repair table integrity problems as
well as region consistency problems while online.
-

Key: HBASE-5128
URL: https://issues.apache.org/jira/browse/HBASE-5128
Project: HBase
Issue Type: New Feature
Components: hbck
Affects Versions: 0.90.5, 0.92.0, 0.94.0, 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0

Attachments: hbase-5128-0.90-v2.patch, hbase-5128-0.90-v2b.patch,
hbase-5128-0.92-v2.patch, hbase-5128-0.94-v2.patch,
hbase-5128-trunk-v2.patch, hbase-5128-trunk.patch, hbase-5128-v3.patch

The current (0.90.5, 0.92.0rc2) versions of hbck detects most of region
consistency and table integrity invariant violations. However with '-fix' it
can only automatically repair region consistency cases having to do with
deployment problems. This updated version should be able to handle all cases
(including a new orphan regiondir case). When complete will likely deprecate
the OfflineMetaRepair tool and subsume several open META-hole related issue.
Here's the approach (from the comment of at the top of the new version of the
file).
{code}
/**
* HBaseFsck (hbck) is a tool for checking and repairing region consistency
and
* table integrity.
*
* Region consistency checks verify that META, region deployment on
* region servers and the state of data in HDFS (.regioninfo files) all are in
* accordance.
*
* Table integrity checks verify that that all possible row keys can resolve
to
* exactly one region of a table. This means there are no individual
degenerate
* or backwards regions; no holes between regions; and that there no
overlapping
* regions.
*
* The general repair strategy works in these steps.
* 1) Repair Table Integrity on HDFS. (merge or fabricate regions)
* 2) Repair Region Consistency with META and assignments
*
* For table integrity repairs, the tables their region directories are
scanned
* for .regioninfo files. Each table's integrity is then verified. If there
* are any orphan regions (regions with no .regioninfo files), or holes, new
* regions are fabricated. Backwards regions are sidelined as well as empty
* degenerate (endkey==startkey) regions. If there are any overlapping
regions,
* a new region is created and all data is merged into the new region.
*
* Table integrity repairs deal solely with HDFS and can be done offline --
the
* hbase region servers or master do not need to be running. These phase can
be
* use to completely reconstruct the META table in an offline fashion.
*
* Region consistency requires three conditions -- 1) valid .regioninfo file
* present in an hdfs region dir, 2) valid row with .regioninfo data in META,
* and 3) a region is deployed only at the regionserver that is was assigned
to.
*
* Region consistency requires hbck to contact the HBase master and region
* servers, so the connect() must first be called successfully. Much of the
* region consistency information is transient and less risky to repair.
*/
{code}

[jira] [Commented] (HBASE-5573) Replace client ZooKeeper watchers by simple ZooKeeper reads

2012-03-23 Thread nkeywal (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236499#comment-13236499
]

nkeywal commented on HBASE-5573:

bq. System.exit(1);
Actually is was already like that in hbasefsck, I replaced a tracker by a
watcher that does not watch to read the data, that's all.

bq. Why not add a create method to ZooKeeperWatcher that takes a name, conf,
and Abortable? Or is that a ZKW Constructor altogether?
Yes, the question is what to do when you're asked to abort. Here I reused the
approach in hbasefsck, just exit.

bq. N, can you explain more about what is going on here. How is it that we are
not taking a Watcher when we are creating a ZKW? Because we don't call start?
(If so, that'd be 'elegant' solution)

A ZKW is a watcher. When you create a ZKW, you create a RecoverableZooKeeper
with yourself as a parameter. Pseudo code is:
{noformat}
class RecoverableZooKeeper {
ZooKeeper zk;
RecoverableZooKeeper (Watcher w){ zk=new ZooKeeper(w) }
}

class ZooKeeperWatcher implements Watcher {
RecoverableZooKeeper rz;
ZooKeeperWatcher (){ rz = new RecoverableZooKeeper(this); }
}
{noformat}

Using 'this' in a constructor is looking for problems but it works in this case
(remember, that's the existing code, not mine :-) ). Basically all these
classes are very strongly coupled. When I tried to partially decouple them it
exploded in my hands because you anyway need a watcher to manage the session
expiry stuff. I don't have a middle solution here: it's either a full rewriting
with a lot of fun to keep the existing interfaces for backward compatibility or
nothing.

So in the final patch I've just done some cleanup (removed the last usage of
getZooKeeperWatcher) and the usage of any watcher. So there's no proof in the
code, just that actually all the functions we use on the client don't use a
watcher. Anyway, they have a session in the ZK servers so they are expensive.
But thanks to #5399 the session on ZK will be closed after 5 minutes. So if you
have an architecture with clients coming up and down, you will be able to
increase the number of clients.

Three last comments:
- one of the design issue is that there ate two API: you can use directly any
of the ZKW, RZK, RK object or you can go through the static ZKUtils. May be the
intermediate solutions lie around this area.
- even if the existing design should not be shown to innocent scholars it's not
that terrible, because it's small. I didn't really like my first patches
because I was adding more classes and complexity without fixing the design.
- On the long term, I think that it actually make sense to have a watcher in
the client. It's not about the previous code: The previous code was not really
using watchers. The previous code was setting watchers without using them. The
new code (after #5399 and #5573) does not use or set watchers. But when you
have a fat client architecture like we have, it makes sense to share some
global state information, and it scales better when the info is pushed vs.
pulled. Having said that, there are many questions left: possible issues in how
expensive it is with ZooKeeper today, may be ZooKeeper is not really designed
for this (it's not really a global coordination work, as the client would be
readers only) and so on. FWIW, it seems that the current limit is around 10K
sessions in ZK:

{panel}
Patrick Hunt / Nov 18, 2010; 8:57pm
Re: number of clients/watchers

fyi: I haven't heard of anyone running over 10k sessions. I've tried 20k before
and had issues.
[...]
A session is represented by a ZooKeeper object. One session per object. So if
you have 10 client hosts each creating it's own ZooKeeper instance you'll have
10 sessions. This is regardless of the number of znodes, watches, etc...
Watches were designed to be lightweight and you can maintain a large number of
them. (25million spread across 500 sessions in my example)
{panel}

There were also a discussion on ZK mailing list about lightweith sessions.
http://markmail.org/message/cyow2xkneh2t3juc

Replace client ZooKeeper watchers by simple ZooKeeper reads
---

Key: HBASE-5573
URL: https://issues.apache.org/jira/browse/HBASE-5573
Project: HBase
Issue Type: Improvement
Components: client, zookeeper
Affects Versions: 0.96.0
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
Attachments: 5573.v1.patch, 5573.v2.patch, 5573.v4.patch,
5573.v6.patch

Some code in the package needs to read data in ZK. This could be done by a
simple read, but is actually implemented with a watcher. This holds ZK
resources.
Fixing this could also be an opportunity to remove the need for the client to
provide the master address and port.

--
This message

[jira] [Updated] (HBASE-5625) Avoid byte buffer allocations when reading a value from a Result object

2012-03-23 Thread Zhihong Yu (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Yu updated HBASE-5625:
--

Status: Patch Available  (was: Open)

 Avoid byte buffer allocations when reading a value from a Result object
 ---

 Key: HBASE-5625
 URL: https://issues.apache.org/jira/browse/HBASE-5625
 Project: HBase
  Issue Type: Improvement
  Components: client
Affects Versions: 0.92.1
Reporter: Tudor Scurtu
  Labels: patch
 Attachments: 5625.txt


 When calling Result.getValue(), an extra dummy KeyValue and its associated 
 underlying byte array are allocated, as well as a persistent buffer that will 
 contain the returned value.
 These can be avoided by reusing a static array for the dummy object and by 
 passing a ByteBuffer object as a value destination buffer to the read method.
 The current functionality is maintained, and we have added a separate method 
 call stack that employs the described changes. I will provide more details 
 with the patch.
 Running tests with a profiler, the reduction of read time seems to be of up 
 to 40%.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5625) Avoid byte buffer allocations when reading a value from a Result object

[
https://issues.apache.org/jira/browse/HBASE-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236520#comment-13236520
]

Hadoop QA commented on HBASE-5625:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12519598/5625.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.

-1 patch. The patch command could not apply the patch.

Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1274//console

This message is automatically generated.

Avoid byte buffer allocations when reading a value from a Result object
---

[jira] [Commented] (HBASE-5625) Avoid byte buffer allocations when reading a value from a Result object

2012-03-23 Thread Himanshu Vashishtha (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236529#comment-13236529
 ] 

Zhihong Yu commented on HBASE-5625:
---

@Tudor:
Can you submit a patch for trunk?
Thanks

 Avoid byte buffer allocations when reading a value from a Result object
 ---

 Key: HBASE-5625
 URL: https://issues.apache.org/jira/browse/HBASE-5625
 Project: HBase
  Issue Type: Improvement
  Components: client
Affects Versions: 0.92.1
Reporter: Tudor Scurtu
  Labels: patch
 Attachments: 5625.txt


 When calling Result.getValue(), an extra dummy KeyValue and its associated 
 underlying byte array are allocated, as well as a persistent buffer that will 
 contain the returned value.
 These can be avoided by reusing a static array for the dummy object and by 
 passing a ByteBuffer object as a value destination buffer to the read method.
 The current functionality is maintained, and we have added a separate method 
 call stack that employs the described changes. I will provide more details 
 with the patch.
 Running tests with a profiler, the reduction of read time seems to be of up 
 to 40%.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5222) Stopping replication via the stop_replication command in hbase shell on a slave cluster isn't acknowledged in the replication sink


[ 
https://issues.apache.org/jira/browse/HBASE-5222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236639#comment-13236639
 ] 

Himanshu Vashishtha commented on HBASE-5222:


@JD: When you want to use replication, you ought to run these commands (plus 
setting replication hbase.replication to true in the hase-config.xml) on the 
master cluster. The slave cluster configs is not changed (in case of simple 
Master-slave replication). 
So, in case when hbase.replication is false (or default) on the slave, its 
replication specific code will be null, and make these commands non-effective; 
no?

 Stopping replication via the stop_replication command in hbase shell on a 
 slave cluster isn't acknowledged in the replication sink
 

 Key: HBASE-5222
 URL: https://issues.apache.org/jira/browse/HBASE-5222
 Project: HBase
  Issue Type: Bug
  Components: replication, shell
Affects Versions: 0.90.4
Reporter: Josh Wymer

 After running stop_replication in the hbase shell on our slave cluster we 
 saw replication continue for weeks. Turns out that the replication sink is 
 missing a check to get the replication state and therefore continued to write.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5625) Avoid byte buffer allocations when reading a value from a Result object

2012-03-23 Thread Tudor Scurtu (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tudor Scurtu updated HBASE-5625:


Attachment: 5625v2.txt

Correct trunk diff.

 Avoid byte buffer allocations when reading a value from a Result object
 ---

 Key: HBASE-5625
 URL: https://issues.apache.org/jira/browse/HBASE-5625
 Project: HBase
  Issue Type: Improvement
  Components: client
Affects Versions: 0.92.1
Reporter: Tudor Scurtu
  Labels: patch
 Attachments: 5625.txt, 5625v2.txt


 When calling Result.getValue(), an extra dummy KeyValue and its associated 
 underlying byte array are allocated, as well as a persistent buffer that will 
 contain the returned value.
 These can be avoided by reusing a static array for the dummy object and by 
 passing a ByteBuffer object as a value destination buffer to the read method.
 The current functionality is maintained, and we have added a separate method 
 call stack that employs the described changes. I will provide more details 
 with the patch.
 Running tests with a profiler, the reduction of read time seems to be of up 
 to 40%.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5596) Few minor bugs from HBASE-5209

2012-03-23 Thread David S. Wang (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236676#comment-13236676
 ] 

David S. Wang commented on HBASE-5596:
--

Stack, this patch is to address Jon's comments in the reviewboard.

I'll move UNKNOWN_SERVERNAME up into ServerName and resubmit shortly.

 Few minor bugs from HBASE-5209
 --

 Key: HBASE-5596
 URL: https://issues.apache.org/jira/browse/HBASE-5596
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.92.1, 0.94.0
Reporter: David S. Wang
Assignee: David S. Wang
Priority: Minor
 Attachments: HBASE-5596.patch


 A few leftover bugs from HBASE-5209.  Comments are documented here:
 https://reviews.apache.org/r/3892/

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5222) Stopping replication via the stop_replication command in hbase shell on a slave cluster isn't acknowledged in the replication sink

2012-03-23 Thread Josh Wymer (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236692#comment-13236692
 ] 

Josh Wymer commented on HBASE-5222:
---

@HV, @JD: Please correct me if I'm wrong here. If you stop replication on the 
master, the logs are no longer stored to be pushed down stream like they would 
with replication enabled. Instead they would be cleaned up based on the default 
timeout. If we need to stop replicating to a slave cluster for maintenance, etc 
we don't want the master throwing away non-replicated logs (thinking it has no 
need to keep them). The bug, however, causes the slave to keep accepting logs 
even while disabled although the other processes on slave cluster respect the 
disabled flag.

 Stopping replication via the stop_replication command in hbase shell on a 
 slave cluster isn't acknowledged in the replication sink
 

 Key: HBASE-5222
 URL: https://issues.apache.org/jira/browse/HBASE-5222
 Project: HBase
  Issue Type: Bug
  Components: replication, shell
Affects Versions: 0.90.4
Reporter: Josh Wymer

 After running stop_replication in the hbase shell on our slave cluster we 
 saw replication continue for weeks. Turns out that the replication sink is 
 missing a check to get the replication state and therefore continued to write.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5128) [uber hbck] Enable hbck to automatically repair table integrity problems as well as region consistency problems while online.

2012-03-23 Thread Jonathan Hsieh (Updated) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jonathan Hsieh updated HBASE-5128:
--

Attachment: hbase-5128-v4.patch
hbase-5128-0.94-v4.patch
hbase-5128-0.92-v4.patch

Updated to address ted's last concern, arcanist fixes, and a handful of findbug
fixes.

[uber hbck] Enable hbck to automatically repair table integrity problems as
well as region consistency problems while online.
-

Attachments: hbase-5128-0.90-v2.patch, hbase-5128-0.90-v2b.patch,
hbase-5128-0.92-v2.patch, hbase-5128-0.92-v4.patch, hbase-5128-0.94-v2.patch,
hbase-5128-0.94-v4.patch, hbase-5128-trunk-v2.patch, hbase-5128-trunk.patch,
hbase-5128-v3.patch, hbase-5128-v4.patch

[jira] [Commented] (HBASE-5128) [uber hbck] Enable hbck to automatically repair table integrity problems as well as region consistency problems while online.

2012-03-23 Thread Jonathan Hsieh (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236704#comment-13236704
]

Jonathan Hsieh commented on HBASE-5128:
---

Porting to changes to 0.90 is causing some test flakyness in that version. My
plan is to work these out (there are more constraints there -- need to figure
out why they flake, need to avoid a master-side HBASE-5563 change, and figure
out the ramifications. I plan on opening a new issue to back port this patch
to 0.90. While trunk/0.94/0.92 versions are very similar, 0.90 has several
differences.

[uber hbck] Enable hbck to automatically repair table integrity problems as
well as region consistency problems while online.
-

[jira] [Commented] (HBASE-5625) Avoid byte buffer allocations when reading a value from a Result object

2012-03-23 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236703#comment-13236703
]

Hadoop QA commented on HBASE-5625:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12519634/5625v2.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.mapreduce.TestImportTsv
org.apache.hadoop.hbase.mapred.TestTableMapReduce
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1275//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1275//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1275//console

This message is automatically generated.

Avoid byte buffer allocations when reading a value from a Result object
---

[jira] [Commented] (HBASE-5128) [uber hbck] Enable hbck to automatically repair table integrity problems as well as region consistency problems while online.

[
https://issues.apache.org/jira/browse/HBASE-5128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236727#comment-13236727
]

jirapos...@reviews.apache.org commented on HBASE-5128:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3435/
---

(Updated 2012-03-23 16:13:50.054043)

Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, and Jean-Daniel
Cryans.

Changes
---

Addresses a few last concerns and does some arcanist and findbugs related
tweaks.

Summary
---

This should nearly be to ready for integration. This has the same control flow
as the trunk/0.92/0.94 versions but has a few differences.

- It needs to track HTableDescritors instead of reading them from the file
system.
- It uses a different HBaseFsckRepair.forceOfflineInZK method -- which for some
reason means we don't need HBASE-5563.
- Uses HServerAddress instead of ServerName

This version is close to what we've used on production clusters.

This addresses bug HBASE-5128.
https://issues.apache.org/jira/browse/HBASE-5128

Diffs (updated)
-

src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 1a4f7f1
src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 3c635d4
src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java d47ef10
src/main/java/org/apache/hadoop/hbase/master/HMaster.java cd1755f
src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java c0aaf65
src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java 5916d9c
src/main/java/org/apache/hadoop/hbase/util/hbck/OfflineMetaRepair.java
d57bb6b

src/main/java/org/apache/hadoop/hbase/util/hbck/TableIntegrityErrorHandler.java
PRE-CREATION

src/main/java/org/apache/hadoop/hbase/util/hbck/TableIntegrityErrorHandlerImpl.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java d9a2a02
src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java 937781d
src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsckComparator.java
0599da1
src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java dbb97f8

src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildBase.java
2b4cac8

src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildHole.java
ebbeead

src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildOverlap.java
b175548

Diff: https://reviews.apache.org/r/3435/diff

Testing
---

All TestHBaseFsck unit tests pass. Currently running full suite.

Thanks,

jmhsieh

[uber hbck] Enable hbck to automatically repair table integrity problems as
well as region consistency problems while online.
-

[jira] [Commented] (HBASE-5625) Avoid byte buffer allocations when reading a value from a Result object

2012-03-23 Thread Himanshu Vashishtha (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236731#comment-13236731
 ] 

Zhihong Yu commented on HBASE-5625:
---

I don't see where loadValue() is called.

Minor comments:
{code}
+long longkeylength = KEY_INFRASTRUCTURE_SIZE + rlength + flength + qlength;
{code}
we normally use variable names such as longKeyLength where the second and 
subsequent words have leading capital letter.
{code}
+if(flength != 0) {
...
+if(qlength != 0) {
{code}
Please insert space between if and (.
{code}
+KeyValue searchTerm = getSearchTerm(kvs[0].getRow(), family, foffset, 
flength, qualifier, qoffset, qlength);
{code}
Recent vote is to limit ling length within 100 characters.
{code}
+if (lLength  Integer.MAX_VALUE)
+  throw new IllegalArgumentException(KeyValue length  + lLength +
+ Integer.MAX_VALUE);
{code}
Please enclose the throw statement in curly braces.
{code}
+   * The KeyValue for the most recent for a given column. If the column does
+   * not exist in the result set - if it wasn't selected in the query 
(Get/Scan)
+   * or just does not exist in the row the return value is null.
{code}
I think the above can be phrased as:
{code}
+   * The most recent KeyValue for a given column. If the column does
+   * not exist in the result set - if it wasn't selected in the query 
(Get/Scan)
+   * or just does not exist in the row - the return value is null.
{code}

 Avoid byte buffer allocations when reading a value from a Result object
 ---

 Key: HBASE-5625
 URL: https://issues.apache.org/jira/browse/HBASE-5625
 Project: HBase
  Issue Type: Improvement
  Components: client
Affects Versions: 0.92.1
Reporter: Tudor Scurtu
  Labels: patch
 Attachments: 5625.txt, 5625v2.txt


 When calling Result.getValue(), an extra dummy KeyValue and its associated 
 underlying byte array are allocated, as well as a persistent buffer that will 
 contain the returned value.
 These can be avoided by reusing a static array for the dummy object and by 
 passing a ByteBuffer object as a value destination buffer to the read method.
 The current functionality is maintained, and we have added a separate method 
 call stack that employs the described changes. I will provide more details 
 with the patch.
 Running tests with a profiler, the reduction of read time seems to be of up 
 to 40%.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5222) Stopping replication via the stop_replication command in hbase shell on a slave cluster isn't acknowledged in the replication sink

[
https://issues.apache.org/jira/browse/HBASE-5222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236742#comment-13236742
]

Himanshu Vashishtha commented on HBASE-5222:

@Josh: In case you want to do some maintenance on the slave cluster, while you
do want to resume the replication once it is restored, you don't need to pull
the stop trigger. The master cluster RS see that they can't connect to the
slave cluster's RS anymore and will keep on waiting till they are up
(sleeping/awaking loop).
But in case you are also stopping the slave cluster's ZK, then you might have
to remove/add it again. As the Master cluster just stops caring about it then.

There is also a jira in the upstream in which you can enable/disable a
particular peer (HBase-3143).
So, afaik, running commands on the slave cluster are futile as its the master
cluster which does all the work.

PS: This is based on few days using plus 1 day code digging (yesterday:)
experience. So, let's see what JD says.

Stopping replication via the stop_replication command in hbase shell on a
slave cluster isn't acknowledged in the replication sink

Key: HBASE-5222
URL: https://issues.apache.org/jira/browse/HBASE-5222
Project: HBase
Issue Type: Bug
Components: replication, shell
Affects Versions: 0.90.4
Reporter: Josh Wymer

After running stop_replication in the hbase shell on our slave cluster we
saw replication continue for weeks. Turns out that the replication sink is
missing a check to get the replication state and therefore continued to write.

[jira] [Updated] (HBASE-5596) Few minor bugs from HBASE-5209


 [ 
https://issues.apache.org/jira/browse/HBASE-5596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David S. Wang updated HBASE-5596:
-

Status: Open  (was: Patch Available)

 Few minor bugs from HBASE-5209
 --

 Key: HBASE-5596
 URL: https://issues.apache.org/jira/browse/HBASE-5596
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.92.1, 0.94.0
Reporter: David S. Wang
Assignee: David S. Wang
Priority: Minor
 Attachments: HBASE_5596.patch


 A few leftover bugs from HBASE-5209.  Comments are documented here:
 https://reviews.apache.org/r/3892/

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5596) Few minor bugs from HBASE-5209


 [ 
https://issues.apache.org/jira/browse/HBASE-5596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David S. Wang updated HBASE-5596:
-

Attachment: HBASE_5596.patch

 Few minor bugs from HBASE-5209
 --

 Key: HBASE-5596
 URL: https://issues.apache.org/jira/browse/HBASE-5596
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.92.1, 0.94.0
Reporter: David S. Wang
Assignee: David S. Wang
Priority: Minor
 Attachments: HBASE_5596.patch


 A few leftover bugs from HBASE-5209.  Comments are documented here:
 https://reviews.apache.org/r/3892/

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5596) Few minor bugs from HBASE-5209


 [ 
https://issues.apache.org/jira/browse/HBASE-5596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David S. Wang updated HBASE-5596:
-

Status: Patch Available  (was: Open)

 Few minor bugs from HBASE-5209
 --

 Key: HBASE-5596
 URL: https://issues.apache.org/jira/browse/HBASE-5596
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.92.1, 0.94.0
Reporter: David S. Wang
Assignee: David S. Wang
Priority: Minor
 Attachments: HBASE_5596.patch


 A few leftover bugs from HBASE-5209.  Comments are documented here:
 https://reviews.apache.org/r/3892/

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5596) Few minor bugs from HBASE-5209


 [ 
https://issues.apache.org/jira/browse/HBASE-5596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David S. Wang updated HBASE-5596:
-

Attachment: (was: HBASE-5596.patch)

 Few minor bugs from HBASE-5209
 --

 Key: HBASE-5596
 URL: https://issues.apache.org/jira/browse/HBASE-5596
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.92.1, 0.94.0
Reporter: David S. Wang
Assignee: David S. Wang
Priority: Minor
 Attachments: HBASE_5596.patch


 A few leftover bugs from HBASE-5209.  Comments are documented here:
 https://reviews.apache.org/r/3892/

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5596) Few minor bugs from HBASE-5209

2012-03-23 Thread David S. Wang (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236764#comment-13236764
 ] 

David S. Wang commented on HBASE-5596:
--

Moved UNKNOWN_SERVERNAME to ServerName.  Otherwise same patch.  Passed all unit 
tests.

 Few minor bugs from HBASE-5209
 --

 Key: HBASE-5596
 URL: https://issues.apache.org/jira/browse/HBASE-5596
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.92.1, 0.94.0
Reporter: David S. Wang
Assignee: David S. Wang
Priority: Minor
 Attachments: HBASE_5596.patch


 A few leftover bugs from HBASE-5209.  Comments are documented here:
 https://reviews.apache.org/r/3892/

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5222) Stopping replication via the stop_replication command in hbase shell on a slave cluster isn't acknowledged in the replication sink

2012-03-23 Thread Jean-Daniel Cryans (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236767#comment-13236767
]

Jean-Daniel Cryans commented on HBASE-5222:
---

bq. So, let's see what JD says.

Here he goes:

bq. When you want to use replication, you ought to run these commands

Not sure which commands you're talking about. In the specific case of
{{stop_replication}}, it's a kill switch in the proper sense (quote from
wikipedia):

bq. a kill switch is designed and configured to a) completely abort the
operation at all costs and b) be operable in a manner that is quick, simple (so
that even a panicking user with impaired executive function can operate it),
and, usually, c) be obvious even to an untrained operator or a bystander

We hit on a) and b), the c) part might not be there yet. The issue here is that
the command is respected on the master cluster (when ran there) but not on the
slave cluster (when ran there).

bq. If you stop replication on the master, the logs are no longer stored to be
pushed down stream like they would with replication enabled.

Yep.

bq. The bug, however, causes the slave to keep accepting logs even while
disabled although the other processes on slave cluster respect the disabled flag

Since it's a kill switch, what's going to happen is the slave cluster is going
to *drop the log edits*. This is not what you want, you want is HBASE-3134.

bq. So, afaik, running commands on the slave cluster are futile as its the
master cluster which does all the work.

I think you understand the issue here reasonably well, and indeed most of the
commands won't do anything on the slave cluster, except here the kill switch
should stop all replication-related activity including applying incoming logs.

Stopping replication via the stop_replication command in hbase shell on a
slave cluster isn't acknowledged in the replication sink

Key: HBASE-5222
URL: https://issues.apache.org/jira/browse/HBASE-5222
Project: HBase
Issue Type: Bug
Components: replication, shell
Affects Versions: 0.90.4
Reporter: Josh Wymer

[jira] [Commented] (HBASE-5128) [uber hbck] Enable hbck to automatically repair table integrity problems as well as region consistency problems while online.

[
https://issues.apache.org/jira/browse/HBASE-5128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236773#comment-13236773
]

Hadoop QA commented on HBASE-5128:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12519649/hbase-5128-v4.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 21 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1276//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1276//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1276//console

This message is automatically generated.

[uber hbck] Enable hbck to automatically repair table integrity problems as
well as region consistency problems while online.
-

[jira] [Commented] (HBASE-5623) Race condition when rolling the HLog and hlogFlush


[ 
https://issues.apache.org/jira/browse/HBASE-5623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236771#comment-13236771
 ] 

Zhihong Yu commented on HBASE-5623:
---

I looped TestLogRollingNoCluster 20 times and didn't see NPE.

 Race condition when rolling the HLog and hlogFlush
 --

 Key: HBASE-5623
 URL: https://issues.apache.org/jira/browse/HBASE-5623
 Project: HBase
  Issue Type: Bug
  Components: wal
Affects Versions: 0.94.0
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Priority: Critical
 Attachments: 5623v2.txt, HBASE-5623_v0.patch


 When doing a ycsb test with a large number of handlers 
 (regionserver.handler.count=60), I get the following exceptions:
 {code}
 Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: 
 java.lang.NullPointerException
   at 
 org.apache.hadoop.io.SequenceFile$Writer.getLength(SequenceFile.java:1099)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.getLength(SequenceFileLogWriter.java:314)
   at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1291)
   at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1388)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:2192)
   at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1985)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3400)
   at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:366)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1351)
   at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:920)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:152)
   at $Proxy1.multi(Unknown Source)
   at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1691)
   at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1689)
   at 
 org.apache.hadoop.hbase.client.ServerCallable.withoutRetries(ServerCallable.java:214)
 {code}
 and 
 {code}
   java.lang.NullPointerException
   at 
 org.apache.hadoop.io.SequenceFile$Writer.checkAndWriteSync(SequenceFile.java:1026)
   at 
 org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1068)
   at 
 org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1035)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.append(SequenceFileLogWriter.java:279)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.hlogFlush(HLog.java:1237)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1271)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1391)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:2192)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1985)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3400)
   at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:366)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1351)
 {code}
 It seems the root cause of the issue is that we open a new log writer and 
 close the old one at HLog#rollWriter() holding the updateLock, but the other 
 threads doing syncer() calls
 {code} 
 logSyncerThread.hlogFlush(this.writer);
 {code}
 without holding the updateLock. LogSyncer only synchronizes against 
 concurrent appends and flush(), but not on the passed writer, which can be 
 closed already by rollWriter(). In this case, since 
 SequenceFile#Writer.close() sets it's out field as null, we get the NPE. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5616) Make compaction code standalone


 [ 
https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-5616:
-

Assignee: stack
  Status: Open  (was: Patch Available)

 Make compaction code standalone
 ---

 Key: HBASE-5616
 URL: https://issues.apache.org/jira/browse/HBASE-5616
 Project: HBase
  Issue Type: Improvement
Reporter: stack
Assignee: stack
 Attachments: 5616.txt, 5616v3.txt, 5616v6.txt, standalone.txt


 This is part of hbase-2462.  Make the compaction code standalone so can run 
 it independent of hbase.  Will make it easier to profile and try stuff out.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5616) Make compaction code standalone


 [ 
https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-5616:
-

Attachment: 5616v6.txt

Fixed the test failure.

Also changed my mind moving CompactionTool out to src/test rather than have it 
in main code base.  I kept the breaking out  of the compaction code from Store 
into its own class and made it so its possible to stand up a single Store in a 
unit test.

Until CompactionTool can be done w/o my need of the Mockito crutch because the 
code is a hairball, I think CompactionTool has to be out in test.

 Make compaction code standalone
 ---

 Key: HBASE-5616
 URL: https://issues.apache.org/jira/browse/HBASE-5616
 Project: HBase
  Issue Type: Improvement
Reporter: stack
 Attachments: 5616.txt, 5616v3.txt, 5616v6.txt, standalone.txt


 This is part of hbase-2462.  Make the compaction code standalone so can run 
 it independent of hbase.  Will make it easier to profile and try stuff out.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5616) Make compaction code standalone

2012-03-23 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-5616:
-

Status: Patch Available  (was: Open)

 Make compaction code standalone
 ---

 Key: HBASE-5616
 URL: https://issues.apache.org/jira/browse/HBASE-5616
 Project: HBase
  Issue Type: Improvement
Reporter: stack
Assignee: stack
 Attachments: 5616.txt, 5616v3.txt, 5616v6.txt, standalone.txt


 This is part of hbase-2462.  Make the compaction code standalone so can run 
 it independent of hbase.  Will make it easier to profile and try stuff out.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5444) Add PB-based calls to HMasterRegionInterface


[ 
https://issues.apache.org/jira/browse/HBASE-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236785#comment-13236785
 ] 

jirapos...@reviews.apache.org commented on HBASE-5444:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4463/#review6286
---



pom.xml
https://reviews.apache.org/r/4463/#comment13630

I have some enhancement to the pom.  Please check my patch on

https://issues.apache.org/jira/browse/HBASE-5619



src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon
https://reviews.apache.org/r/4463/#comment13632

I called it HBaseProtos instead of CommonProtocol since it is not a 
protocol itself.



src/main/java/org/apache/hadoop/hbase/protobuf/MasterRegionInterface.java
https://reviews.apache.org/r/4463/#comment13633

Should it be Private?  Should we call it MasterRegionProtocol?



src/main/proto/MasterRegionProtocol.proto
https://reviews.apache.org/r/4463/#comment13634

This looks like a property.  I have similar message for RegionProtocols.  
Can we put this in hbase.proto and call it Property so that we can share it?



src/main/proto/hbase.proto
https://reviews.apache.org/r/4463/#comment13642

Do we have column family information?


- Jimmy


On 2012-03-23 04:58:51, Gregory Chanan wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4463/
bq.  ---
bq.  
bq.  (Updated 2012-03-23 04:58:51)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Adds PB-based calls replacing HMasterRegionInterface.
bq.  
bq.  There are some temporary hacks, e.g. converting PB-based ServerLoad to 
existing HServerLoad so I didn't need to convert ClusterStatus (which brings in 
a lot of other changes).  That will be cleaned up in HBASE-5445.
bq.  
bq.  
bq.  This addresses bug HBASE-5444.
bq.  https://issues.apache.org/jira/browse/HBASE-5444
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.pom.xml 10b13ef 
bq.
src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon 
69434f7 
bq.
src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon 
ae76204 
bq.src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 9df4c10 
bq.src/main/java/org/apache/hadoop/hbase/HConstants.java 347 
bq.src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java 
cbfa489 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 0db2760 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 2916d68 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 
fd97830 
bq.src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java f1f06b0 
bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 
d47ef10 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java cd1755f 
bq.src/main/java/org/apache/hadoop/hbase/master/MXBean.java 7f44dc2 
bq.src/main/java/org/apache/hadoop/hbase/master/MXBeanImpl.java 45b8fe7 
bq.src/main/java/org/apache/hadoop/hbase/master/MasterDumpServlet.java 
be63838 
bq.src/main/java/org/apache/hadoop/hbase/master/ServerManager.java cbd55f7 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/MasterRegionInterface.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/PBHelper.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
e0af8fb 
bq.src/main/proto/MasterRegionProtocol.proto PRE-CREATION 
bq.src/main/proto/hbase.proto PRE-CREATION 
bq.src/main/resources/hbase-webapps/master/table.jsp 811df46 
bq.src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 6af9188 
bq.src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java 
368a0e5 
bq.src/test/java/org/apache/hadoop/hbase/io/TestHbaseObjectWritable.java 
f2f8ee3 
bq.src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 
841649a 
bq.src/test/java/org/apache/hadoop/hbase/master/TestMXBean.java 379f70c 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java
 e99d251 
bq.  
bq.  Diff: https://reviews.apache.org/r/4463/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Ran jenkins job, all unit tests passed.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Gregory
bq.  
bq.



 Add PB-based calls to HMasterRegionInterface
 

 Key: HBASE-5444
 URL: https://issues.apache.org/jira/browse/HBASE-5444
 Project: HBase
  Issue Type: Sub-task

[jira] [Commented] (HBASE-5616) Make compaction code standalone

2012-03-23 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236788#comment-13236788
 ] 

jirapos...@reviews.apache.org commented on HBASE-5616:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4469/
---

Review request for hbase.


Summary
---

Introduces a standalone CompactionTool under src/test. Can call it from its main
and have it run compactions on arbitrary storefiles.

Compaction code is moved out of Store into a new Compactor class.

CompactionTool needs a Store and a mocked up Region to run because
compacting uses a StoreScanner (A StoreScanner needs a Store. A
Store needs an HRegion). Rather than expect to be passed a coherent
HRegion pointer, instead, we fake up one using Mockito.

I tried to break out of HRegion a Region Interface. This Region Interface
would have a basic subset of HRegion functionality and we'd pass this
instead of HRegion to SplitTransaction, Store, CompactionRequest, etc.
but the change would be massive. Everything expects to be able to do
anything on an HRegion. This is work we need to do but Im not doing
it as part of this patch.

M src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
Use new CompactionTool instead of asking Store directly to compact.
M src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java
Formatting.
M src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
Format the log message so properly spaced.
A src/test/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java
New compactiontool. Runs all compactions.
M src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
Make getOpenAndCloseThreadPool static.
M src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
Move compaction code out to CompactionTool.
Refactor so can override ttl and Store home directory so Store
is more mockable; can now stand up a Store on its own w/o real
HRegion context.
M src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
Formatting.


This addresses bug hbase-5616.
https://issues.apache.org/jira/browse/hbase-5616


Diffs
-

  src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java b52e5d3 
  src/main/java/org/apache/hadoop/hbase/regionserver/Compactor.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 4efdc6b 
  src/main/java/org/apache/hadoop/hbase/regionserver/Store.java dcede5a 
  
src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionProgress.java
 553eee0 
  src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java d2329e1 
  src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 4bfd42f 
  src/test/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java 
PRE-CREATION 
  src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java 
91ac652 

Diff: https://reviews.apache.org/r/4469/diff


Testing
---


Thanks,

Michael



 Make compaction code standalone
 ---

 Key: HBASE-5616
 URL: https://issues.apache.org/jira/browse/HBASE-5616
 Project: HBase
  Issue Type: Improvement
Reporter: stack
Assignee: stack
 Attachments: 5616.txt, 5616v3.txt, 5616v6.txt, standalone.txt


 This is part of hbase-2462.  Make the compaction code standalone so can run 
 it independent of hbase.  Will make it easier to profile and try stuff out.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size

2012-03-23 Thread Jean-Daniel Cryans (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236790#comment-13236790
 ] 

Jean-Daniel Cryans commented on HBASE-5190:
---

It's not exactly 10, it's num_handlers * 10 so by default 100. Is that better?

 Limit the IPC queue size based on calls' payload size
 -

 Key: HBASE-5190
 URL: https://issues.apache.org/jira/browse/HBASE-5190
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.90.5
Reporter: Jean-Daniel Cryans
Assignee: Jean-Daniel Cryans
 Fix For: 0.94.0

 Attachments: HBASE-5190-v2.patch, HBASE-5190-v3.patch, 
 HBASE-5190.patch


 Currently we limit the number of calls in the IPC queue only on their count. 
 It used to be really high and was dropped down recently to num_handlers * 10 
 (so 100 by default) because it was easy to OOME yourself when huge calls were 
 being queued. It's still possible to hit this problem if you use really big 
 values and/or a lot of handlers, so the idea is that we should take into 
 account the payload size. I can see 3 solutions:
  - Do the accounting outside of the queue itself for all calls coming in and 
 out and when a call doesn't fit, throw a retryable exception.
  - Same accounting but instead block the call when it comes in until space is 
 made available.
  - Add a new parameter for the maximum size (in bytes) of a Call and then set 
 the size the IPC queue (in terms of the number of items) so that it could 
 only contain as many items as some predefined maximum size (in bytes) for the 
 whole queue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size

2012-03-23 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236810#comment-13236810
 ] 

stack commented on HBASE-5190:
--

@JD Can do in another issue.

 Limit the IPC queue size based on calls' payload size
 -

 Key: HBASE-5190
 URL: https://issues.apache.org/jira/browse/HBASE-5190
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.90.5
Reporter: Jean-Daniel Cryans
Assignee: Jean-Daniel Cryans
 Fix For: 0.94.0

 Attachments: HBASE-5190-v2.patch, HBASE-5190-v3.patch, 
 HBASE-5190.patch


 Currently we limit the number of calls in the IPC queue only on their count. 
 It used to be really high and was dropped down recently to num_handlers * 10 
 (so 100 by default) because it was easy to OOME yourself when huge calls were 
 being queued. It's still possible to hit this problem if you use really big 
 values and/or a lot of handlers, so the idea is that we should take into 
 account the payload size. I can see 3 solutions:
  - Do the accounting outside of the queue itself for all calls coming in and 
 out and when a call doesn't fit, throw a retryable exception.
  - Same accounting but instead block the call when it comes in until space is 
 made available.
  - Add a new parameter for the maximum size (in bytes) of a Call and then set 
 the size the IPC queue (in terms of the number of items) so that it could 
 only contain as many items as some predefined maximum size (in bytes) for the 
 whole queue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5616) Make compaction code standalone


[ 
https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236819#comment-13236819
 ] 

jirapos...@reviews.apache.org commented on HBASE-5616:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4469/#review6293
---


CompactionTool is a cool idea.
It's early, so I don't follow completely why this required moving the 
compaction code into a separate class, especially because you need an instance 
of a store to run its methods.


src/main/java/org/apache/hadoop/hbase/regionserver/Compactor.java
https://reviews.apache.org/r/4469/#comment13643

If you need to pass in an instance of Store, what is the advantage?
Could also make it a public static helper method.



src/test/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java
https://reviews.apache.org/r/4469/#comment13645

That would have been possible before separating the compaction code out, 
right?


- Lars


On 2012-03-23 17:02:01, Michael Stack wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4469/
bq.  ---
bq.  
bq.  (Updated 2012-03-23 17:02:01)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Introduces a standalone CompactionTool under src/test. Can call it from 
its main
bq.  and have it run compactions on arbitrary storefiles.
bq.  
bq.  Compaction code is moved out of Store into a new Compactor class.
bq.  
bq.  CompactionTool needs a Store and a mocked up Region to run because
bq.  compacting uses a StoreScanner (A StoreScanner needs a Store. A
bq.  Store needs an HRegion). Rather than expect to be passed a coherent
bq.  HRegion pointer, instead, we fake up one using Mockito.
bq.  
bq.  I tried to break out of HRegion a Region Interface. This Region Interface
bq.  would have a basic subset of HRegion functionality and we'd pass this
bq.  instead of HRegion to SplitTransaction, Store, CompactionRequest, etc.
bq.  but the change would be massive. Everything expects to be able to do
bq.  anything on an HRegion. This is work we need to do but Im not doing
bq.  it as part of this patch.
bq.  
bq.  M src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
bq.  Use new CompactionTool instead of asking Store directly to compact.
bq.  M src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java
bq.  Formatting.
bq.  M src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
bq.  Format the log message so properly spaced.
bq.  A src/test/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java
bq.  New compactiontool. Runs all compactions.
bq.  M src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
bq.  Make getOpenAndCloseThreadPool static.
bq.  M src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
bq.  Move compaction code out to CompactionTool.
bq.  Refactor so can override ttl and Store home directory so Store
bq.  is more mockable; can now stand up a Store on its own w/o real
bq.  HRegion context.
bq.  M src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
bq.  Formatting.
bq.  
bq.  
bq.  This addresses bug hbase-5616.
bq.  https://issues.apache.org/jira/browse/hbase-5616
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 
b52e5d3 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/Compactor.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 4efdc6b 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/Store.java dcede5a 
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionProgress.java
 553eee0 
bq.src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java d2329e1 
bq.src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 4bfd42f 
bq.src/test/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java 
PRE-CREATION 
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java 
91ac652 
bq.  
bq.  Diff: https://reviews.apache.org/r/4469/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Michael
bq.  
bq.



 Make compaction code standalone
 ---

 Key: HBASE-5616
 URL: https://issues.apache.org/jira/browse/HBASE-5616
 Project: HBase
  Issue Type: Improvement
Reporter: stack
Assignee: stack
 Attachments: 5616.txt, 5616v3.txt, 5616v6.txt, standalone.txt


 This is part of hbase-2462.  Make the compaction code standalone so can run

[jira] [Commented] (HBASE-5613) ThriftServer getTableRegions does not return serverName and port

2012-03-23 Thread Scott Chen (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236822#comment-13236822
 ] 

Scott Chen commented on HBASE-5613:
---

@Ted: You're right. I haven't rebase this branch. Let me do that and update the 
patch.

 ThriftServer getTableRegions does not return serverName and port
 

 Key: HBASE-5613
 URL: https://issues.apache.org/jira/browse/HBASE-5613
 Project: HBase
  Issue Type: Bug
  Components: thrift
Reporter: Scott Chen
Assignee: Scott Chen
Priority: Minor
 Fix For: 0.94.0, 0.96.0

 Attachments: HBASE-5613.0.94.txt, HBASE-5613.D2403.1.patch, 
 HBASE-5613.D2403.2.patch, HBASE-5613.D2403.3.patch, HBASE-5613.D2403.4.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5616) Make compaction code standalone

2012-03-23 Thread Matt Corgan (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236825#comment-13236825
 ] 

Matt Corgan commented on HBASE-5616:


btw - this feature is exactly what i need to continue optimizing the Trie 
DataBlockEncoder in HBASE-4676.  Thanks Stack!

 Make compaction code standalone
 ---

 Key: HBASE-5616
 URL: https://issues.apache.org/jira/browse/HBASE-5616
 Project: HBase
  Issue Type: Improvement
Reporter: stack
Assignee: stack
 Attachments: 5616.txt, 5616v3.txt, 5616v6.txt, standalone.txt


 This is part of hbase-2462.  Make the compaction code standalone so can run 
 it independent of hbase.  Will make it easier to profile and try stuff out.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5596) Few minor bugs from HBASE-5209

2012-03-23 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236832#comment-13236832
]

Hadoop QA commented on HBASE-5596:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12519657/HBASE_5596.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestRollingRestart
org.apache.hadoop.hbase.master.TestMasterShutdown
org.apache.hadoop.hbase.master.TestDistributedLogSplitting
org.apache.hadoop.hbase.client.TestInstantSchemaChangeSplit
org.apache.hadoop.hbase.client.TestInstantSchemaChangeFailover

org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable
org.apache.hadoop.hbase.master.TestMasterFailover
org.apache.hadoop.hbase.mapreduce.TestImportTsv
org.apache.hadoop.hbase.mapred.TestTableMapReduce
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1277//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1277//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1277//console

This message is automatically generated.

Few minor bugs from HBASE-5209
--

Key: HBASE-5596
URL: https://issues.apache.org/jira/browse/HBASE-5596
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.92.1, 0.94.0
Reporter: David S. Wang
Assignee: David S. Wang
Priority: Minor
Attachments: HBASE_5596.patch

A few leftover bugs from HBASE-5209. Comments are documented here:
https://reviews.apache.org/r/3892/

[jira] [Commented] (HBASE-5616) Make compaction code standalone


[ 
https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236835#comment-13236835
 ] 

jirapos...@reviews.apache.org commented on HBASE-5616:
--



bq.  On 2012-03-23 17:32:04, Lars Hofhansl wrote:
bq.   CompactionTool is a cool idea.
bq.   It's early, so I don't follow completely why this required moving the 
compaction code into a separate class, especially because you need an instance 
of a store to run its methods.

Store is 2200 lines long.

Need to make a start somewhere breaking up these Store and HRegion, etc., 
behemoths.  The tendency otherwise is that they get bigger with time.

Also, arguing its all a ball of wax so we should shut down any attempt at 
untangling is probably not what you were reasoning (smile).

My main motive breaking out CompactionTool made is so I could run compactions 
in a profiler w/o a bunch of other moving parts in the way.  I did some of that 
yesterday.  Nothing obviously dumb going on at first glance.  Requires more 
study figuring optimizations.

Thanks for the review.


bq.  On 2012-03-23 17:32:04, Lars Hofhansl wrote:
bq.   src/main/java/org/apache/hadoop/hbase/regionserver/Compactor.java, line 
45
bq.   https://reviews.apache.org/r/4469/diff/1/?file=95053#file95053line45
bq.  
bq.   If you need to pass in an instance of Store, what is the advantage?
bq.   Could also make it a public static helper method.

I could do that.  If I went this way, it'd be harder to evolve though (public, 
static passing in all args).

This review comment makes me want to shutdown access to Compactor even more -- 
go via CompactionTool if you need it.   I should add @InterfaceAudience.Private 
and remove the 'public' from the Class.  If you are good w/ this, I'll stick up 
another patch w/ that on it.


bq.  On 2012-03-23 17:32:04, Lars Hofhansl wrote:
bq.   src/test/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java, 
line 135
bq.   https://reviews.apache.org/r/4469/diff/1/?file=95059#file95059line135
bq.  
bq.   That would have been possible before separating the compaction code 
out, right?

Yes.  Would have made no sense since only the one client.


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4469/#review6293
---


On 2012-03-23 17:02:01, Michael Stack wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4469/
bq.  ---
bq.  
bq.  (Updated 2012-03-23 17:02:01)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Introduces a standalone CompactionTool under src/test. Can call it from 
its main
bq.  and have it run compactions on arbitrary storefiles.
bq.  
bq.  Compaction code is moved out of Store into a new Compactor class.
bq.  
bq.  CompactionTool needs a Store and a mocked up Region to run because
bq.  compacting uses a StoreScanner (A StoreScanner needs a Store. A
bq.  Store needs an HRegion). Rather than expect to be passed a coherent
bq.  HRegion pointer, instead, we fake up one using Mockito.
bq.  
bq.  I tried to break out of HRegion a Region Interface. This Region Interface
bq.  would have a basic subset of HRegion functionality and we'd pass this
bq.  instead of HRegion to SplitTransaction, Store, CompactionRequest, etc.
bq.  but the change would be massive. Everything expects to be able to do
bq.  anything on an HRegion. This is work we need to do but Im not doing
bq.  it as part of this patch.
bq.  
bq.  M src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
bq.  Use new CompactionTool instead of asking Store directly to compact.
bq.  M src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java
bq.  Formatting.
bq.  M src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
bq.  Format the log message so properly spaced.
bq.  A src/test/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java
bq.  New compactiontool. Runs all compactions.
bq.  M src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
bq.  Make getOpenAndCloseThreadPool static.
bq.  M src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
bq.  Move compaction code out to CompactionTool.
bq.  Refactor so can override ttl and Store home directory so Store
bq.  is more mockable; can now stand up a Store on its own w/o real
bq.  HRegion context.
bq.  M src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
bq.  Formatting.
bq.  
bq.  
bq.  This addresses bug hbase-5616.
bq.  https://issues.apache.org/jira/browse/hbase-5616
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.

[jira] [Commented] (HBASE-5616) Make compaction code standalone

2012-03-23 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236836#comment-13236836
]

Hadoop QA commented on HBASE-5616:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12519660/5616v6.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 5 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1278//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1278//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1278//console

This message is automatically generated.

Make compaction code standalone
---

Key: HBASE-5616
URL: https://issues.apache.org/jira/browse/HBASE-5616
Project: HBase
Issue Type: Improvement
Reporter: stack
Assignee: stack
Attachments: 5616.txt, 5616v3.txt, 5616v6.txt, standalone.txt

This is part of hbase-2462. Make the compaction code standalone so can run
it independent of hbase. Will make it easier to profile and try stuff out.

[jira] [Commented] (HBASE-5616) Make compaction code standalone


[ 
https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236842#comment-13236842
 ] 

jirapos...@reviews.apache.org commented on HBASE-5616:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4469/
---

(Updated 2012-03-23 17:53:11.446024)


Review request for hbase.


Changes
---

Shutdown access to new Compactor class so can evolve it easier in future.


Summary
---

Introduces a standalone CompactionTool under src/test. Can call it from its main
and have it run compactions on arbitrary storefiles.

Compaction code is moved out of Store into a new Compactor class.

CompactionTool needs a Store and a mocked up Region to run because
compacting uses a StoreScanner (A StoreScanner needs a Store. A
Store needs an HRegion). Rather than expect to be passed a coherent
HRegion pointer, instead, we fake up one using Mockito.

I tried to break out of HRegion a Region Interface. This Region Interface
would have a basic subset of HRegion functionality and we'd pass this
instead of HRegion to SplitTransaction, Store, CompactionRequest, etc.
but the change would be massive. Everything expects to be able to do
anything on an HRegion. This is work we need to do but Im not doing
it as part of this patch.

M src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
Use new CompactionTool instead of asking Store directly to compact.
M src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java
Formatting.
M src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
Format the log message so properly spaced.
A src/test/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java
New compactiontool. Runs all compactions.
M src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
Make getOpenAndCloseThreadPool static.
M src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
Move compaction code out to CompactionTool.
Refactor so can override ttl and Store home directory so Store
is more mockable; can now stand up a Store on its own w/o real
HRegion context.
M src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
Formatting.


This addresses bug hbase-5616.
https://issues.apache.org/jira/browse/hbase-5616


Diffs (updated)
-

  src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java b52e5d3 
  src/main/java/org/apache/hadoop/hbase/regionserver/Compactor.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 4efdc6b 
  src/main/java/org/apache/hadoop/hbase/regionserver/Store.java dcede5a 
  
src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionProgress.java
 553eee0 
  src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java d2329e1 
  src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 4bfd42f 
  src/test/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java 
PRE-CREATION 
  src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java 
91ac652 

Diff: https://reviews.apache.org/r/4469/diff


Testing
---


Thanks,

Michael



 Make compaction code standalone
 ---

 Key: HBASE-5616
 URL: https://issues.apache.org/jira/browse/HBASE-5616
 Project: HBase
  Issue Type: Improvement
Reporter: stack
Assignee: stack
 Attachments: 5616.txt, 5616v3.txt, 5616v6.txt, 5616v7.txt, 
 standalone.txt


 This is part of hbase-2462.  Make the compaction code standalone so can run 
 it independent of hbase.  Will make it easier to profile and try stuff out.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5616) Make compaction code standalone


 [ 
https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-5616:
-

Attachment: 5616v7.txt

Incorporate Lars review. Shutdown access to Compactor so can be easily evolved 
in future.

 Make compaction code standalone
 ---

 Key: HBASE-5616
 URL: https://issues.apache.org/jira/browse/HBASE-5616
 Project: HBase
  Issue Type: Improvement
Reporter: stack
Assignee: stack
 Attachments: 5616.txt, 5616v3.txt, 5616v6.txt, 5616v7.txt, 
 standalone.txt


 This is part of hbase-2462.  Make the compaction code standalone so can run 
 it independent of hbase.  Will make it easier to profile and try stuff out.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5613) ThriftServer getTableRegions does not return serverName and port

2012-03-23 Thread Phabricator (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HBASE-5613:
---

Attachment: HBASE-5613.D2403.5.patch

sc updated the revision HBASE-5613 [jira] ThriftServer getTableRegions does 
not return serverName and port.
Reviewers: dhruba, tedyu, JIRA

  Rebase

REVISION DETAIL
  https://reviews.facebook.net/D2403

AFFECTED FILES
  src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java
  src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServer.java


 ThriftServer getTableRegions does not return serverName and port
 

 Key: HBASE-5613
 URL: https://issues.apache.org/jira/browse/HBASE-5613
 Project: HBase
  Issue Type: Bug
  Components: thrift
Reporter: Scott Chen
Assignee: Scott Chen
Priority: Minor
 Fix For: 0.94.0, 0.96.0

 Attachments: HBASE-5613.0.94.txt, HBASE-5613.D2403.1.patch, 
 HBASE-5613.D2403.2.patch, HBASE-5613.D2403.3.patch, HBASE-5613.D2403.4.patch, 
 HBASE-5613.D2403.5.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5596) Few minor bugs from HBASE-5209

2012-03-23 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David S. Wang updated HBASE-5596:
-

Status: Open  (was: Patch Available)

Evidently the tests that passed on my machine failed here.  Looking into why 
this is intermittent.

 Few minor bugs from HBASE-5209
 --

 Key: HBASE-5596
 URL: https://issues.apache.org/jira/browse/HBASE-5596
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.92.1, 0.94.0
Reporter: David S. Wang
Assignee: David S. Wang
Priority: Minor
 Attachments: HBASE_5596.patch


 A few leftover bugs from HBASE-5209.  Comments are documented here:
 https://reviews.apache.org/r/3892/

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5619) Create PB protocols for HRegionInterface


[ 
https://issues.apache.org/jira/browse/HBASE-5619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236846#comment-13236846
 ] 

jirapos...@reviews.apache.org commented on HBASE-5619:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4054/#review6294
---


Looks good overall.
Minor comments below.


src/main/proto/RegionAdmin.proto
https://reviews.apache.org/r/4054/#comment13644

Is it possible that clusterIdLeastSignificantBits is present but 
clusterIdMostSignificantBits is absent ?



src/main/proto/RegionAdmin.proto
https://reviews.apache.org/r/4054/#comment13646

Should this rpc be called getOnlineRegions ?



src/main/proto/RegionClient.proto
https://reviews.apache.org/r/4054/#comment13647

Increment doesn't extend Mutation:

public class Increment implements Row {




src/main/proto/RegionClient.proto
https://reviews.apache.org/r/4054/#comment13649

Remove 'is'



src/main/proto/RegionClient.proto
https://reviews.apache.org/r/4054/#comment13651

white space.



src/main/proto/RegionClient.proto
https://reviews.apache.org/r/4054/#comment13652

Should this line be removed ?



src/main/proto/RegionClient.proto
https://reviews.apache.org/r/4054/#comment13653

'it's appeared' - 'it appears'



src/main/proto/RegionClient.proto
https://reviews.apache.org/r/4054/#comment13654

I don't see region member in Exec.java
Do we need this ?


- Ted


On 2012-03-22 20:20:39, Jimmy Xiang wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4054/
bq.  ---
bq.  
bq.  (Updated 2012-03-22 20:20:39)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This is the first draft of the ProtoBuff HRegionProtocol.  The 
corresponding java vs pb method mapping is attached to the jira: 
https://issues.apache.org/jira/browse/HBASE-5443
bq.  
bq.  Please review.  I'd like to move ahead after we get to some agreement.
bq.  
bq.  
bq.  This addresses bug HBASE-5619.
bq.  https://issues.apache.org/jira/browse/HBASE-5619
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.pom.xml 10b13ef 
bq.src/main/proto/RegionAdmin.proto PRE-CREATION 
bq.src/main/proto/RegionClient.proto PRE-CREATION 
bq.src/main/proto/hbase.proto PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/4054/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Jimmy
bq.  
bq.



 Create PB protocols for HRegionInterface
 

 Key: HBASE-5619
 URL: https://issues.apache.org/jira/browse/HBASE-5619
 Project: HBase
  Issue Type: Sub-task
  Components: ipc, master, migration, regionserver
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0

 Attachments: hbase-5619.patch


 Subtask of HBase-5443, separate HRegionInterface into admin protocol and 
 client protocol, create the PB protocol buffer files

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5623) Race condition when rolling the HLog and hlogFlush


[ 
https://issues.apache.org/jira/browse/HBASE-5623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236853#comment-13236853
 ] 

Lars Hofhansl commented on HBASE-5623:
--

Looking at SequenceFileLogWriter, I notice that other methods on it catch 
Exception or even NullPointerException (see close() and sync()).

If we want maximum concurrency here, we need to do the same we do for 
writer.sync(). I.e. try without lock held, if failing get the lock and retry, 
along with SequenceFileLogWriter.getLength() catching Exception and retrow as 
IOException.

 Race condition when rolling the HLog and hlogFlush
 --

 Key: HBASE-5623
 URL: https://issues.apache.org/jira/browse/HBASE-5623
 Project: HBase
  Issue Type: Bug
  Components: wal
Affects Versions: 0.94.0
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Priority: Critical
 Attachments: 5623v2.txt, HBASE-5623_v0.patch


 When doing a ycsb test with a large number of handlers 
 (regionserver.handler.count=60), I get the following exceptions:
 {code}
 Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: 
 java.lang.NullPointerException
   at 
 org.apache.hadoop.io.SequenceFile$Writer.getLength(SequenceFile.java:1099)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.getLength(SequenceFileLogWriter.java:314)
   at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1291)
   at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1388)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:2192)
   at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1985)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3400)
   at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:366)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1351)
   at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:920)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:152)
   at $Proxy1.multi(Unknown Source)
   at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1691)
   at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1689)
   at 
 org.apache.hadoop.hbase.client.ServerCallable.withoutRetries(ServerCallable.java:214)
 {code}
 and 
 {code}
   java.lang.NullPointerException
   at 
 org.apache.hadoop.io.SequenceFile$Writer.checkAndWriteSync(SequenceFile.java:1026)
   at 
 org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1068)
   at 
 org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1035)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.append(SequenceFileLogWriter.java:279)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.hlogFlush(HLog.java:1237)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1271)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1391)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:2192)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1985)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3400)
   at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:366)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1351)
 {code}
 It seems the root cause of the issue is that we open a new log writer and 
 close the old one at HLog#rollWriter() holding the updateLock, but the other 
 threads doing syncer() calls
 {code} 
 logSyncerThread.hlogFlush(this.writer);
 {code}
 without holding the updateLock. LogSyncer only synchronizes against 
 concurrent appends and flush(), but not on the passed writer, which can be 
 closed already by rollWriter(). In this case, since 
 SequenceFile#Writer.close() sets it's out field as null, we get the NPE. 

--
This

[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size

2012-03-23 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236855#comment-13236855
 ] 

Lars Hofhansl commented on HBASE-5190:
--

@J-D: Oh yeah forgot that. Fine then. :)
Still +1

 Limit the IPC queue size based on calls' payload size
 -

 Key: HBASE-5190
 URL: https://issues.apache.org/jira/browse/HBASE-5190
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.90.5
Reporter: Jean-Daniel Cryans
Assignee: Jean-Daniel Cryans
 Fix For: 0.94.0

 Attachments: HBASE-5190-v2.patch, HBASE-5190-v3.patch, 
 HBASE-5190.patch


 Currently we limit the number of calls in the IPC queue only on their count. 
 It used to be really high and was dropped down recently to num_handlers * 10 
 (so 100 by default) because it was easy to OOME yourself when huge calls were 
 being queued. It's still possible to hit this problem if you use really big 
 values and/or a lot of handlers, so the idea is that we should take into 
 account the payload size. I can see 3 solutions:
  - Do the accounting outside of the queue itself for all calls coming in and 
 out and when a call doesn't fit, throw a retryable exception.
  - Same accounting but instead block the call when it comes in until space is 
 made available.
  - Add a new parameter for the maximum size (in bytes) of a Call and then set 
 the size the IPC queue (in terms of the number of items) so that it could 
 only contain as many items as some predefined maximum size (in bytes) for the 
 whole queue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5616) Make compaction code standalone


[ 
https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236860#comment-13236860
 ] 

jirapos...@reviews.apache.org commented on HBASE-5616:
--



bq.  On 2012-03-23 17:32:04, Lars Hofhansl wrote:
bq.   CompactionTool is a cool idea.
bq.   It's early, so I don't follow completely why this required moving the 
compaction code into a separate class, especially because you need an instance 
of a store to run its methods.
bq.  
bq.  Michael Stack wrote:
bq.  Store is 2200 lines long.
bq.  
bq.  Need to make a start somewhere breaking up these Store and HRegion, 
etc., behemoths.  The tendency otherwise is that they get bigger with time.
bq.  
bq.  Also, arguing its all a ball of wax so we should shut down any attempt 
at untangling is probably not what you were reasoning (smile).
bq.  
bq.  My main motive breaking out CompactionTool made is so I could run 
compactions in a profiler w/o a bunch of other moving parts in the way.  I did 
some of that yesterday.  Nothing obviously dumb going on at first glance.  
Requires more study figuring optimizations.
bq.  
bq.  Thanks for the review.

Fair enough.


- Lars


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4469/#review6293
---


On 2012-03-23 17:53:11, Michael Stack wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4469/
bq.  ---
bq.  
bq.  (Updated 2012-03-23 17:53:11)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Introduces a standalone CompactionTool under src/test. Can call it from 
its main
bq.  and have it run compactions on arbitrary storefiles.
bq.  
bq.  Compaction code is moved out of Store into a new Compactor class.
bq.  
bq.  CompactionTool needs a Store and a mocked up Region to run because
bq.  compacting uses a StoreScanner (A StoreScanner needs a Store. A
bq.  Store needs an HRegion). Rather than expect to be passed a coherent
bq.  HRegion pointer, instead, we fake up one using Mockito.
bq.  
bq.  I tried to break out of HRegion a Region Interface. This Region Interface
bq.  would have a basic subset of HRegion functionality and we'd pass this
bq.  instead of HRegion to SplitTransaction, Store, CompactionRequest, etc.
bq.  but the change would be massive. Everything expects to be able to do
bq.  anything on an HRegion. This is work we need to do but Im not doing
bq.  it as part of this patch.
bq.  
bq.  M src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
bq.  Use new CompactionTool instead of asking Store directly to compact.
bq.  M src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java
bq.  Formatting.
bq.  M src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
bq.  Format the log message so properly spaced.
bq.  A src/test/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java
bq.  New compactiontool. Runs all compactions.
bq.  M src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
bq.  Make getOpenAndCloseThreadPool static.
bq.  M src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
bq.  Move compaction code out to CompactionTool.
bq.  Refactor so can override ttl and Store home directory so Store
bq.  is more mockable; can now stand up a Store on its own w/o real
bq.  HRegion context.
bq.  M src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
bq.  Formatting.
bq.  
bq.  
bq.  This addresses bug hbase-5616.
bq.  https://issues.apache.org/jira/browse/hbase-5616
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 
b52e5d3 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/Compactor.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 4efdc6b 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/Store.java dcede5a 
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionProgress.java
 553eee0 
bq.src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java d2329e1 
bq.src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 4bfd42f 
bq.src/test/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java 
PRE-CREATION 
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java 
91ac652 
bq.  
bq.  Diff: https://reviews.apache.org/r/4469/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Michael
bq.  
bq.



 Make compaction code standalone
 ---

 Key: HBASE-5616

[jira] [Commented] (HBASE-5494) Introduce a zk hosted table-wide read/write lock so only one table operation at a time

[
https://issues.apache.org/jira/browse/HBASE-5494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236857#comment-13236857
]

stack commented on HBASE-5494:
--

Thinking on this, I started to sketch a scheme where /unassigned in zk became
instead /regions and under /regions there'd be a subdir per table and then
under each table, you'd do something like what is described here
http://zookeeper.apache.org/doc/r3.1.2/recipes.html#Shared+Locks only instead
of read/write we'd have a sort of exclusive-access vs non-exclusive-access.

Already the above would be a massive refactor not to mind compounding it with
the fact that doing an exclusive-operation, you'll usually want to do something
like close all regions on a table (disabling) and to do that you'd need to be
able to do non-exclusive operations...

Let me start over and make a little grid of what operations we do not want to
run concurrently. Will post here. Might be able to get away w/ something
less disruptive and more basic.

Introduce a zk hosted table-wide read/write lock so only one table operation
at a time
--

Key: HBASE-5494
URL: https://issues.apache.org/jira/browse/HBASE-5494
Project: HBase
Issue Type: Improvement
Reporter: stack

I saw this facility over in the accumulo code base.
Currently we just try to sort out the mess when splits come in during an
online schema edit; somehow we figure we can figure all possible region
transition combinations and make the right call.
We could try and narrow the number of combinations by taking out a zk table
lock when doing table operations.
For example, on split or merge, we could take a read-only lock meaning the
table can't be disabled while these are running.
We could then take a write only lock if we want to ensure the table doesn't
change while disabling or enabling process is happening.
Shouldn't be too hard to add.

[jira] [Updated] (HBASE-5596) Few minor bugs from HBASE-5209


 [ 
https://issues.apache.org/jira/browse/HBASE-5596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David S. Wang updated HBASE-5596:
-

Attachment: (was: HBASE_5596.patch)

 Few minor bugs from HBASE-5209
 --

 Key: HBASE-5596
 URL: https://issues.apache.org/jira/browse/HBASE-5596
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.92.1, 0.94.0
Reporter: David S. Wang
Assignee: David S. Wang
Priority: Minor
 Attachments: HBASE-5596.patch


 A few leftover bugs from HBASE-5209.  Comments are documented here:
 https://reviews.apache.org/r/3892/

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5596) Few minor bugs from HBASE-5209


 [ 
https://issues.apache.org/jira/browse/HBASE-5596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David S. Wang updated HBASE-5596:
-

Status: Patch Available  (was: Open)

Submitted wrong patch.  Sorry.

 Few minor bugs from HBASE-5209
 --

 Key: HBASE-5596
 URL: https://issues.apache.org/jira/browse/HBASE-5596
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.92.1, 0.94.0
Reporter: David S. Wang
Assignee: David S. Wang
Priority: Minor
 Attachments: HBASE-5596.patch


 A few leftover bugs from HBASE-5209.  Comments are documented here:
 https://reviews.apache.org/r/3892/

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5623) Race condition when rolling the HLog and hlogFlush

2012-03-23 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236870#comment-13236870
 ] 

stack commented on HBASE-5623:
--

Yeah, I should make the test fail on these but its kinda hard to put your 
fingers in there.  I was seeing the NPEs running it up in eclipse.  I need to 
amend the test so if fails outright on NPE.

 Race condition when rolling the HLog and hlogFlush
 --

 Key: HBASE-5623
 URL: https://issues.apache.org/jira/browse/HBASE-5623
 Project: HBase
  Issue Type: Bug
  Components: wal
Affects Versions: 0.94.0
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Priority: Critical
 Attachments: 5623v2.txt, HBASE-5623_v0.patch


 When doing a ycsb test with a large number of handlers 
 (regionserver.handler.count=60), I get the following exceptions:
 {code}
 Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: 
 java.lang.NullPointerException
   at 
 org.apache.hadoop.io.SequenceFile$Writer.getLength(SequenceFile.java:1099)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.getLength(SequenceFileLogWriter.java:314)
   at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1291)
   at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1388)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:2192)
   at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1985)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3400)
   at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:366)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1351)
   at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:920)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:152)
   at $Proxy1.multi(Unknown Source)
   at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1691)
   at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1689)
   at 
 org.apache.hadoop.hbase.client.ServerCallable.withoutRetries(ServerCallable.java:214)
 {code}
 and 
 {code}
   java.lang.NullPointerException
   at 
 org.apache.hadoop.io.SequenceFile$Writer.checkAndWriteSync(SequenceFile.java:1026)
   at 
 org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1068)
   at 
 org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1035)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.append(SequenceFileLogWriter.java:279)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.hlogFlush(HLog.java:1237)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1271)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1391)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:2192)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1985)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3400)
   at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:366)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1351)
 {code}
 It seems the root cause of the issue is that we open a new log writer and 
 close the old one at HLog#rollWriter() holding the updateLock, but the other 
 threads doing syncer() calls
 {code} 
 logSyncerThread.hlogFlush(this.writer);
 {code}
 without holding the updateLock. LogSyncer only synchronizes against 
 concurrent appends and flush(), but not on the passed writer, which can be 
 closed already by rollWriter(). In this case, since 
 SequenceFile#Writer.close() sets it's out field as null, we get the NPE. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For

[jira] [Commented] (HBASE-5619) Create PB protocols for HRegionInterface


[ 
https://issues.apache.org/jira/browse/HBASE-5619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236880#comment-13236880
 ] 

jirapos...@reviews.apache.org commented on HBASE-5619:
--



bq.  On 2012-03-23 17:54:17, Ted Yu wrote:
bq.   Looks good overall.
bq.   Minor comments below.

Thanks for reviewing.


bq.  On 2012-03-23 17:54:17, Ted Yu wrote:
bq.   src/main/proto/RegionAdmin.proto, line 115
bq.   https://reviews.apache.org/r/4054/diff/3/?file=94679#file94679line115
bq.  
bq.   Is it possible that clusterIdLeastSignificantBits is present but 
clusterIdMostSignificantBits is absent ?

They both should be either there, or not.  I can add an optional message UUID, 
which should have both listSigBits and mostSigBits. How is that?


bq.  On 2012-03-23 17:54:17, Ted Yu wrote:
bq.   src/main/proto/RegionClient.proto, line 147
bq.   https://reviews.apache.org/r/4054/diff/3/?file=94680#file94680line147
bq.  
bq.   Remove 'is'

Removed


bq.  On 2012-03-23 17:54:17, Ted Yu wrote:
bq.   src/main/proto/RegionClient.proto, line 153
bq.   https://reviews.apache.org/r/4054/diff/3/?file=94680#file94680line153
bq.  
bq.   white space.

Removed.


bq.  On 2012-03-23 17:54:17, Ted Yu wrote:
bq.   src/main/proto/RegionAdmin.proto, line 171
bq.   https://reviews.apache.org/r/4054/diff/3/?file=94679#file94679line171
bq.  
bq.   Should this rpc be called getOnlineRegions ?

For repeated parameter, for example, region, the generated method is something 
like getRegionList().  So it may be better to use singular for parameters.
For similar reason, I did the same for method name.  Another reason for that 
is, in the future, we may use this method to just get one online region given a 
region name.
This way, we can limit the number of methods.


bq.  On 2012-03-23 17:54:17, Ted Yu wrote:
bq.   src/main/proto/RegionClient.proto, line 139
bq.   https://reviews.apache.org/r/4054/diff/3/?file=94680#file94680line139
bq.  
bq.   Increment doesn't extend Mutation:
bq.   
bq.   public class Increment implements Row {
bq.  

That's right. But we can still make increment a mutation, in protocol buffer. 
Since Increment doesn't extend Mutation, it is a little bit different to map an 
Increment to a Mutate.
So we don't have to have a separate method for Increment.


bq.  On 2012-03-23 17:54:17, Ted Yu wrote:
bq.   src/main/proto/RegionClient.proto, line 171
bq.   https://reviews.apache.org/r/4054/diff/3/?file=94680#file94680line171
bq.  
bq.   Should this line be removed ?

Increment is treated as a Mutate, so it is needed.


bq.  On 2012-03-23 17:54:17, Ted Yu wrote:
bq.   src/main/proto/RegionClient.proto, line 208
bq.   https://reviews.apache.org/r/4054/diff/3/?file=94680#file94680line208
bq.  
bq.   'it's appeared' - 'it appears'

Fixed.


bq.  On 2012-03-23 17:54:17, Ted Yu wrote:
bq.   src/main/proto/RegionClient.proto, line 335
bq.   https://reviews.apache.org/r/4054/diff/3/?file=94680#file94680line335
bq.  
bq.   I don't see region member in Exec.java
bq.   Do we need this ?

I see. Let me enhance the doc.  The reason for that is Exec is called for a 
region.  In the request level, there is a default region.  Users can specify 
different region for each individual Exec.


- Jimmy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4054/#review6294
---


On 2012-03-22 20:20:39, Jimmy Xiang wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4054/
bq.  ---
bq.  
bq.  (Updated 2012-03-22 20:20:39)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This is the first draft of the ProtoBuff HRegionProtocol.  The 
corresponding java vs pb method mapping is attached to the jira: 
https://issues.apache.org/jira/browse/HBASE-5443
bq.  
bq.  Please review.  I'd like to move ahead after we get to some agreement.
bq.  
bq.  
bq.  This addresses bug HBASE-5619.
bq.  https://issues.apache.org/jira/browse/HBASE-5619
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.pom.xml 10b13ef 
bq.src/main/proto/RegionAdmin.proto PRE-CREATION 
bq.src/main/proto/RegionClient.proto PRE-CREATION 
bq.src/main/proto/hbase.proto PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/4054/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Jimmy
bq.  
bq.



 Create PB protocols for HRegionInterface
 

 Key: HBASE-5619
 URL: https://issues.apache.org/jira/browse/HBASE-5619
 Project: HBase
  Issue Type:

[jira] [Commented] (HBASE-5623) Race condition when rolling the HLog and hlogFlush

2012-03-23 Thread Enis Soztutar (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236889#comment-13236889
 ] 

Enis Soztutar commented on HBASE-5623:
--

@Stack, thanks for the unit test. I'll test the the unit test with and without 
the patch today. Normally, it is very easy to replicate this bug. I just set up 
a 4 node cluster with Hadoop-1.0.1, and run ycsb with -load -threads 30.  

@Lars
Yes, it is the same with HBASE-5336. I had searched for the exception but did 
not find that issue. Anyway I'll close that one as duplicate. 
bq. If we want maximum concurrency here, we need to do the same we do for 
writer.sync(). I.e. try without lock held, if failing get the lock and retry, 
along with SequenceFileLogWriter.getLength() catching Exception and retrow as 
IOException.
Yes we can do that, but we cannot do this double-checking for the 
logSyncerThread.hlogFlush(this.writer) since it is not idempotent, so in any 
case we should be holding the updateLock, no? From my understanding you are 
suggesting that we sync on the hlogFlush, and do the retry mechanism in 
writer.sync and getLength(), catching all the exceptions in the first try, 
right? 

 Race condition when rolling the HLog and hlogFlush
 --

 Key: HBASE-5623
 URL: https://issues.apache.org/jira/browse/HBASE-5623
 Project: HBase
  Issue Type: Bug
  Components: wal
Affects Versions: 0.94.0
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Priority: Critical
 Attachments: 5623v2.txt, HBASE-5623_v0.patch


 When doing a ycsb test with a large number of handlers 
 (regionserver.handler.count=60), I get the following exceptions:
 {code}
 Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: 
 java.lang.NullPointerException
   at 
 org.apache.hadoop.io.SequenceFile$Writer.getLength(SequenceFile.java:1099)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.getLength(SequenceFileLogWriter.java:314)
   at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1291)
   at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1388)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:2192)
   at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1985)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3400)
   at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:366)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1351)
   at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:920)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:152)
   at $Proxy1.multi(Unknown Source)
   at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1691)
   at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1689)
   at 
 org.apache.hadoop.hbase.client.ServerCallable.withoutRetries(ServerCallable.java:214)
 {code}
 and 
 {code}
   java.lang.NullPointerException
   at 
 org.apache.hadoop.io.SequenceFile$Writer.checkAndWriteSync(SequenceFile.java:1026)
   at 
 org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1068)
   at 
 org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1035)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.append(SequenceFileLogWriter.java:279)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.hlogFlush(HLog.java:1237)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1271)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1391)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:2192)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1985)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3400)
   at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:366)

[jira] [Resolved] (HBASE-5336) Spurious exceptions in HConnectionImplementation

2012-03-23 Thread Enis Soztutar (Resolved) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Enis Soztutar resolved HBASE-5336.
--

Resolution: Duplicate

Duplicate of HBASE-5623

 Spurious exceptions in HConnectionImplementation
 

 Key: HBASE-5336
 URL: https://issues.apache.org/jira/browse/HBASE-5336
 Project: HBase
  Issue Type: Bug
Reporter: Lars Hofhansl

 I have seen this on the client a few time during heave write testing:
 java.util.concurrent.ExecutionException: java.io.IOException: 
 java.io.IOException: java.lang.NullPointerException
   at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
   at java.util.concurrent.FutureTask.get(FutureTask.java:83)
   at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1524)
   at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1376)
   at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:891)
   at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:743)
   at org.apache.hadoop.hbase.client.HTable.put(HTable.java:730)
   at NewsFeedCreate.insert(NewsFeedCreate.java:91)
   at NewsFeedCreate$1.run(NewsFeedCreate.java:38)
   at java.lang.Thread.run(Thread.java:619)
 Caused by: java.io.IOException: java.io.IOException: 
 java.lang.NullPointerException
   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
   at 
 sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
   at 
 sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
   at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
   at 
 org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
   at 
 org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
   at 
 org.apache.hadoop.hbase.client.ServerCallable.translateException(ServerCallable.java:228)
   at 
 org.apache.hadoop.hbase.client.ServerCallable.withoutRetries(ServerCallable.java:212)
   at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1360)
   at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1348)
   at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
   at java.util.concurrent.FutureTask.run(FutureTask.java:138)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   ... 1 more
 Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: 
 java.lang.NullPointerException
   at 
 org.apache.hadoop.io.SequenceFile$Writer.getLength(SequenceFile.java:1099)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.getLength(SequenceFileLogWriter.java:243)
   at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1289)
   at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1386)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:2161)
   at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1954)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3363)
   at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
   at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:899)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
   at $Proxy1.multi(Unknown Source)
   at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1353)
   at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1351)
   at 
 org.apache.hadoop.hbase.client.ServerCallable.withoutRetries(ServerCallable.java:210)
   ... 7 more

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see:

[jira] [Commented] (HBASE-5619) Create PB protocols for HRegionInterface

2012-03-23 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236894#comment-13236894
 ] 

jirapos...@reviews.apache.org commented on HBASE-5619:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4054/#review6299
---



src/main/proto/RegionAdmin.proto
https://reviews.apache.org/r/4054/#comment13666

Introducing message UUID would be better.


- Ted


On 2012-03-22 20:20:39, Jimmy Xiang wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4054/
bq.  ---
bq.  
bq.  (Updated 2012-03-22 20:20:39)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This is the first draft of the ProtoBuff HRegionProtocol.  The 
corresponding java vs pb method mapping is attached to the jira: 
https://issues.apache.org/jira/browse/HBASE-5443
bq.  
bq.  Please review.  I'd like to move ahead after we get to some agreement.
bq.  
bq.  
bq.  This addresses bug HBASE-5619.
bq.  https://issues.apache.org/jira/browse/HBASE-5619
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.pom.xml 10b13ef 
bq.src/main/proto/RegionAdmin.proto PRE-CREATION 
bq.src/main/proto/RegionClient.proto PRE-CREATION 
bq.src/main/proto/hbase.proto PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/4054/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Jimmy
bq.  
bq.



 Create PB protocols for HRegionInterface
 

 Key: HBASE-5619
 URL: https://issues.apache.org/jira/browse/HBASE-5619
 Project: HBase
  Issue Type: Sub-task
  Components: ipc, master, migration, regionserver
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0

 Attachments: hbase-5619.patch


 Subtask of HBase-5443, separate HRegionInterface into admin protocol and 
 client protocol, create the PB protocol buffer files

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5623) Race condition when rolling the HLog and hlogFlush

2012-03-23 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236907#comment-13236907
 ] 

Lars Hofhansl commented on HBASE-5623:
--

@Enis: For logSyncerThread.hlogFlush(this.writer) we need to do that in that 
method, LogSyncer is non-static, so we can use the HLog's updateLock in its 
methods.

 Race condition when rolling the HLog and hlogFlush
 --

 Key: HBASE-5623
 URL: https://issues.apache.org/jira/browse/HBASE-5623
 Project: HBase
  Issue Type: Bug
  Components: wal
Affects Versions: 0.94.0
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Priority: Critical
 Attachments: 5623v2.txt, HBASE-5623_v0.patch


 When doing a ycsb test with a large number of handlers 
 (regionserver.handler.count=60), I get the following exceptions:
 {code}
 Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: 
 java.lang.NullPointerException
   at 
 org.apache.hadoop.io.SequenceFile$Writer.getLength(SequenceFile.java:1099)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.getLength(SequenceFileLogWriter.java:314)
   at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1291)
   at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1388)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:2192)
   at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1985)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3400)
   at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:366)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1351)
   at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:920)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:152)
   at $Proxy1.multi(Unknown Source)
   at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1691)
   at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1689)
   at 
 org.apache.hadoop.hbase.client.ServerCallable.withoutRetries(ServerCallable.java:214)
 {code}
 and 
 {code}
   java.lang.NullPointerException
   at 
 org.apache.hadoop.io.SequenceFile$Writer.checkAndWriteSync(SequenceFile.java:1026)
   at 
 org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1068)
   at 
 org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1035)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.append(SequenceFileLogWriter.java:279)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.hlogFlush(HLog.java:1237)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1271)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1391)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:2192)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1985)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3400)
   at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:366)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1351)
 {code}
 It seems the root cause of the issue is that we open a new log writer and 
 close the old one at HLog#rollWriter() holding the updateLock, but the other 
 threads doing syncer() calls
 {code} 
 logSyncerThread.hlogFlush(this.writer);
 {code}
 without holding the updateLock. LogSyncer only synchronizes against 
 concurrent appends and flush(), but not on the passed writer, which can be 
 closed already by rollWriter(). In this case, since 
 SequenceFile#Writer.close() sets it's out field as null, we get the NPE. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on

[jira] [Commented] (HBASE-5619) Create PB protocols for HRegionInterface


[ 
https://issues.apache.org/jira/browse/HBASE-5619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236913#comment-13236913
 ] 

jirapos...@reviews.apache.org commented on HBASE-5619:
--



bq.  On 2012-03-23 18:30:08, Ted Yu wrote:
bq.   src/main/proto/RegionAdmin.proto, line 115
bq.   https://reviews.apache.org/r/4054/diff/3/?file=94679#file94679line115
bq.  
bq.   Introducing message UUID would be better.

Fixed.


- Jimmy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4054/#review6299
---


On 2012-03-22 20:20:39, Jimmy Xiang wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4054/
bq.  ---
bq.  
bq.  (Updated 2012-03-22 20:20:39)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This is the first draft of the ProtoBuff HRegionProtocol.  The 
corresponding java vs pb method mapping is attached to the jira: 
https://issues.apache.org/jira/browse/HBASE-5443
bq.  
bq.  Please review.  I'd like to move ahead after we get to some agreement.
bq.  
bq.  
bq.  This addresses bug HBASE-5619.
bq.  https://issues.apache.org/jira/browse/HBASE-5619
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.pom.xml 10b13ef 
bq.src/main/proto/RegionAdmin.proto PRE-CREATION 
bq.src/main/proto/RegionClient.proto PRE-CREATION 
bq.src/main/proto/hbase.proto PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/4054/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Jimmy
bq.  
bq.



 Create PB protocols for HRegionInterface
 

 Key: HBASE-5619
 URL: https://issues.apache.org/jira/browse/HBASE-5619
 Project: HBase
  Issue Type: Sub-task
  Components: ipc, master, migration, regionserver
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0

 Attachments: hbase-5619.patch


 Subtask of HBase-5443, separate HRegionInterface into admin protocol and 
 client protocol, create the PB protocol buffer files

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-3776) Add Bloom Filter Support to HFileOutputFormat


 [ 
https://issues.apache.org/jira/browse/HBASE-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-3776:
-

Fix Version/s: (was: 0.94.0)
   0.96.0

I don't buy that this is a 0.94 blocker. Pull back if you disagree.

 Add Bloom Filter Support to HFileOutputFormat
 -

 Key: HBASE-3776
 URL: https://issues.apache.org/jira/browse/HBASE-3776
 Project: HBase
  Issue Type: Sub-task
Reporter: Nicolas Spiegelberg
Assignee: Jerry Chen
Priority: Blocker
  Labels: hbase
 Fix For: 0.96.0


 Add Bloom Filter support for bulk imports.  Lacking a bloom filter, even on a 
 single imported file, can cause perf degradation.  Since we now set our 
 compression type based on the HBase CF configuration, it would be good to 
 follow this path for the bloom filter addition.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5613) ThriftServer getTableRegions does not return serverName and port

[
https://issues.apache.org/jira/browse/HBASE-5613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236916#comment-13236916
]

Hadoop QA commented on HBASE-5613:
--

-1 overall. Here are the results of testing the latest attachment

http://issues.apache.org/jira/secure/attachment/12519672/HBASE-5613.D2403.5.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.TestReplication

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1280//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1280//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1280//console

This message is automatically generated.

ThriftServer getTableRegions does not return serverName and port

Key: HBASE-5613
URL: https://issues.apache.org/jira/browse/HBASE-5613
Project: HBase
Issue Type: Bug
Components: thrift
Reporter: Scott Chen
Assignee: Scott Chen
Priority: Minor
Fix For: 0.94.0, 0.96.0

Attachments: HBASE-5613.0.94.txt, HBASE-5613.D2403.1.patch,
HBASE-5613.D2403.2.patch, HBASE-5613.D2403.3.patch, HBASE-5613.D2403.4.patch,
HBASE-5613.D2403.5.patch

[jira] [Updated] (HBASE-4914) Enhance MapReduce TableInputFormat to Support N-mappers per Region


 [ 
https://issues.apache.org/jira/browse/HBASE-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-4914:
-

Fix Version/s: (was: 0.94.0)
   0.96.0

Moving out of 0.94. Pull back if you disagree.

 Enhance MapReduce TableInputFormat to Support N-mappers per Region
 --

 Key: HBASE-4914
 URL: https://issues.apache.org/jira/browse/HBASE-4914
 Project: HBase
  Issue Type: Sub-task
  Components: client, regionserver
Reporter: Nicolas Spiegelberg
Priority: Blocker
 Fix For: 0.96.0


 Current TableInputFormat based MR jobs create exactly one mapper per region 
 where each mapper sets one Scan with appropriate start/stop row keys. This 
 change allows jobs to be run with any number of mappers per region, so that 
 when a mapper fails, there will be less data to be reprocessed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-3967) Support deletes in HFileOutputFormat based bulk import mechanism


 [ 
https://issues.apache.org/jira/browse/HBASE-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-3967:
-

Fix Version/s: (was: 0.94.0)
   0.96.0

Moving out of 0.94. Pull back if you disagree.

 Support deletes in HFileOutputFormat based bulk import mechanism
 

 Key: HBASE-3967
 URL: https://issues.apache.org/jira/browse/HBASE-3967
 Project: HBase
  Issue Type: Sub-task
Reporter: Kannan Muthukkaruppan
Priority: Blocker
 Fix For: 0.96.0

 Attachments: diff.patch


 During bulk imports, it'll be useful to be able to do delete mutations 
 (either to delete data that already exists in HBase or was inserted earlier 
 during this run of the import). 
 For example, we have a use case, where we are processing a log of data which 
 may have both inserts and deletes in the mix and we want to upload that into 
 HBase using the bulk import mechanism.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5616) Make compaction code standalone

[
https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236921#comment-13236921
]

Hadoop QA commented on HBASE-5616:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12519671/5616v7.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 5 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.TestReplication
org.apache.hadoop.hbase.mapreduce.TestImportTsv
org.apache.hadoop.hbase.mapred.TestTableMapReduce
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
org.apache.hadoop.hbase.master.TestSplitLogManager

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1279//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1279//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1279//console

This message is automatically generated.

Make compaction code standalone
---

This is part of hbase-2462. Make the compaction code standalone so can run
it independent of hbase. Will make it easier to profile and try stuff out.

[jira] [Updated] (HBASE-5623) Race condition when rolling the HLog and hlogFlush


 [ 
https://issues.apache.org/jira/browse/HBASE-5623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-5623:
-

Fix Version/s: 0.94.0

I think this needs to be in 0.94.0.

 Race condition when rolling the HLog and hlogFlush
 --

 Key: HBASE-5623
 URL: https://issues.apache.org/jira/browse/HBASE-5623
 Project: HBase
  Issue Type: Bug
  Components: wal
Affects Versions: 0.94.0
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Priority: Critical
 Fix For: 0.94.0

 Attachments: 5623v2.txt, HBASE-5623_v0.patch


 When doing a ycsb test with a large number of handlers 
 (regionserver.handler.count=60), I get the following exceptions:
 {code}
 Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: 
 java.lang.NullPointerException
   at 
 org.apache.hadoop.io.SequenceFile$Writer.getLength(SequenceFile.java:1099)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.getLength(SequenceFileLogWriter.java:314)
   at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1291)
   at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1388)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:2192)
   at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1985)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3400)
   at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:366)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1351)
   at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:920)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:152)
   at $Proxy1.multi(Unknown Source)
   at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1691)
   at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1689)
   at 
 org.apache.hadoop.hbase.client.ServerCallable.withoutRetries(ServerCallable.java:214)
 {code}
 and 
 {code}
   java.lang.NullPointerException
   at 
 org.apache.hadoop.io.SequenceFile$Writer.checkAndWriteSync(SequenceFile.java:1026)
   at 
 org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1068)
   at 
 org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1035)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.append(SequenceFileLogWriter.java:279)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.hlogFlush(HLog.java:1237)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1271)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1391)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:2192)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1985)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3400)
   at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:366)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1351)
 {code}
 It seems the root cause of the issue is that we open a new log writer and 
 close the old one at HLog#rollWriter() holding the updateLock, but the other 
 threads doing syncer() calls
 {code} 
 logSyncerThread.hlogFlush(this.writer);
 {code}
 without holding the updateLock. LogSyncer only synchronizes against 
 concurrent appends and flush(), but not on the passed writer, which can be 
 closed already by rollWriter(). In this case, since 
 SequenceFile#Writer.close() sets it's out field as null, we get the NPE. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-3852) ThriftServer leaks scanners


 [ 
https://issues.apache.org/jira/browse/HBASE-3852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-3852:
-

Fix Version/s: (was: 0.94.0)
   0.94.1

Let's try for 0.94.1

 ThriftServer leaks scanners
 ---

 Key: HBASE-3852
 URL: https://issues.apache.org/jira/browse/HBASE-3852
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.2
Reporter: Jean-Daniel Cryans
Priority: Critical
 Fix For: 0.94.1

 Attachments: 3852.txt, thrift2-scanner.patch


 The scannerMap in ThriftServer relies on the user to clean it by closing the 
 scanner. If that doesn't happen, the ResultScanner will stay in the thrift 
 server's memory and if any pre-fetching was done, it will also start 
 accumulating Results (with all their data).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5616) Make compaction code standalone


 [ 
https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-5616:
-

Status: Open  (was: Patch Available)

 Make compaction code standalone
 ---

 Key: HBASE-5616
 URL: https://issues.apache.org/jira/browse/HBASE-5616
 Project: HBase
  Issue Type: Improvement
Reporter: stack
Assignee: stack
 Attachments: 5616.txt, 5616v3.txt, 5616v6.txt, 5616v7.txt, 
 5616v7.txt, standalone.txt


 This is part of hbase-2462.  Make the compaction code standalone so can run 
 it independent of hbase.  Will make it easier to profile and try stuff out.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5596) Few minor bugs from HBASE-5209

[
https://issues.apache.org/jira/browse/HBASE-5596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236938#comment-13236938
]

Hadoop QA commented on HBASE-5596:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12519676/HBASE-5596.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.TestReplication

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1281//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1281//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1281//console

This message is automatically generated.

Few minor bugs from HBASE-5209
--

A few leftover bugs from HBASE-5209. Comments are documented here:
https://reviews.apache.org/r/3892/

[jira] [Updated] (HBASE-5616) Make compaction code standalone


 [ 
https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-5616:
-

Attachment: 5616v7.txt

Retry.  TestSplitLogManager passes locally.

 Make compaction code standalone
 ---

 Key: HBASE-5616
 URL: https://issues.apache.org/jira/browse/HBASE-5616
 Project: HBase
  Issue Type: Improvement
Reporter: stack
Assignee: stack
 Attachments: 5616.txt, 5616v3.txt, 5616v6.txt, 5616v7.txt, 
 5616v7.txt, standalone.txt


 This is part of hbase-2462.  Make the compaction code standalone so can run 
 it independent of hbase.  Will make it easier to profile and try stuff out.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5451) Switch RPC call envelope/headers to PBs

2012-03-23 Thread Devaraj Das (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236933#comment-13236933
 ] 

Devaraj Das commented on HBASE-5451:


Could someone please look at this. Thanks!

 Switch RPC call envelope/headers to PBs
 ---

 Key: HBASE-5451
 URL: https://issues.apache.org/jira/browse/HBASE-5451
 Project: HBase
  Issue Type: Sub-task
  Components: ipc, master, migration, regionserver
Affects Versions: 0.94.0
Reporter: Todd Lipcon
Assignee: Devaraj Das
 Fix For: 0.96.0

 Attachments: rpc-proto.2.txt, rpc-proto.3.txt, rpc-proto.patch.1_2




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5616) Make compaction code standalone


 [ 
https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-5616:
-

Status: Patch Available  (was: Open)

 Make compaction code standalone
 ---

 Key: HBASE-5616
 URL: https://issues.apache.org/jira/browse/HBASE-5616
 Project: HBase
  Issue Type: Improvement
Reporter: stack
Assignee: stack
 Attachments: 5616.txt, 5616v3.txt, 5616v6.txt, 5616v7.txt, 
 5616v7.txt, standalone.txt


 This is part of hbase-2462.  Make the compaction code standalone so can run 
 it independent of hbase.  Will make it easier to profile and try stuff out.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5494) Introduce a zk hosted table-wide read/write lock so only one table operation at a time

[
https://issues.apache.org/jira/browse/HBASE-5494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236943#comment-13236943
]

stack commented on HBASE-5494:
--

Hmm... NVM. First I need to figure out complete set of zk states and
operations they trigger. Making a start up here:
https://docs.google.com/spreadsheet/ccc?key=0Aoa3E58mCyOfdFFCY3RmMW9ZeEliTndLNGRNWVBCZkE
Don't look yet. It has nothing in it yet.

Introduce a zk hosted table-wide read/write lock so only one table operation
at a time
--

Key: HBASE-5494
URL: https://issues.apache.org/jira/browse/HBASE-5494
Project: HBase
Issue Type: Improvement
Reporter: stack

[jira] [Commented] (HBASE-5451) Switch RPC call envelope/headers to PBs

2012-03-23 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236955#comment-13236955
 ] 

stack commented on HBASE-5451:
--

Sorry DD.  Trying to get Benoit to review it before committing.  I've given him 
his requisite $3.00 but I might have to give him more to bump me up in his 
queue.

 Switch RPC call envelope/headers to PBs
 ---

 Key: HBASE-5451
 URL: https://issues.apache.org/jira/browse/HBASE-5451
 Project: HBase
  Issue Type: Sub-task
  Components: ipc, master, migration, regionserver
Affects Versions: 0.94.0
Reporter: Todd Lipcon
Assignee: Devaraj Das
 Fix For: 0.96.0

 Attachments: rpc-proto.2.txt, rpc-proto.3.txt, rpc-proto.patch.1_2




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5444) Add PB-based calls to HMasterRegionInterface


[ 
https://issues.apache.org/jira/browse/HBASE-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236971#comment-13236971
 ] 

jirapos...@reviews.apache.org commented on HBASE-5444:
--



bq.  On 2012-03-23 17:01:13, Jimmy Xiang wrote:
bq.   src/main/proto/hbase.proto, line 33
bq.   https://reviews.apache.org/r/4463/diff/1/?file=94842#file94842line33
bq.  
bq.   Do we have column family information?

I don't know what you mean by this question.

I don't actually need this for HBASE-5444; I'll remove it and bring it back in 
HBASE-5445.


bq.  On 2012-03-23 17:01:13, Jimmy Xiang wrote:
bq.   pom.xml, line 768
bq.   https://reviews.apache.org/r/4463/diff/1/?file=94822#file94822line768
bq.  
bq.   I have some enhancement to the pom.  Please check my patch on
bq.   
bq.   https://issues.apache.org/jira/browse/HBASE-5619

Will do.


bq.  On 2012-03-23 17:01:13, Jimmy Xiang wrote:
bq.   
src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon, line 
40
bq.   https://reviews.apache.org/r/4463/diff/1/?file=94823#file94823line40
bq.  
bq.   I called it HBaseProtos instead of CommonProtocol since it is not a 
protocol itself.

Sounds good, will do.


bq.  On 2012-03-23 17:01:13, Jimmy Xiang wrote:
bq.   
src/main/java/org/apache/hadoop/hbase/protobuf/MasterRegionInterface.java, line 
34
bq.   https://reviews.apache.org/r/4463/diff/1/?file=94838#file94838line34
bq.  
bq.   Should it be Private?  Should we call it MasterRegionProtocol?

I'll make it private.

I'll leave it named Interface for now, just because it is replacing 
HMasterRegionInterface.  I agree we should be consistent.  Perhaps someone 
else can chime in with an opinion on Interface vs Protocol.


bq.  On 2012-03-23 17:01:13, Jimmy Xiang wrote:
bq.   src/main/proto/MasterRegionProtocol.proto, line 40
bq.   https://reviews.apache.org/r/4463/diff/1/?file=94841#file94841line40
bq.  
bq.   This looks like a property.  I have similar message for 
RegionProtocols.  Can we put this in hbase.proto and call it Property so that 
we can share it?

Property is kind of a non-descriptive name.  How about NameValuePair?


- Gregory


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4463/#review6286
---


On 2012-03-23 04:58:51, Gregory Chanan wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4463/
bq.  ---
bq.  
bq.  (Updated 2012-03-23 04:58:51)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Adds PB-based calls replacing HMasterRegionInterface.
bq.  
bq.  There are some temporary hacks, e.g. converting PB-based ServerLoad to 
existing HServerLoad so I didn't need to convert ClusterStatus (which brings in 
a lot of other changes).  That will be cleaned up in HBASE-5445.
bq.  
bq.  
bq.  This addresses bug HBASE-5444.
bq.  https://issues.apache.org/jira/browse/HBASE-5444
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.pom.xml 10b13ef 
bq.
src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon 
69434f7 
bq.
src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon 
ae76204 
bq.src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 9df4c10 
bq.src/main/java/org/apache/hadoop/hbase/HConstants.java 347 
bq.src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java 
cbfa489 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 0db2760 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 2916d68 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 
fd97830 
bq.src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java f1f06b0 
bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 
d47ef10 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java cd1755f 
bq.src/main/java/org/apache/hadoop/hbase/master/MXBean.java 7f44dc2 
bq.src/main/java/org/apache/hadoop/hbase/master/MXBeanImpl.java 45b8fe7 
bq.src/main/java/org/apache/hadoop/hbase/master/MasterDumpServlet.java 
be63838 
bq.src/main/java/org/apache/hadoop/hbase/master/ServerManager.java cbd55f7 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/MasterRegionInterface.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/PBHelper.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
e0af8fb 
bq.src/main/proto/MasterRegionProtocol.proto PRE-CREATION 
bq.src/main/proto/hbase.proto PRE-CREATION 
bq.src/main/resources/hbase-webapps/master/table.jsp 811df46

[jira] [Commented] (HBASE-5596) Few minor bugs from HBASE-5209

2012-03-23 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236972#comment-13236972
 ] 

stack commented on HBASE-5596:
--

OK +1.  Maybe you want to looksee Jon?

 Few minor bugs from HBASE-5209
 --

 Key: HBASE-5596
 URL: https://issues.apache.org/jira/browse/HBASE-5596
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.92.1, 0.94.0
Reporter: David S. Wang
Assignee: David S. Wang
Priority: Minor
 Attachments: HBASE-5596.patch


 A few leftover bugs from HBASE-5209.  Comments are documented here:
 https://reviews.apache.org/r/3892/

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5619) Create PB protocols for HRegionInterface


[ 
https://issues.apache.org/jira/browse/HBASE-5619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236975#comment-13236975
 ] 

jirapos...@reviews.apache.org commented on HBASE-5619:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4054/
---

(Updated 2012-03-23 19:29:52.180425)


Review request for hbase.


Changes
---

Addressed Ted's comments.


Summary
---

This is the first draft of the ProtoBuff HRegionProtocol.  The corresponding 
java vs pb method mapping is attached to the jira: 
https://issues.apache.org/jira/browse/HBASE-5443

Please review.  I'd like to move ahead after we get to some agreement.


This addresses bug HBASE-5619.
https://issues.apache.org/jira/browse/HBASE-5619


Diffs (updated)
-

  pom.xml 10b13ef 
  src/main/proto/RegionAdmin.proto PRE-CREATION 
  src/main/proto/RegionClient.proto PRE-CREATION 
  src/main/proto/hbase.proto PRE-CREATION 

Diff: https://reviews.apache.org/r/4054/diff


Testing
---


Thanks,

Jimmy



 Create PB protocols for HRegionInterface
 

 Key: HBASE-5619
 URL: https://issues.apache.org/jira/browse/HBASE-5619
 Project: HBase
  Issue Type: Sub-task
  Components: ipc, master, migration, regionserver
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0

 Attachments: hbase-5619.patch


 Subtask of HBase-5443, separate HRegionInterface into admin protocol and 
 client protocol, create the PB protocol buffer files

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5616) Make compaction code standalone

2012-03-23 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236989#comment-13236989
]

Hadoop QA commented on HBASE-5616:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12519692/5616v7.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 5 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1282//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1282//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1282//console

This message is automatically generated.

Make compaction code standalone
---

This is part of hbase-2462. Make the compaction code standalone so can run
it independent of hbase. Will make it easier to profile and try stuff out.

[jira] [Commented] (HBASE-5128) [uber hbck] Enable hbck to automatically repair table integrity problems as well as region consistency problems while online.


[ 
https://issues.apache.org/jira/browse/HBASE-5128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236997#comment-13236997
 ] 

jirapos...@reviews.apache.org commented on HBASE-5128:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3435/#review6304
---

Ship it!


Went through a third.  Minors below that should not hold up commit.   Get it 
in!!!  Great stuff Jon.


src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/3435/#comment13682

Good doc (though I've said this previous, its still good doc)



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/3435/#comment13683

Why TreeMap it if its encoded region names?  These are hashes so no value 
sorting them?



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/3435/#comment13684

Ditto on sort here?  Why sort by table name?  How does sort prevent dupes?



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/3435/#comment13685

This almost recommends that HBaseFsck becomes a shell that does nothing but 
instantiate another class that does acual fixup.  clearState in that case would 
throw away the instantiated 'Fsck' class and create a completely new instance 
rather than zero out data members as this does.  For the future.


- Michael


On 2012-03-23 16:13:50, jmhsieh wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3435/
bq.  ---
bq.  
bq.  (Updated 2012-03-23 16:13:50)
bq.  
bq.  
bq.  Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, and 
Jean-Daniel Cryans.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This should nearly be to ready for integration.  This has the same control 
flow as the trunk/0.92/0.94 versions but has a few differences.  
bq.  
bq.  - It needs to track HTableDescritors instead of reading them from the file 
system.
bq.  - It uses a different HBaseFsckRepair.forceOfflineInZK method -- which for 
some reason means we don't need HBASE-5563.
bq.  - Uses HServerAddress instead of ServerName
bq.  
bq.  This version is close to what we've used on production clusters.
bq.  
bq.  
bq.  This addresses bug HBASE-5128.
bq.  https://issues.apache.org/jira/browse/HBASE-5128
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 1a4f7f1 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 3c635d4 
bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 
d47ef10 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java cd1755f 
bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java c0aaf65 
bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java 5916d9c 
bq.src/main/java/org/apache/hadoop/hbase/util/hbck/OfflineMetaRepair.java 
d57bb6b 
bq.
src/main/java/org/apache/hadoop/hbase/util/hbck/TableIntegrityErrorHandler.java 
PRE-CREATION 
bq.
src/main/java/org/apache/hadoop/hbase/util/hbck/TableIntegrityErrorHandlerImpl.java
 PRE-CREATION 
bq.src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java d9a2a02 
bq.src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java 937781d 
bq.src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsckComparator.java 
0599da1 
bq.src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java 
dbb97f8 
bq.
src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildBase.java 
2b4cac8 
bq.
src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildHole.java 
ebbeead 
bq.
src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildOverlap.java
 b175548 
bq.  
bq.  Diff: https://reviews.apache.org/r/3435/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  All TestHBaseFsck unit tests pass.  Currently running full suite.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  jmhsieh
bq.  
bq.



 [uber hbck] Enable hbck to automatically repair table integrity problems as 
 well as region consistency problems while online.
 -

 Key: HBASE-5128
 URL: https://issues.apache.org/jira/browse/HBASE-5128
 Project: HBase
  Issue Type: New Feature
  Components: hbck
Affects Versions: 0.90.5, 0.92.0, 0.94.0, 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0

[jira] [Commented] (HBASE-5444) Add PB-based calls to HMasterRegionInterface

2012-03-23 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236999#comment-13236999
 ] 

jirapos...@reviews.apache.org commented on HBASE-5444:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4463/
---

(Updated 2012-03-23 19:55:28.089163)


Review request for hbase.


Changes
---

Update for Jimmy's review.


Summary
---

Adds PB-based calls replacing HMasterRegionInterface.

There are some temporary hacks, e.g. converting PB-based ServerLoad to existing 
HServerLoad so I didn't need to convert ClusterStatus (which brings in a lot of 
other changes).  That will be cleaned up in HBASE-5445.


This addresses bug HBASE-5444.
https://issues.apache.org/jira/browse/HBASE-5444


Diffs (updated)
-

  pom.xml 10b13ef 
  src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon 
69434f7 
  src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon 
ae76204 
  src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 9df4c10 
  src/main/java/org/apache/hadoop/hbase/HConstants.java 347 
  src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java cbfa489 
  src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 0db2760 
  src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 2916d68 
  src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java fd97830 
  src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java f1f06b0 
  src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java d47ef10 
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java cd1755f 
  src/main/java/org/apache/hadoop/hbase/master/MXBean.java 7f44dc2 
  src/main/java/org/apache/hadoop/hbase/master/MXBeanImpl.java 45b8fe7 
  src/main/java/org/apache/hadoop/hbase/master/MasterDumpServlet.java be63838 
  src/main/java/org/apache/hadoop/hbase/master/ServerManager.java cbd55f7 
  src/main/java/org/apache/hadoop/hbase/protobuf/MasterRegionInterface.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/protobuf/PBHelper.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e0af8fb 
  src/main/proto/MasterRegionProtocol.proto PRE-CREATION 
  src/main/proto/hbase.proto PRE-CREATION 
  src/main/resources/hbase-webapps/master/table.jsp 811df46 
  src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 6af9188 
  src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java 
368a0e5 
  src/test/java/org/apache/hadoop/hbase/io/TestHbaseObjectWritable.java f2f8ee3 
  src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 
841649a 
  src/test/java/org/apache/hadoop/hbase/master/TestMXBean.java 379f70c 
  
src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java
 e99d251 

Diff: https://reviews.apache.org/r/4463/diff


Testing
---

Ran jenkins job, all unit tests passed.


Thanks,

Gregory



 Add PB-based calls to HMasterRegionInterface
 

 Key: HBASE-5444
 URL: https://issues.apache.org/jira/browse/HBASE-5444
 Project: HBase
  Issue Type: Sub-task
  Components: ipc, master, migration, regionserver
Reporter: Todd Lipcon
Assignee: Gregory Chanan



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5573) Replace client ZooKeeper watchers by simple ZooKeeper reads


[ 
https://issues.apache.org/jira/browse/HBASE-5573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237006#comment-13237006
 ] 

stack commented on HBASE-5573:
--

Ok on your your just redoing what hbasefsck was doing anyways.

Regards the pseudo-code you drew out for me where you show how RZKW relates to 
ZKW relates to ZK, it makes my head hurt.  If the 'Watcher' implementation was 
broken out into a standalone class that might help some but my guess is that 
that'd be big mess to untangle (...exploded in my hands).

bq. So there's no proof in the code, just that actually all the functions we 
use on the client don't use a watcher.

Excellent

bq. one of the design issue is that there ate two API: you can use directly any 
of the ZKW, RZK, RK object or you can go through the static ZKUtils. May be the 
intermediate solutions lie around this area.

OK.  Any recommendation you can make here having been down deep in this code?  
We should make everyone go via ZKUtils and via ZKAssign, etc., and clean up any 
other errant use of zkw directly?  Would that be good to do?

bq. But when you have a fat client architecture like we have, it makes sense to 
share some global state information, and it scales better when the info is 
pushed vs. pulled.

I'd like to get to the case where we have not watchers -- which this patch is 
finishing -- and then have the above discussion subsequently.  I'd think its 
more scalable if clients do not keep open sessions and keep watchers.  But we 
can talk about that some other time after we've let go at least of watchers.

Let me look at your last patch.

 Replace client ZooKeeper watchers by simple ZooKeeper reads
 ---

 Key: HBASE-5573
 URL: https://issues.apache.org/jira/browse/HBASE-5573
 Project: HBase
  Issue Type: Improvement
  Components: client, zookeeper
Affects Versions: 0.96.0
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
 Attachments: 5573.v1.patch, 5573.v2.patch, 5573.v4.patch, 
 5573.v6.patch


 Some code in the package needs to read data in ZK. This could be done by a 
 simple read, but is actually implemented with a watcher. This holds ZK 
 resources.
 Fixing this could also be an opportunity to remove the need for the client to 
 provide the master address and port.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5619) Create PB protocols for HRegionInterface


[ 
https://issues.apache.org/jira/browse/HBASE-5619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237034#comment-13237034
 ] 

Zhihong Yu commented on HBASE-5619:
---

@Stack, @Benoit:
Please take a look.

 Create PB protocols for HRegionInterface
 

 Key: HBASE-5619
 URL: https://issues.apache.org/jira/browse/HBASE-5619
 Project: HBase
  Issue Type: Sub-task
  Components: ipc, master, migration, regionserver
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0

 Attachments: hbase-5619.patch


 Subtask of HBase-5443, separate HRegionInterface into admin protocol and 
 client protocol, create the PB protocol buffer files

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5573) Replace client ZooKeeper watchers by simple ZooKeeper reads

[
https://issues.apache.org/jira/browse/HBASE-5573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237089#comment-13237089
]

stack commented on HBASE-5573:
--

Patch looks good. Only question is the one I had yesterday where in
HBaseTestingUtility#getZooKeeperWatcher, if its aborted, it does nothing but
this ZKW is being used by test code so I'd think if an abort, it shouldn't be
suppressed -- rather we should complain loudly? Rethrow as RuntimeException?

Do you want to be consistent? You call methods getZKW most times and then
getZooKeeperWatcher in this test code (I prefer the latter).

Replace client ZooKeeper watchers by simple ZooKeeper reads
---

[jira] [Updated] (HBASE-5613) ThriftServer getTableRegions does not return serverName and port

2012-03-23 Thread Zhihong Yu (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Yu updated HBASE-5613:
--

Comment: was deleted

(was: -1 overall.  Here are the results of testing the latest attachment 
  
http://issues.apache.org/jira/secure/attachment/12519564/HBASE-5613.D2403.4.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 3 new or modified tests.

-1 patch.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1270//console

This message is automatically generated.)

 ThriftServer getTableRegions does not return serverName and port
 

 Key: HBASE-5613
 URL: https://issues.apache.org/jira/browse/HBASE-5613
 Project: HBase
  Issue Type: Bug
  Components: thrift
Reporter: Scott Chen
Assignee: Scott Chen
Priority: Minor
 Fix For: 0.94.0, 0.96.0

 Attachments: HBASE-5613.0.94.txt, HBASE-5613.D2403.1.patch, 
 HBASE-5613.D2403.2.patch, HBASE-5613.D2403.3.patch, HBASE-5613.D2403.4.patch, 
 HBASE-5613.D2403.5.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5613) ThriftServer getTableRegions does not return serverName and port


[ 
https://issues.apache.org/jira/browse/HBASE-5613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237105#comment-13237105
 ] 

Zhihong Yu commented on HBASE-5613:
---

TestReplication is not related to thrift.

Will integrate if there is no objection.

 ThriftServer getTableRegions does not return serverName and port
 

 Key: HBASE-5613
 URL: https://issues.apache.org/jira/browse/HBASE-5613
 Project: HBase
  Issue Type: Bug
  Components: thrift
Reporter: Scott Chen
Assignee: Scott Chen
Priority: Minor
 Fix For: 0.94.0, 0.96.0

 Attachments: HBASE-5613.0.94.txt, HBASE-5613.D2403.1.patch, 
 HBASE-5613.D2403.2.patch, HBASE-5613.D2403.3.patch, HBASE-5613.D2403.4.patch, 
 HBASE-5613.D2403.5.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size


[ 
https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237124#comment-13237124
 ] 

Lars Hofhansl commented on HBASE-5190:
--

Wanna commit, J-D?

 Limit the IPC queue size based on calls' payload size
 -

 Key: HBASE-5190
 URL: https://issues.apache.org/jira/browse/HBASE-5190
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.90.5
Reporter: Jean-Daniel Cryans
Assignee: Jean-Daniel Cryans
 Fix For: 0.94.0

 Attachments: HBASE-5190-v2.patch, HBASE-5190-v3.patch, 
 HBASE-5190.patch


 Currently we limit the number of calls in the IPC queue only on their count. 
 It used to be really high and was dropped down recently to num_handlers * 10 
 (so 100 by default) because it was easy to OOME yourself when huge calls were 
 being queued. It's still possible to hit this problem if you use really big 
 values and/or a lot of handlers, so the idea is that we should take into 
 account the payload size. I can see 3 solutions:
  - Do the accounting outside of the queue itself for all calls coming in and 
 out and when a call doesn't fit, throw a retryable exception.
  - Same accounting but instead block the call when it comes in until space is 
 made available.
  - Add a new parameter for the maximum size (in bytes) of a Call and then set 
 the size the IPC queue (in terms of the number of items) so that it could 
 only contain as many items as some predefined maximum size (in bytes) for the 
 whole queue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5608) MR testcases are failing in QA builds

2012-03-23 Thread Jonathan Hsieh (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237133#comment-13237133
 ] 

Jonathan Hsieh commented on HBASE-5608:
---

@Greg

I cannot access the physical machines to get the actual logs, but we should be 
able to configure jenkins to save particular files after a build.

Looks like trunk saves all build and test artifacts from the most recent run 
here 
https://builds.apache.org/view/G-L/view/HBase/job/PreCommit-HBASE-Build/ws/.  
Can you take a look and identify the file you need to help debug?

We can enable it for a little while until we get it resolved.

 MR testcases are failing in QA builds
 -

 Key: HBASE-5608
 URL: https://issues.apache.org/jira/browse/HBASE-5608
 Project: HBase
  Issue Type: Bug
  Components: build, mapreduce, test
Affects Versions: 0.92.2
 Environment: Hadoop QA - precommit builds
Reporter: Laxman
Priority: Blocker
  Labels: build-failure, mapreduce, test-fail

 Many of the MR testcases are failing in PreCommit builds (triggered by Hadoop 
 QA).
 Failing testcases are
 a) TestImportTsv
 b) TestHFileOutputFormat
 c) TestTableMapReduce

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5625) Avoid byte buffer allocations when reading a value from a Result object


[ 
https://issues.apache.org/jira/browse/HBASE-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237134#comment-13237134
 ] 

stack commented on HBASE-5625:
--

How do I know the provided buffer big enough?

{code}
+  public KeyValue(byte [] buffer,
{code}

What can we do to ensure that the checks in createByteArray and writeByteArray, 
your new method, are shared rather than duplicated?

Put all of the below on one line or add curlies:

{code}
+if (!Bytes.equals(family, foffset, flength, this.bytes, o, fl))
+  return false;
{code}

Ditto for other similar formattings

Do we need both containsNonEmptyColumn and containsEmptyColumn?

Should loadValue be in KeyValue?

I like createFirstOnRow as method name instead of getSearchTerm.  The former 
describes what is happening?

We should use your new binarySearch instead of what we have now?

Thanks for looking into this stuff.

What made you start digging here?   You have big Result objects with lots of 
kvs?

 Avoid byte buffer allocations when reading a value from a Result object
 ---

 Key: HBASE-5625
 URL: https://issues.apache.org/jira/browse/HBASE-5625
 Project: HBase
  Issue Type: Improvement
  Components: client
Affects Versions: 0.92.1
Reporter: Tudor Scurtu
  Labels: patch
 Attachments: 5625.txt, 5625v2.txt


 When calling Result.getValue(), an extra dummy KeyValue and its associated 
 underlying byte array are allocated, as well as a persistent buffer that will 
 contain the returned value.
 These can be avoided by reusing a static array for the dummy object and by 
 passing a ByteBuffer object as a value destination buffer to the read method.
 The current functionality is maintained, and we have added a separate method 
 call stack that employs the described changes. I will provide more details 
 with the patch.
 Running tests with a profiler, the reduction of read time seems to be of up 
 to 40%.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5613) ThriftServer getTableRegions does not return serverName and port


[ 
https://issues.apache.org/jira/browse/HBASE-5613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237136#comment-13237136
 ] 

Zhihong Yu commented on HBASE-5613:
---

Integrated to TRUNK.

Thanks for the patch Scott

Thanks for the review Lars

HBASE-5613.D2403.5.patch doesn't apply to 0.94 - will do integration later.

 ThriftServer getTableRegions does not return serverName and port
 

 Key: HBASE-5613
 URL: https://issues.apache.org/jira/browse/HBASE-5613
 Project: HBase
  Issue Type: Bug
  Components: thrift
Reporter: Scott Chen
Assignee: Scott Chen
Priority: Minor
 Fix For: 0.94.0, 0.96.0

 Attachments: HBASE-5613.0.94.txt, HBASE-5613.D2403.1.patch, 
 HBASE-5613.D2403.2.patch, HBASE-5613.D2403.3.patch, HBASE-5613.D2403.4.patch, 
 HBASE-5613.D2403.5.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5623) Race condition when rolling the HLog and hlogFlush


 [ 
https://issues.apache.org/jira/browse/HBASE-5623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-5623:
-

Attachment: 5623.txt

How about this?
Could possibly generally synchronize in hLogFlush, since we're just appending 
and not sync'ing.

 Race condition when rolling the HLog and hlogFlush
 --

 Key: HBASE-5623
 URL: https://issues.apache.org/jira/browse/HBASE-5623
 Project: HBase
  Issue Type: Bug
  Components: wal
Affects Versions: 0.94.0
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Priority: Critical
 Fix For: 0.94.0

 Attachments: 5623.txt, 5623v2.txt, HBASE-5623_v0.patch


 When doing a ycsb test with a large number of handlers 
 (regionserver.handler.count=60), I get the following exceptions:
 {code}
 Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: 
 java.lang.NullPointerException
   at 
 org.apache.hadoop.io.SequenceFile$Writer.getLength(SequenceFile.java:1099)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.getLength(SequenceFileLogWriter.java:314)
   at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1291)
   at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1388)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:2192)
   at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1985)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3400)
   at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:366)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1351)
   at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:920)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:152)
   at $Proxy1.multi(Unknown Source)
   at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1691)
   at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1689)
   at 
 org.apache.hadoop.hbase.client.ServerCallable.withoutRetries(ServerCallable.java:214)
 {code}
 and 
 {code}
   java.lang.NullPointerException
   at 
 org.apache.hadoop.io.SequenceFile$Writer.checkAndWriteSync(SequenceFile.java:1026)
   at 
 org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1068)
   at 
 org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1035)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.append(SequenceFileLogWriter.java:279)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.hlogFlush(HLog.java:1237)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1271)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1391)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:2192)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1985)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3400)
   at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:366)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1351)
 {code}
 It seems the root cause of the issue is that we open a new log writer and 
 close the old one at HLog#rollWriter() holding the updateLock, but the other 
 threads doing syncer() calls
 {code} 
 logSyncerThread.hlogFlush(this.writer);
 {code}
 without holding the updateLock. LogSyncer only synchronizes against 
 concurrent appends and flush(), but not on the passed writer, which can be 
 closed already by rollWriter(). In this case, since 
 SequenceFile#Writer.close() sets it's out field as null, we get the NPE. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see:

[jira] [Commented] (HBASE-5573) Replace client ZooKeeper watchers by simple ZooKeeper reads

2012-03-23 Thread nkeywal (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237143#comment-13237143
]

nkeywal commented on HBASE-5573:

bq. OK. Any recommendation you can make here having been down deep in this
code? We should make everyone go via ZKUtils and via ZKAssign, etc., and clean
up any other errant use of zkw directly? Would that be good to do?
It would do no harm as it's not good to have two APIs. I could be a first step
to change the internal design. I haven't checked the impact.

bq. Do you want to be consistent? You call methods getZKW most times and then
getZooKeeperWatcher in this test code (I prefer the latter).
Ok, I will change all this to getZooKeeperWatcher.

Replace client ZooKeeper watchers by simple ZooKeeper reads
---

[jira] [Updated] (HBASE-5624) Aborting regionserver when splitting region, may cause daughter region not assigned by ServerShutdownHandler.