[jira] [Issue Comment Edited] (HBASE-5545) region can't be opened for a long time. Because the creating File failed.

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256938#comment-13256938
 ] 

Lars Hofhansl edited comment on HBASE-5545 at 4/18/12 8:43 PM:
---

I'm going to commit this to 0.94 and 0.96 in the next few minutes.

  was (Author: lhofhansl):
I'm going to commit this 0.94 and 0.96 in the next few minutes.
  
> region can't be opened for a long time. Because the creating File failed.
> -
>
> Key: HBASE-5545
> URL: https://issues.apache.org/jira/browse/HBASE-5545
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.90.6
>Reporter: gaojinchao
>Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.7, 0.92.2, 0.94.0
>
> Attachments: HBASE-5545.patch, HBASE-5545.patch
>
>
> Scenario:
> 
> 1. File is created 
> 2. But while writing data, all datanodes might have crashed. So writing data 
> will fail.
> 3. Now even if close is called in finally block, close also will fail and 
> throw the Exception because writing data failed.
> 4. After this if RS try to create the same file again, then 
> AlreadyBeingCreatedException will come.
> Suggestion to handle this scenario.
> ---
> 1. Check for the existence of the file, if exists delete the file and create 
> new file. 
> Here delete call for the file will not check whether the file is open or 
> closed.
> Overwrite Option:
> 
> 1. Overwrite option will be applicable if you are trying to overwrite a 
> closed file.
> 2. If the file is not closed, then even with overwrite option Same 
> AlreadyBeingCreatedException will be thrown.
> This is the expected behaviour to avoid the Multiple clients writing to same 
> file.
> Region server logs:
> org.apache.hadoop.ipc.RemoteException: 
> org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to 
> create file /hbase/test1/12c01902324218d14b17a5880f24f64b/.tmp/.regioninfo 
> for 
> DFSClient_hb_rs_158-1-131-48,20020,1331107668635_1331107669061_-252463556_25 
> on client 158.1.132.19 because current leaseholder is trying to recreate file.
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:1570)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1440)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1382)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:658)
> at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:547)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1137)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1133)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1131)
> at org.apache.hadoop.ipc.Client.call(Client.java:961)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:245)
> at $Proxy6.create(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at $Proxy6.create(Unknown Source)
> at 
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.(DFSClient.java:3643)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:778)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:364)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:630)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:611)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:518)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkRegioninfoOnFilesystem(HRegion.java:424)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:340)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2672)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2658)
> at 
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:330)
> at 
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:116)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:158)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.

[jira] [Issue Comment Edited] (HBASE-5782) Edits can be appended out of seqid order since HBASE-4487

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256203#comment-13256203
 ] 

Lars Hofhansl edited comment on HBASE-5782 at 4/18/12 5:03 AM:
---

This is what I would commit to 0.94. (plus a test that either Stack or I would 
write).

  was (Author: lhofhansl):
This is would I would commit to 0.94. (plus a test that either Stack or I 
would write).
  
> Edits can be appended out of seqid order since HBASE-4487
> -
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 0.94.0
>Reporter: Gopinathan A
>Assignee: Lars Hofhansl
>Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782-v3.txt, 
> 5782.txt, 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the 
> regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. 
> HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5782) Edits can be appended out of seqid order since HBASE-4487

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256094#comment-13256094
 ] 

Lars Hofhansl edited comment on HBASE-5782 at 4/18/12 12:15 AM:


I'm +1 on committing on my patch for 0.94.0. We can then either revisit for 
0.94.1 or 0.96.

  was (Author: lhofhansl):
I'm +1 on committing on my patch for 0.94.0. We can then either revisit for 
0.94.1+ or 0.96.
  
> Edits can be appended out of seqid order since HBASE-4487
> -
>
> Key: HBASE-5782
> URL: https://issues.apache.org/jira/browse/HBASE-5782
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 0.94.0
>Reporter: Gopinathan A
>Assignee: ramkrishna.s.vasudevan
>Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782.txt, 
> 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt
>
>
> Create a table with 1000 splits, after the region assignemnt, kill the 
> regionserver wich contains META table.
> Here few regions are missing after the log splitting and region assigment. 
> HBCK report shows multiple region holes are got created.
> Same scenario was verified mulitple times in 0.92.1, no issues.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5795) hbase-3927 breaks 0.92<->0.94 compatibility

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254414#comment-13254414
 ] 

Lars Hofhansl edited comment on HBASE-5795 at 4/15/12 7:31 PM:
---

+1 on patch.

  was (Author: lhofhansl):
+1 om patch.
  
> hbase-3927 breaks 0.92<->0.94 compatibility
> ---
>
> Key: HBASE-5795
> URL: https://issues.apache.org/jira/browse/HBASE-5795
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
> Fix For: 0.94.0
>
> Attachments: 5794.txt, 5795-v1.txt
>
>
> This commit broke our 0.92/0.94 compatibility:
> {code}
> 
> r1136686 | stack | 2011-06-16 14:18:08 -0700 (Thu, 16 Jun 2011) | 1 line
> HBASE-3927 display total uncompressed byte size of a region in web UI
> {code}
> I just tried the new RC for 0.94.  I brought up a 0.94 master on a 0.92 
> cluster and rather than just digest version 1 of the HServerLoad, I get this:
> {code}
> 2012-04-14 22:47:59,752 WARN org.apache.hadoop.ipc.HBaseServer: Unable to 
> read call parameters for client 10.4.14.38
> java.io.IOException: Error in readFields
> at 
> org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:684)
> at 
> org.apache.hadoop.hbase.ipc.Invocation.readFields(Invocation.java:125)
> at 
> org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:1269)
> at 
> org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1184)
> at 
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:722)
> at 
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:513)
> at 
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:488)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: A record version mismatch occured. Expecting v2, found v1
> at 
> org.apache.hadoop.io.VersionedWritable.readFields(VersionedWritable.java:46)
> at 
> org.apache.hadoop.hbase.HServerLoad$RegionLoad.readFields(HServerLoad.java:379)
> at 
> org.apache.hadoop.hbase.HServerLoad.readFields(HServerLoad.java:686)
> at 
> org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:681)
> ... 9 more
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5677) The master never does balance because duplicate openhandled the one region

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252516#comment-13252516
 ] 

Lars Hofhansl edited comment on HBASE-5677 at 4/12/12 4:00 PM:
---

@xufeng: Are you saying your change is good to go in?


  was (Author: lhofhansl):
@xufeng: Are you saying you change is good to go in?

  
> The master never does balance because duplicate openhandled the one region
> --
>
> Key: HBASE-5677
> URL: https://issues.apache.org/jira/browse/HBASE-5677
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.6
> Environment: 0.90
>Reporter: xufeng
>Assignee: xufeng
> Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0
>
> Attachments: 5677-proposal.txt, HBASE-5677-90-v1.patch, 
> surefire-report_no_patched_v1.html, surefire-report_patched_v1.html
>
>
> If region be assigned When the master is doing initialization(before do 
> processFailover),the region will be duplicate openhandled.
> because the unassigned node in zookeeper will be handled again in 
> AssignmentManager#processFailover()
> it cause the region in RIT,thus the master never does balance.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5741) ImportTsv does not check for table existence

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252095#comment-13252095
 ] 

Lars Hofhansl edited comment on HBASE-5741 at 4/12/12 12:33 AM:


Even HFileOutputFormat uses the table in HBase to find out about splits. We 
will rarely bulk import into an empty table that is not pre-split...? (I'm 
assuming here, you folks at Cloudera see many more use cases that I do)

If you believe strongly that we should add this, let's do it :)
We can do Import in a separate patch, or not do that as we see fit.


  was (Author: lhofhansl):
Even the created HFileOutputFormat use the table in HBase to find out about 
splits. We will rarely bulk import into an empty table that is not 
pre-split...? (I'm assuming here, you folks at Cloudera see many more use cases 
that I do)

If you believe strongly that we should add this, let's do it :)
We can do Import in a separate patch, or not do that as we see fit.

  
> ImportTsv does not check for table existence 
> -
>
> Key: HBASE-5741
> URL: https://issues.apache.org/jira/browse/HBASE-5741
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 0.90.4
>Reporter: Clint Heath
>Assignee: Himanshu Vashishtha
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 5741-94.txt, 5741-v3.txt, HBase-5741-v2.patch, 
> HBase-5741.patch
>
>
> The usage statement for the "importtsv" command to hbase claims this:
> "Note: if you do not use this option, then the target table must already 
> exist in HBase" (in reference to the "importtsv.bulk.output" command-line 
> option)
> The truth is, the table must exist no matter what, importtsv cannot and will 
> not create it for you.
> This is the case because the createSubmittableJob method of ImportTsv does 
> not even attempt to check if the table exists already, much less create it:
> (From org.apache.hadoop.hbase.mapreduce.ImportTsv.java)
> 305 HTable table = new HTable(conf, tableName);
> The HTable method signature in use there assumes the table exists and runs a 
> meta scan on it:
> (From org.apache.hadoop.hbase.client.HTable.java)
> 142 * Creates an object to access a HBase table.
> ...
> 151 public HTable(Configuration conf, final String tableName)
> What we should do inside of createSubmittableJob is something similar to what 
> the "completebulkloads" command would do:
> (Taken from org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.java)
> 690 boolean tableExists = this.doesTableExist(tableName);
> 691 if (!tableExists) this.createTable(tableName,dirPath);
> Currently the docs are misleading, the table in fact must exist prior to 
> running importtsv. We should check if it exists rather than assume it's 
> already there and throw the below exception:
> 12/03/14 17:15:42 WARN client.HConnectionManager$HConnectionImplementation: 
> Encountered problems when prefetch META table: 
> org.apache.hadoop.hbase.TableNotFoundException: Cannot find row in .META. for 
> table: myTable2, row=myTable2,,99
>   at 
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
> ...

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5720) HFileDataBlockEncoderImpl uses wrong header size when reading HFiles with no checksums

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13249435#comment-13249435
 ] 

Lars Hofhansl edited comment on HBASE-5720 at 4/8/12 12:44 AM:
---

Added you as contributor Matt.

  was (Author: lhofhansl):
Added you as conributor Matt.
  
> HFileDataBlockEncoderImpl uses wrong header size when reading HFiles with no 
> checksums
> --
>
> Key: HBASE-5720
> URL: https://issues.apache.org/jira/browse/HBASE-5720
> Project: HBase
>  Issue Type: Bug
>  Components: io, regionserver
>Affects Versions: 0.94.0
>Reporter: Matt Corgan
>Assignee: Matt Corgan
>Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 5720-trunk-v2.txt, 5720-trunk.txt, 5720v4.txt, 
> 5720v4.txt, 5720v4.txt, HBASE-5720-v1.patch, HBASE-5720-v2.patch, 
> HBASE-5720-v3.patch
>
>
> When reading a .92 HFile without checksums, encoding it, and storing in the 
> block cache, the HFileDataBlockEncoderImpl always allocates a dummy header 
> appropriate for checksums even though there are none.  This corrupts the 
> byte[].
> Attaching a patch that allocates a DUMMY_HEADER_NO_CHECKSUM in that case 
> which I think is the desired behavior.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5682) Allow HConnectionImplementation to recover from ZK connection loss (for 0.94 only)

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13243662#comment-13243662
 ] 

Lars Hofhansl edited comment on HBASE-5682 at 4/1/12 5:49 AM:
--

Found the problem.
The ClusterId could remain null permanently if 
HConnection.getZookeeperWatcher() was called. That would initialize 
HConnectionImplementation.zookeeper, and hence not reset clusterid in 
ensureZookeeperTrackers.
TestZookeeper.testClientSessionExpired does that.

Also in TestZookeeper.testClientSessionExpired the state might be CONNECTING 
rather than CONNECTED depending on timing.

Upon inspection I also made clusterId, rootRegionTracker, masterAddressTracker, 
and zooKeeper volatile, because they can be modified by a different thread, but 
are not exclusively accessed in a synchronized block (existing problem).

New patch that fixes the problem, passes all tests.

TestZookeeper seems to have good coverage. If I can think of more tests, I'll 
add them there.

  was (Author: lhofhansl):
Found the problem.
The ClusterId could be remain null permanently if 
HConnection.getZookeeperWatcher() was called. That would initialize 
HConnectionImplementation.zookeeper, and hence not reset clusterid in 
ensureZookeeperTrackers.
TestZookeeper.testClientSessionExpired does that.

Also in TestZookeeper.testClientSessionExpired the state might be CONNECTING 
rather than CONNECTED depending on timing.

Upon inspection I also made clusterId, rootRegionTracker, masterAddressTracker, 
and zooKeeper volatile, because they can be modified by a different thread, but 
are not exclusively accessed in a synchronized block (exiting problem).

New patch that fixes the problem, passes all tests.

TestZookeeper seems to have good coverage. If I can think of more tests, I'll 
add them there.
  
> Allow HConnectionImplementation to recover from ZK connection loss (for 0.94 
> only)
> --
>
> Key: HBASE-5682
> URL: https://issues.apache.org/jira/browse/HBASE-5682
> Project: HBase
>  Issue Type: Improvement
>  Components: client
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
>Priority: Critical
> Fix For: 0.94.0
>
> Attachments: 5682-all-v2.txt, 5682-all.txt, 5682-v2.txt, 5682.txt
>
>
> Just realized that without this HBASE-4805 is broken.
> I.e. there's no point keeping a persistent HConnection around if it can be 
> rendered permanently unusable if the ZK connection is lost temporarily.
> Note that this is fixed in 0.96 with HBASE-5399 (but that seems to big to 
> backport)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5682) Add retry logic in HConnectionImplementation#resetZooKeeperTrackers (for 0.94 only)

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242873#comment-13242873
 ] 

Lars Hofhansl edited comment on HBASE-5682 at 3/31/12 3:01 AM:
---

I am willing to sink the 0.94.0rc for this.

  was (Author: lhofhansl):
I am willing to sink with the 0.94.0rc for this.
  
> Add retry logic in HConnectionImplementation#resetZooKeeperTrackers (for 0.94 
> only)
> ---
>
> Key: HBASE-5682
> URL: https://issues.apache.org/jira/browse/HBASE-5682
> Project: HBase
>  Issue Type: Improvement
>  Components: client
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.1
>
> Attachments: 5682.txt
>
>
> Just realized that without this HBASE-4805 is broken.
> I.e. there's no point keeping a persistent HConnection around if it can be 
> rendered permanently unusable if the ZK connection is lost temporarily.
> Note that this is fixed in 0.96 with HBASE-5399 (but that seems to big to 
> backport)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5682) Add retry logic in HConnectionImplementation#resetZooKeeperTrackers (port to 0.94)

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242871#comment-13242871
 ] 

Lars Hofhansl edited comment on HBASE-5682 at 3/31/12 12:07 AM:


Here's a patch.
Please have a careful look. I can upload to RB too.

The idea is that if this is an unmanaged Connection (see HBASE-4805), a ZK 
connection is re-establish whenever needed (if it was lost before).

This patch is somewhat more complicated than I'd like, because I did not want 
to change the behavior for managed (default) connections.
If we like I can make this the default behavior... Seems much more robust than 
the current behavior.

I tested this manually, and the connection (if created with 
HConnectionManager.createConnection, and hence unmanaged) recovers from loosing 
both the HBase and ZK connections.

(Interestingly in plain HBase 0.94 the client *never* recovers from this - even 
with the default connection behavior.)


  was (Author: lhofhansl):
Here's a patch.
Please have a careful look. I can upload to RB too.

The idea is that if this is an unmanaged Connection (see HBASE-5399), a ZK 
connection is re-establish whenever needed (if it was lost before).

This patch is somewhat more complicated than I'd like, because I did not want 
to change the behavior for managed (default) connections.
If we like I can make this the default behavior... Seems much more robust than 
the current behavior.

I tested this manually, and the connection (if created with 
HConnectionManager.createConnection, and hence unmanaged) recovers from loosing 
both the HBase and ZK connections.

(Interestingly in plain HBase 0.94 the client *never* recovers from this - even 
with the default connection behavior.)

  
> Add retry logic in HConnectionImplementation#resetZooKeeperTrackers (port to 
> 0.94)
> --
>
> Key: HBASE-5682
> URL: https://issues.apache.org/jira/browse/HBASE-5682
> Project: HBase
>  Issue Type: Sub-task
>  Components: client
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.1
>
> Attachments: 5682.txt
>
>
> Just realized that without this HBASE-4805 is broken.
> I.e. there's no point keeping a persistent HConnection around if it can be 
> rendered permanently unusable if the ZK connection is lost temporarily.
> Note that this is fixed in 0.96 with HBASE-5399 (but that seems to big to 
> backport)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5592) Make it easier to get a table from shell

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13234986#comment-13234986
 ] 

Lars Hofhansl edited comment on HBASE-5592 at 3/21/12 8:26 PM:
---

Let's just mark this one fixed.
Jesse will revert with his changes (in 0.94+) if necessary.

  was (Author: lhofhansl):
Let's just mark this one fix.
Jesse will revert with his changes (in 0.94+) if necessary.
  
> Make it easier to get a table from shell
> 
>
> Key: HBASE-5592
> URL: https://issues.apache.org/jira/browse/HBASE-5592
> Project: HBase
>  Issue Type: Improvement
>  Components: shell
>Affects Versions: 0.94.0
>Reporter: Ben West
>Assignee: Ben West
>Priority: Trivial
>  Labels: shell
> Fix For: 0.92.2, 0.94.0, 0.96.0
>
> Attachments: publicTable.patch
>
>
> The one argument constructor to HTable was removed at some point, which means 
> that you now have to pass in a Configuration to instantiate an HTable. This 
> is annoying for me when I create quick scripts.
> This JIRA is a tiny patch which lets you get an HTable instance in the shell 
> by doing
> {code}foo_table = @shell.hbase_table('foo').table{code}
> Basically, it is changing table to be a public member rather than a private 
> one.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5569) TestAtomicOperation.testMultiRowMutationMultiThreads fails occasionally

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13230207#comment-13230207
 ] 

Lars Hofhansl edited comment on HBASE-5569 at 3/15/12 2:42 PM:
---

Ran all tests in TestAtomicOperation more that 3000 times without a failure.

  was (Author: lhofhansl):
Ran all test in TestAtomicOperation more that 3000 times without a failure.
  
> TestAtomicOperation.testMultiRowMutationMultiThreads fails occasionally
> ---
>
> Key: HBASE-5569
> URL: https://issues.apache.org/jira/browse/HBASE-5569
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
>Priority: Minor
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 5569-v2.txt, 5569.txt, 
> TestAtomicOperation-output.trunk_120313.rar
>
>
> What I pieced together so far is that it is the *scanning* side that has 
> problems sometimes.
> Every time I see a assertion failure in the log I see this before:
> {quote}
> 2012-03-12 21:48:49,523 DEBUG [Thread-211] regionserver.StoreScanner(499): 
> Storescanner.peek() is changed where before = 
> rowB/colfamily11:qual1/75366/Put/vlen=6,and after = 
> rowB/colfamily11:qual1/75203/DeleteColumn/vlen=0
> {quote}
> The order of if the Put and Delete is sometimes reversed.
> The test threads should always see exactly one KV, if the "before" was the 
> Put the thread see 0 KVs, if the "before" was the Delete the threads see 2 
> KVs.
> This debug message comes from StoreScanner to checkReseek. It seems we still 
> some consistency issue with scanning sometimes :(

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5569) TestAtomicOperation.testMultiRowMutationMultiThreads fails occasionally

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229028#comment-13229028
 ] 

Lars Hofhansl edited comment on HBASE-5569 at 3/14/12 6:48 AM:
---

But note that sometime the other case happens, and we see two rows!

  was (Author: lhofhansl):
But note that sometime the other case happens, and we two rows!
  
> TestAtomicOperation.testMultiRowMutationMultiThreads fails occasionally
> ---
>
> Key: HBASE-5569
> URL: https://issues.apache.org/jira/browse/HBASE-5569
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Priority: Minor
> Attachments: TestAtomicOperation-output.trunk_120313.rar
>
>
> What I pieced together so far is that it is the *scanning* side that has 
> problems sometimes.
> Every time I see a assertion failure in the log I see this before:
> {quote}
> 2012-03-12 21:48:49,523 DEBUG [Thread-211] regionserver.StoreScanner(499): 
> Storescanner.peek() is changed where before = 
> rowB/colfamily11:qual1/75366/Put/vlen=6,and after = 
> rowB/colfamily11:qual1/75203/DeleteColumn/vlen=0
> {quote}
> The order of if the Put and Delete is sometimes reversed.
> The test threads should always see exactly one KV, if the "before" was the 
> Put the thread see 0 KVs, if the "before" was the Delete the threads see 2 
> KVs.
> This debug message comes from StoreScanner to checkReseek. It seems we still 
> some consistency issue with scanning sometimes :(

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5569) TestAtomicOperation.testMultiRowMutationMultiThreads fails occasionally

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13228998#comment-13228998
 ] 

Lars Hofhansl edited comment on HBASE-5569 at 3/14/12 6:29 AM:
---

Well... The whole point of the new API was to have atomic operations.
The Put and the Delete are executed atomically together and visible at the same 
time.
Note that the code alternates putting row and deleting row2, and then putting 
row2 and deleting row. The scan than ensure that only exactly one column is 
visible.

In this case the scan *itself* is inconsistent. And worse, as Nicolas (N) found 
out is that even testRowMutationMultiThreads fails sometimes, and that is just 
a single row and should never happen.

So I am not entirely convinced the test is at fault.

For example the scenario described above:
if
{code}
Put p = new Put(row2, ts);
p.add(fam1, qual1, value1);
mrm.add(p);
Delete d = new Delete(row);
d.deleteColumns(fam1, qual1, ts);
mrm.add(d);
{code}
happened between 
{code}
region.mutateRowsWithLocks(mrm, rowsToLock);
{code}

and
{code}

Scan s = new Scan(row);
RegionScanner rs = region.getScanner(s);
  List r = new ArrayList();
  while(rs.next(r));
{code}

Both the Put and the Delete would happen atomically with the same WALEdit and 
the same MVCC writepoint. So the scan will now see the other row (it sees 
either row or row, because row -RowA- sorts before row2 -RowB-)
This has nothing to do with race conditions between threads, but only occurs 
with flushes in the test. I'll remove the forced flushes and then run the test 
again.


  was (Author: lhofhansl):
Well... The whole point of the new API was to have atomic operations.
The Put and the Delete are executed atomically together and visible at the same 
time.
Note that the code alternates putting row and deleting row2, and then putting 
row2 and deleting row. The scan than ensure that only exactly one column is 
visible.

In this case the scan *itself* is inconsistent. And worse, as Nicolas (N) found 
out is that even testRowMutationMultiThreads fails sometimes, and that is just 
a single row and should never happen.

So I am not entirely convinced the test is at fault.

For example the scenario described above if Between the time thread1 execute
if
{code}
Put p = new Put(row2, ts);
p.add(fam1, qual1, value1);
mrm.add(p);
Delete d = new Delete(row);
d.deleteColumns(fam1, qual1, ts);
mrm.add(d);
{code}
happened between 
{code}
region.mutateRowsWithLocks(mrm, rowsToLock);
{code}

and
{code}

Scan s = new Scan(row);
RegionScanner rs = region.getScanner(s);
  List r = new ArrayList();
  while(rs.next(r));
{code}

Both the Put and the Delete would happen atomically with the same WALEdit and 
the same MVCC writepoint. So the scan will now see the other row.
This has nothing to do with race conditions between threads, but only occurs 
with flushes in the test. I'll remove the forced flushes and then run the test 
again.
  
> TestAtomicOperation.testMultiRowMutationMultiThreads fails occasionally
> ---
>
> Key: HBASE-5569
> URL: https://issues.apache.org/jira/browse/HBASE-5569
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Priority: Minor
> Attachments: TestAtomicOperation-output.trunk_120313.rar
>
>
> What I pieced together so far is that it is the *scanning* side that has 
> problems sometimes.
> Every time I see a assertion failure in the log I see this before:
> {quote}
> 2012-03-12 21:48:49,523 DEBUG [Thread-211] regionserver.StoreScanner(499): 
> Storescanner.peek() is changed where before = 
> rowB/colfamily11:qual1/75366/Put/vlen=6,and after = 
> rowB/colfamily11:qual1/75203/DeleteColumn/vlen=0
> {quote}
> The order of if the Put and Delete is sometimes reversed.
> The test threads should always see exactly one KV, if the "before" was the 
> Put the thread see 0 KVs, if the "before" was the Delete the threads see 2 
> KVs.
> This debug message comes from StoreScanner to checkReseek. It seems we still 
> some consistency issue with scanning sometimes :(

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5542) Unify HRegion.mutateRowsWithLocks() and HRegion.processRow()

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226368#comment-13226368
 ] 

Lars Hofhansl edited comment on HBASE-5542 at 3/9/12 7:35 PM:
--

Can we let this sit for a bit? I would like to spend some time thinking about 
this. doMiniBatchPut almost follows the same logic, should be able to unify 
that code as well. It's not needed for 0.94, so it's not time critical.

Thanks for working on this Scott!

  was (Author: lhofhansl):
Can we let this sit for a bit. I would like to spend some time thinking 
about this. doMiniBatchPut almost follows the same logic, should be able to 
unify that code as well. It's not needed for 0.94, so it's not time critical.

Thanks for working on this Scott!
  
> Unify HRegion.mutateRowsWithLocks() and HRegion.processRow()
> 
>
> Key: HBASE-5542
> URL: https://issues.apache.org/jira/browse/HBASE-5542
> Project: HBase
>  Issue Type: Improvement
>Reporter: Scott Chen
>Assignee: Scott Chen
> Fix For: 0.96.0
>
> Attachments: HBASE-5542.D2217.1.patch, HBASE-5542.D2217.2.patch
>
>
> mutateRowsWithLocks() does atomic mutations on multiple rows.
> processRow() does atomic read-modify-writes on a single row.
> It will be useful to generalize both and have a
> processRowsWithLocks() that does atomic read-modify-writes on multiple rows.
> This also helps reduce some redundancy in the codes.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5509) MR based copier for copying HFiles (trunk version)

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13224075#comment-13224075
 ] 

Lars Hofhansl edited comment on HBASE-5509 at 3/7/12 7:34 AM:
--

Created https://reviews.apache.org/r/4218/ for better commenting.
Let's get this thing into trunk. And maybe 0.94 :)


  was (Author: lhofhansl):
Create https://reviews.apache.org/r/4218/ for better commenting.
Let's get this thing into trunk. And maybe 0.94 :)

  
> MR based copier for copying HFiles (trunk version)
> --
>
> Key: HBASE-5509
> URL: https://issues.apache.org/jira/browse/HBASE-5509
> Project: HBase
>  Issue Type: Sub-task
>  Components: documentation, regionserver
>Reporter: Karthik Ranganathan
>Assignee: Lars Hofhansl
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 5509-v2.txt, 5509.txt
>
>
> This copier is a modification of the distcp tool in HDFS. It does the 
> following:
> 1. List out all the regions in the HBase cluster for the required table
> 2. Write the above out to a file
> 3. Each mapper 
>3.1 lists all the HFiles for a given region by querying the regionserver
>3.2 copies all the HFiles
>3.3 outputs success if the copy succeeded, failure otherwise. Failed 
> regions are retried in another loop
> 4. Mappers are placed on nodes which have maximum locality for a given region 
> to speed up copying

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5512) Add support for INCLUDE_AND_SEEK_USING_HINT

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222180#comment-13222180
 ] 

Lars Hofhansl edited comment on HBASE-5512 at 3/5/12 6:36 AM:
--

Let's wait for more input in HBASE-2038. I'm feeling half way inclined to -1 my 
own patch :)
Imagine we wanted to add INCLUDE_AND_NEXT_ROW as well as INCLUDE_AND_NEXT_COL 
to filter. These outcomes would have to merged with whatever the colum tracker 
returns.
It's not complicated, but will make the code hard to read with questionable 
gain (as I say in HBASE-2038). I think Anoop wants to do some performance 
testing.

  was (Author: lhofhansl):
Let's wait for more input in HBASE-2038. I'm feeling half way include to -1 
my own patch :)
Imagine we wanted to add INCLUDE_AND_NEXT_ROW as well as INCLUDE_AND_NEXT_COL 
to filter. These outcomes would have to merged with whatever the colum tracker 
returns.
It's not complicated, but will make the code hard to read with questionable 
gain (as I say in HBASE-2038). I think Anoop wants to do some performance 
testing.
  
> Add support for INCLUDE_AND_SEEK_USING_HINT
> ---
>
> Key: HBASE-5512
> URL: https://issues.apache.org/jira/browse/HBASE-5512
> Project: HBase
>  Issue Type: Improvement
>Reporter: Zhihong Yu
>Assignee: Lars Hofhansl
> Attachments: 5512-v2.txt, 5512.txt
>
>
> This came up from HBASE-2038
> From Anoop:
> - What we wanted from the filter is include a row and then seek to the next 
> row which we are interested in. I cant see such a facility with our Filter 
> right now. Correct me if I am wrong. So suppose we already seeked to one row 
> and this need to be included in the result, then the Filter should return 
> INCLUDE. Then when the next next() call happens, then only we can return a 
> SEEK_USING_HINT. So one extra row reading is needed. This might create even 
> one unwanted HFileBlock fetch (who knows).
> Can we add reseek() at higher level?
> From Lars:
> Yep, for that we'd need to add INCLUDE_AND_SEEK_USING_HINT (similar to the 
> INCLUDE_AND_SEEK_NEXT_ROW that we already have). Shouldn't be hard to add, 
> I'm happy to do that, if that's the route we want to go with this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-2231) Compaction events should be written to HLog

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13217966#comment-13217966
 ] 

Lars Hofhansl edited comment on HBASE-2231 at 2/28/12 7:00 AM:
---

Moving out of 0.94. Probably not a blocker if it can go unfixed for a long time.

  was (Author: lhofhansl):
Moving out of 0.96. Probably not a blocker if it can go unfixed for a long 
time.
  
> Compaction events should be written to HLog
> ---
>
> Key: HBASE-2231
> URL: https://issues.apache.org/jira/browse/HBASE-2231
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Blocker
>  Labels: moved_from_0_20_5
> Fix For: 0.96.0
>
> Attachments: hbase-2231-testcase.txt, hbase-2231.txt
>
>
> The sequence for a compaction should look like this:
> # Compact region to "new" files
> # Write a "Compacted Region" entry to the HLog
> # Delete "old" files
> This deals with a case where the RS has paused between step 1 and 2 and the 
> regions have since been reassigned.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-4365) Add a decent heuristic for region size

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13211619#comment-13211619
 ] 

Lars Hofhansl edited comment on HBASE-4365 at 2/20/12 12:39 AM:


Wouldn't we potentially do a lot of splitting when there are many regionservers?
(Maybe I am not grokking this fully)

If we take the square of the of the number of regions, and say we have 10gb 
regions and flush size of 128mb, we'd reach the 10gb after at 9 regions of the 
table on the same regionserver.
We were planning a region size of 5gb and flush size of 256mb, that would still 
be 5 regions.
(10gb/128mb ~ 78, 5gb/256mb ~ 19)


  was (Author: lhofhansl):
Wouldn't we potentially do a lot of splitting when there are many 
regionservers?
(Maybe I am not grokking this fully)

If we take the square of the of the number of regions, and say we have 10gb 
regions and flush size of 128mb, we'd read the 10gb after at 9 regions of the 
table on the same regionserver.
We were planning a region size of 5gb and flush size of 256mb, that would still 
be 5 regions.
(10gb/128mb ~ 78, 5gb/256mb ~ 19)

  
> Add a decent heuristic for region size
> --
>
> Key: HBASE-4365
> URL: https://issues.apache.org/jira/browse/HBASE-4365
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.94.0, 0.92.1
>Reporter: Todd Lipcon
>Priority: Critical
>  Labels: usability
> Attachments: 4365.txt
>
>
> A few of us were brainstorming this morning about what the default region 
> size should be. There were a few general points made:
> - in some ways it's better to be too-large than too-small, since you can 
> always split a table further, but you can't merge regions currently
> - with HFile v2 and multithreaded compactions there are fewer reasons to 
> avoid very-large regions (10GB+)
> - for small tables you may want a small region size just so you can 
> distribute load better across a cluster
> - for big tables, multi-GB is probably best

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5424) HTable meet NPE when call getRegionInfo()

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13210845#comment-13210845
 ] 

Lars Hofhansl edited comment on HBASE-5424 at 2/18/12 7:16 AM:
---

Thanks zhiyuan. I'm wondering whether we shouldn't focus on fixing the problem 
that caused the problem rather than pasting over it.

  was (Author: lhofhansl):
That zhiyuan. I'm wondering whether we shouldn't focus on fixing the 
problem that caused the problem rather than pasting over it.
  
> HTable meet NPE when call getRegionInfo()
> -
>
> Key: HBASE-5424
> URL: https://issues.apache.org/jira/browse/HBASE-5424
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.1, 0.90.5
>Reporter: junhua yang
> Attachments: HBASE-5424.patch
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> We meet NPE when call getRegionInfo() in testing environment.
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.hbase.util.Writables.getWritable(Writables.java:75)
> at org.apache.hadoop.hbase.util.Writables.getHRegionInfo(Writables.java:119)
> at org.apache.hadoop.hbase.client.HTable$2.processRow(HTable.java:395)
> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:190)
> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:95)
> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:73)
> at org.apache.hadoop.hbase.client.HTable.getRegionsInfo(HTable.java:418)
> This NPE also make the table.jsp can't show the region information of this 
> table.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5229) Provide basic building blocks for "multi-row" local transactions.

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13200681#comment-13200681
 ] 

Lars Hofhansl edited comment on HBASE-5229 at 2/9/12 9:17 PM:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4822
---


A couple of nits and small implementation details, but overall looks pretty 
good.


http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java


I think is this unnecessary, javadoc should handle inheriting the docs.



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java


or presplitting as is described in other documenttation.



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java


Probably want to wrap NOTE in  tags to call it out.



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java


A javadoc here might be nice to indicate that the nullary constructor is 
actually completely ok to use (as opposed to the more common state of being 
reserved for readFields).



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java


Even though it uses protected structures doesn't mean that its necessarily 
thread safe. In fact, because it is using the standard ArrayList, there is no 
guarantee of safety. Either the class should be marked as not thread safe OR 
the mutations should be wrapped as a concurrent list.



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RowMutation.java


You really don't need to keep around the row anymore either because you can 
get that from the mutations as you already do mutateRows with MultiRowMutation. 
Its nice to store it, but is only going to be checked infrequently and saves 
you a little bit over the wire (which could add up, depending on row size).



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


Suprised this isn't a utility method in HRegion - it seems really useful. 
Maybe worth pulling out for general use.



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


This isn't actually true, right? With multirow, you are actually going to 
lock more than one row (and the lockId null seems kind of a hack around that as 
it is always null, so far).



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


nit: lockID rather than just lid would be slightly descriptive.



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java


But in the comments on the MultiRowMutation you push that checking off onto 
the RS, so no checking really happens then (except, I guess when you try to 
mutate rows on the region and it fails b/c those rows aren't there, but that 
seems kinda late for the check).



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java


Wow, this is ugly. Maybe we should consider some refactoring of this later?



http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java


This class can get easily bloated as we add more types. Might be worth 
considering refactoring this out into its own test.


- Jesse


  was (Author: jirapos...@reviews.apache.org):

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4822
---


A couple of nits and small implementation details, but overall looks pretty 
good.


http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java


I think is this unnecessary, ja

[jira] [Issue Comment Edited] (HBASE-5229) Provide basic building blocks for "multi-row" local transactions.

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201365#comment-13201365
 ] 

Lars Hofhansl edited comment on HBASE-5229 at 2/9/12 9:18 PM:
--


bq.  On 2012-02-06 15:52:43, Amitanand Aiyer wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java,
 line 4212
bq.  > 
bq.  >
bq.  > Do we need to be sorting rowsToLock?
bq.  > 
bq.  > I'm thinking of multiple concurrent mutateRows operation, trying to 
lock the same set of rows.
bq.  > 
bq.  > Perhaps, throwing IOException is going to prevent us from a 
situation where we end up with a deadlock. But, we still might want to sort it 
to ensure (better) progress (no livelock).

MutateRows sorts them (by using a TreeSet with Bytes.BYTES_COMPARATOR, for 
exactly this reason.
Maybe this should be called out here, by making the argument a SortedSet.


- Lars


  was (Author: jirapos...@reviews.apache.org):


bq.  On 2012-02-06 15:52:43, Amitanand Aiyer wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java,
 line 4212
bq.  > 
bq.  >
bq.  > Do we need to be sorting rowsToLock?
bq.  > 
bq.  > I'm thinking of multiple concurrent mutateRows operation, trying to 
lock the same set of rows.
bq.  > 
bq.  > Perhaps, throwing IOException is going to prevent us from a 
situation where we end up with a deadlock. But, we still might want to sort it 
to ensure (better) progress (no livelock).

MutateRows sorts them (by using a TreeSet with Bytes.BYTES_COMPARATOR, for 
exactly this reason.
Maybe this should be called out here, by making the argument a SortedSet.


- Lars


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4833
---


On 2012-02-03 19:59:55, Lars Hofhansl wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3748/
bq.  ---
bq.  
bq.  (Updated 2012-02-03 19:59:55)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This builds on HBASE-3584, HBASE-5203, and HBASE-5304.
bq.  
bq.  Multiple Rows can be locked and applied atomically as long as the 
application ensures that all rows reside in the same Region (by presplitting or 
a custom RegionSplitPolicy).
bq.  At SFDC we can use this to colocate subsets of a tenant's data and allow 
atomic operations over these subsets.
bq.  
bq.  Obviously this is an advanced features and this prominently called out in 
the Javadoc.
bq.  
bq.  
bq.  This addresses bug HBASE-5229.
bq.  https://issues.apache.org/jira/browse/HBASE-5229
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java
 1239953 
bq.  
bq.  Diff: https://reviews.apache.org/r/3748/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Tests added to TestFromClientSide and TestAtomicOperation
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Lars
bq.  
bq.


  
> Provide basic building blocks for "multi-row" local transactio

[jira] [Issue Comment Edited] (HBASE-5229) Provide basic building blocks for "multi-row" local transactions.

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13200911#comment-13200911
 ] 

Lars Hofhansl edited comment on HBASE-5229 at 2/9/12 9:17 PM:
--


bq.  On 2012-02-05 07:26:08, Jesse Yates wrote:
bq.  > A couple of nits and small implementation details, but overall looks 
pretty good.

You're looking at an old version of the patch. :)


bq.  On 2012-02-05 07:26:08, Jesse Yates wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java,
 line 3160
bq.  > 
bq.  >
bq.  > But in the comments on the MultiRowMutation you push that checking 
off onto the RS, so no checking really happens then (except, I guess when you 
try to mutate rows on the region and it fails b/c those rows aren't there, but 
that seems kinda late for the check).

Checking is happening the region.internalMutate.


bq.  On 2012-02-05 07:26:08, Jesse Yates wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java,
 line 786
bq.  > 
bq.  >
bq.  > I think is this unnecessary, javadoc should handle inheriting the 
docs.

It's done elsewhere, it is good to call out that no doc was added here, because 
the interface has the doc.


bq.  On 2012-02-05 07:26:08, Jesse Yates wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java,
 line 284
bq.  > 
bq.  >
bq.  > or presplitting as is described in other documenttation.

Yes, should add this.


bq.  On 2012-02-05 07:26:08, Jesse Yates wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java,
 line 35
bq.  > 
bq.  >
bq.  > Probably want to wrap NOTE in  tags to call it out.

Sure.


bq.  On 2012-02-05 07:26:08, Jesse Yates wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java,
 line 45
bq.  > 
bq.  >
bq.  > A javadoc here might be nice to indicate that the nullary 
constructor is actually completely ok to use (as opposed to the more common 
state of being reserved for readFields).

Good point. Although unless it is called out that you cannot use a constructor 
there should be no reason whyt you couldn't.


bq.  On 2012-02-05 07:26:08, Jesse Yates wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java,
 line 64
bq.  > 
bq.  >
bq.  > Even though it uses protected structures doesn't mean that its 
necessarily thread safe. In fact, because it is using the standard ArrayList, 
there is no guarantee of safety. Either the class should be marked as not 
thread safe OR the mutations should be wrapped as a concurrent list.

I disagree.
This is a client side object and none of the client side objects are threadsafe 
nor should they be (see Put.java/Delete.java/Increment.java/Append.java/etc), 
that's the task of client application.

I misread Ted's comment before, of course this method is not threadsafe.


bq.  On 2012-02-05 07:26:08, Jesse Yates wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RowMutation.java,
 line 95
bq.  > 
bq.  >
bq.  > You really don't need to keep around the row anymore either because 
you can get that from the mutations as you already do mutateRows with 
MultiRowMutation. Its nice to store it, but is only going to be checked 
infrequently and saves you a little bit over the wire (which could add up, 
depending on row size).

Same as Put and Delete (where every KV already has the row).
There is room optimization, but this is not the jira to do that.


bq.  On 2012-02-05 07:26:08, Jesse Yates wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java,
 line 4161
bq.  > 
bq.  >
bq.  > Suprised this isn't a utility method in HRegion - it seems really 
useful. Maybe worth pulling out for general use.

internalMutate?


bq.  On 2012-02-05 07:26:08, Jesse Yates wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java,
 line 4181
bq.  > 

[jira] [Issue Comment Edited] (HBASE-5229) Provide basic building blocks for "multi-row" local transactions.

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201354#comment-13201354
 ] 

Lars Hofhansl edited comment on HBASE-5229 at 2/9/12 9:18 PM:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4833
---



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


Do we need to be sorting rowsToLock?

I'm thinking of multiple concurrent mutateRows operation, trying to lock 
the same set of rows.

Perhaps, throwing IOException is going to prevent us from a situation where 
we end up with a deadlock. But, we still might want to sort it to ensure 
(better) progress (no livelock).


- Amitanand


  was (Author: jirapos...@reviews.apache.org):

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4833
---



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


Do we need to be sorting rowsToLock?

I'm thinking of multiple concurrent mutateRows operation, trying to lock 
the same set of rows.

Perhaps, throwing IOException is going to prevent us from a situation where 
we end up with a deadlock. But, we still might want to sort it to ensure 
(better) progress (no livelock).


- Amitanand


On 2012-02-03 19:59:55, Lars Hofhansl wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3748/
bq.  ---
bq.  
bq.  (Updated 2012-02-03 19:59:55)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This builds on HBASE-3584, HBASE-5203, and HBASE-5304.
bq.  
bq.  Multiple Rows can be locked and applied atomically as long as the 
application ensures that all rows reside in the same Region (by presplitting or 
a custom RegionSplitPolicy).
bq.  At SFDC we can use this to colocate subsets of a tenant's data and allow 
atomic operations over these subsets.
bq.  
bq.  Obviously this is an advanced features and this prominently called out in 
the Javadoc.
bq.  
bq.  
bq.  This addresses bug HBASE-5229.
bq.  https://issues.apache.org/jira/browse/HBASE-5229
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java
 1239953 
bq.  
bq.  Diff: https://reviews.apache.org/r/3748/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Tests added to TestFromClientSide and TestAtomicOperation
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Lars
bq.  
bq.


  
> Provide basic building blocks for "multi-row" local transactions.
> -
>
> Key: HBASE-5229
> URL: https://issues.apache.org/jira/browse/HBASE-5229
> Project: HBase
>  Issue Type: New Feature
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0

[jira] [Issue Comment Edited] (HBASE-5229) Provide basic building blocks for "multi-row" local transactions.

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13199514#comment-13199514
 ] 

Lars Hofhansl edited comment on HBASE-5229 at 2/9/12 9:15 PM:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4788
---



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java


Should be ' to mutateRows'



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java


Should read ' the mutations'



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java


Can the two add() methods be combined into one which accepts Mutation ?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java


Is this method thread-safe ?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java


This comment can be removed, right ?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RowMutation.java


From its name, RowMutation seems to refer to single row. It is a little 
confusing RowMutation extends MultiRowMutation.



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RowMutation.java


version would be read / written twice, right ?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


Should be 'within the region', right ?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


rowsToLock.size() could be smaller than mutations.size(), right ?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java


Can we refer regionName from rm (the MultiRowMutation) ?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java


Should be mutateRows



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java


Should read "atomic mutation"


- Ted


  was (Author: jirapos...@reviews.apache.org):

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4788
---



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java


Should be ' to mutateRows'



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java


Should read ' the mutations'



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java


Can the two add() methods be combined into one which accepts Mutation ?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java


Is this method thread-safe ?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java


This comment can be removed, right ?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RowMutation.java


From its name, RowMutation seems to refer to single row. It is a little 
confusing RowMutation extends MultiRowMutation.



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RowMutation.java


version would be read / written twice, right ?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/j

[jira] [Issue Comment Edited] (HBASE-5229) Provide basic building blocks for "multi-row" local transactions.

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203087#comment-13203087
 ] 

Lars Hofhansl edited comment on HBASE-5229 at 2/8/12 1:13 AM:
--


bq.  On 2012-02-08 01:06:21, Ted Yu wrote:
bq.  >

Ah yes, good catch as usual. If there're no further comments, I'll do that on 
commit.
Thanks Ted.


bq.  On 2012-02-08 01:06:21, Ted Yu wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/MultiRowMutationProtocol.java,
 line 28
bq.  > 
bq.  >
bq.  > Replace operations with transactions.

And presumably remove "atomic" as it is implied by transaction, right?


- Lars


  was (Author: jirapos...@reviews.apache.org):


bq.  On 2012-02-08 01:06:21, Ted Yu wrote:
bq.  >

Ah yes, good catch as usual. If there're no further comments, I'll do that on 
commit.
Thanks Ted.


bq.  On 2012-02-08 01:06:21, Ted Yu wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/MultiRowMutationProtocol.java,
 line 28
bq.  > 
bq.  >
bq.  > Replace operations with transactions.

And presumably remove "atomic" as it is implied by transaction, right?


- Lars


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4889
---


On 2012-02-08 00:15:28, Lars Hofhansl wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3748/
bq.  ---
bq.  
bq.  (Updated 2012-02-08 00:15:28)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This builds on HBASE-3584, HBASE-5203, and HBASE-5304.
bq.  
bq.  Multiple Rows can be locked and applied atomically as long as the 
application ensures that all rows reside in the same Region (by presplitting or 
a custom RegionSplitPolicy).
bq.  At SFDC we can use this to colocate subsets of a tenant's data and allow 
atomic operations over these subsets.
bq.  
bq.  Obviously this is an advanced features and this prominently called out in 
the Javadoc.
bq.  
bq.  
bq.  This addresses bug HBASE-5229.
bq.  https://issues.apache.org/jira/browse/HBASE-5229
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/MultiRowMutationEndpoint.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/MultiRowMutationProtocol.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1241536 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 1241536 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
 1241536 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java
 1241536 
bq.  
bq.  Diff: https://reviews.apache.org/r/3748/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Tests added to TestFromClientSide and TestAtomicOperation
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Lars
bq.  
bq.


  
> Provide basic building blocks for "multi-row" local transactions.
> -
>
> Key: HBASE-5229
> URL: https://issues.apache.org/jira/browse/HBASE-5229
> Project: HBase
>  Issue Type: New Feature
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 5229-endpoint.txt, 5229-multiRow-v2.txt, 
> 5229-multiRow.txt, 5229-seekto-v2.txt, 5229-seekto.txt, 5229.txt
>
>
> In the final iteration, this issue provides a generalized, public 
> mutateRowsWithLocks method on HRegion, that can be used by coprocessors to 
> implement atomic operations efficiently.
> Coprocessors are already region aware, which makes this is a good pairing of 
> APIs. This feature is by design not available to the client via the HTable 
> API.
> It took a long time to arrive at this and I apologize for the public exposure 
> of my (erratic in retrospect) thought processes.
> Was:
> HBase should provide basic building blocks for multi-row local transactions. 
> Local means that we do this by co-loca

[jira] [Issue Comment Edited] (HBASE-5229) Provide basic building blocks for "multi-row" local transactions.

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203080#comment-13203080
 ] 

Lars Hofhansl edited comment on HBASE-5229 at 2/8/12 1:13 AM:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4889
---



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/MultiRowMutationProtocol.java


Replace operations with transactions.



http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java


Please plug in testMultiRowMutation



http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java


testMultiRowMutation here as well.


- Ted


  was (Author: jirapos...@reviews.apache.org):

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4889
---



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/MultiRowMutationProtocol.java


Replace operations with transactions.



http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java


Please plug in testMultiRowMutation



http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java


testMultiRowMutation here as well.


- Ted


On 2012-02-08 00:15:28, Lars Hofhansl wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3748/
bq.  ---
bq.  
bq.  (Updated 2012-02-08 00:15:28)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This builds on HBASE-3584, HBASE-5203, and HBASE-5304.
bq.  
bq.  Multiple Rows can be locked and applied atomically as long as the 
application ensures that all rows reside in the same Region (by presplitting or 
a custom RegionSplitPolicy).
bq.  At SFDC we can use this to colocate subsets of a tenant's data and allow 
atomic operations over these subsets.
bq.  
bq.  Obviously this is an advanced features and this prominently called out in 
the Javadoc.
bq.  
bq.  
bq.  This addresses bug HBASE-5229.
bq.  https://issues.apache.org/jira/browse/HBASE-5229
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/MultiRowMutationEndpoint.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/MultiRowMutationProtocol.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1241536 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 1241536 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
 1241536 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java
 1241536 
bq.  
bq.  Diff: https://reviews.apache.org/r/3748/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Tests added to TestFromClientSide and TestAtomicOperation
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Lars
bq.  
bq.


  
> Provide basic building blocks for "multi-row" local transactions.
> -
>
> Key: HBASE-5229
> URL: https://issues.apache.org/jira/browse/HBASE-5229
> Project: HBase
>  Issue Type: New Feature
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 5229-endpoint.txt, 5229-multiRow-v2.txt, 
> 5229-multiRow.txt, 5229-seekto-v2.txt, 5229-seekto.txt, 5229.txt
>
>
> In the final iteration, this issue provides a generalized, public 
> mutateRowsWithLocks method on HRegion, that can be used by coprocessors to 
> implement atomic operations efficiently.
> Coprocessors are alrea

[jira] [Issue Comment Edited] (HBASE-5229) Provide basic building blocks for "multi-row" local transactions.

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202993#comment-13202993
 ] 

Lars Hofhansl edited comment on HBASE-5229 at 2/8/12 12:29 AM:
---


bq.  On 2012-02-07 23:49:19, Jesse Yates wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/MultiRowMutationEndpoint.java,
 line 39
bq.  > 
bq.  >
bq.  > This could be good to put into client.coprocessor, rather than 
client (when you make the move over).

This is a good point.


- Michael


  was (Author: jirapos...@reviews.apache.org):


bq.  On 2012-02-07 23:49:19, Jesse Yates wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/MultiRowMutationEndpoint.java,
 line 39
bq.  > 
bq.  >
bq.  > This could be good to put into client.coprocessor, rather than 
client (when you make the move over).

This is a good point.


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4886
---


On 2012-02-07 19:41:16, Lars Hofhansl wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3748/
bq.  ---
bq.  
bq.  (Updated 2012-02-07 19:41:16)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This builds on HBASE-3584, HBASE-5203, and HBASE-5304.
bq.  
bq.  Multiple Rows can be locked and applied atomically as long as the 
application ensures that all rows reside in the same Region (by presplitting or 
a custom RegionSplitPolicy).
bq.  At SFDC we can use this to colocate subsets of a tenant's data and allow 
atomic operations over these subsets.
bq.  
bq.  Obviously this is an advanced features and this prominently called out in 
the Javadoc.
bq.  
bq.  
bq.  This addresses bug HBASE-5229.
bq.  https://issues.apache.org/jira/browse/HBASE-5229
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1241350 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 1241350 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/MultiRowMutationEndpoint.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/MultiRowMutationProtocol.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
 1241350 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java
 1241350 
bq.  
bq.  Diff: https://reviews.apache.org/r/3748/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Tests added to TestFromClientSide and TestAtomicOperation
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Lars
bq.  
bq.


  
> Provide basic building blocks for "multi-row" local transactions.
> -
>
> Key: HBASE-5229
> URL: https://issues.apache.org/jira/browse/HBASE-5229
> Project: HBase
>  Issue Type: New Feature
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 5229-endpoint.txt, 5229-multiRow-v2.txt, 
> 5229-multiRow.txt, 5229-seekto-v2.txt, 5229-seekto.txt, 5229.txt
>
>
> In the final iteration, this issue provides a generalized, public 
> mutateRowsWithLocks method on HRegion, that can be used by coprocessors to 
> implement atomic operations efficiently.
> Coprocessors are already region aware, which makes this is a good pairing of 
> APIs. This feature is by design not available to the client via the HTable 
> API.
> It took a long time to arrive at this and I apologize for the public exposure 
> of my (erratic in retrospect) thought processes.
> Was:
> HBase should provide basic building blocks for multi-row local transactions. 
> Local means that we do this by co-locating the data. Global (cross region) 
> transactions are not discussed here.
> After a bit of discussion two solutions have emerged:
> 1. Keep the row-key for determining grouping and location and allow efficient 
> intra-row scanning. A client application would then model ta

[jira] [Issue Comment Edited] (HBASE-5229) Provide basic building blocks for "multi-row" local transactions.

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202996#comment-13202996
 ] 

Lars Hofhansl edited comment on HBASE-5229 at 2/8/12 12:29 AM:
---


bq.  On 2012-02-07 23:49:19, Jesse Yates wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/MultiRowMutationEndpoint.java,
 line 39
bq.  > 
bq.  >
bq.  > This could be good to put into client.coprocessor, rather than 
client (when you make the move over).
bq.  
bq.  Michael Stack wrote:
bq.  This is a good point.

Heh, I am way ahead of you guys :)
Suggested already on the jira.
Will have a new patch soon, hopefully the last revision.
I would like to keep the test in TestFromClientSide, though.


- Lars


  was (Author: jirapos...@reviews.apache.org):


bq.  On 2012-02-07 23:49:19, Jesse Yates wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/MultiRowMutationEndpoint.java,
 line 39
bq.  > 
bq.  >
bq.  > This could be good to put into client.coprocessor, rather than 
client (when you make the move over).
bq.  
bq.  Michael Stack wrote:
bq.  This is a good point.

Heh, I am way ahead of you guys :)
Suggested already on the jira.
Will have a new patch soon, hopefully the last revision.
I would like to keep the test in TestFromClientSide, though.


- Lars


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4886
---


On 2012-02-07 19:41:16, Lars Hofhansl wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3748/
bq.  ---
bq.  
bq.  (Updated 2012-02-07 19:41:16)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This builds on HBASE-3584, HBASE-5203, and HBASE-5304.
bq.  
bq.  Multiple Rows can be locked and applied atomically as long as the 
application ensures that all rows reside in the same Region (by presplitting or 
a custom RegionSplitPolicy).
bq.  At SFDC we can use this to colocate subsets of a tenant's data and allow 
atomic operations over these subsets.
bq.  
bq.  Obviously this is an advanced features and this prominently called out in 
the Javadoc.
bq.  
bq.  
bq.  This addresses bug HBASE-5229.
bq.  https://issues.apache.org/jira/browse/HBASE-5229
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1241350 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 1241350 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/MultiRowMutationEndpoint.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/MultiRowMutationProtocol.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
 1241350 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java
 1241350 
bq.  
bq.  Diff: https://reviews.apache.org/r/3748/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Tests added to TestFromClientSide and TestAtomicOperation
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Lars
bq.  
bq.


  
> Provide basic building blocks for "multi-row" local transactions.
> -
>
> Key: HBASE-5229
> URL: https://issues.apache.org/jira/browse/HBASE-5229
> Project: HBase
>  Issue Type: New Feature
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 5229-endpoint.txt, 5229-multiRow-v2.txt, 
> 5229-multiRow.txt, 5229-seekto-v2.txt, 5229-seekto.txt, 5229.txt
>
>
> In the final iteration, this issue provides a generalized, public 
> mutateRowsWithLocks method on HRegion, that can be used by coprocessors to 
> implement atomic operations efficiently.
> Coprocessors are already region aware, which makes this is a good pairing of 
> APIs. This feature is by design not available to the client via the HTable 
> API.
> It took a long time to arrive at this and I apologize for the public exposure 
> of my (e

[jira] [Issue Comment Edited] (HBASE-5229) Explore building blocks for "multi-row" local transactions.

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202980#comment-13202980
 ] 

Lars Hofhansl edited comment on HBASE-5229 at 2/8/12 12:16 AM:
---

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4886
---

Ship it!


Small nits depending on what you want to do here. Looks good though.


http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/MultiRowMutationEndpoint.java


This could be good to put into client.coprocessor, rather than client (when 
you make the move over).



http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java


It would be nice if this stuff was in its own test, _IF_ you are going to 
pull the endpoint out to src/main. It will help make the functionality a bit 
more obvious.


- Jesse


  was (Author: jirapos...@reviews.apache.org):

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4886
---

Ship it!


Small nits depending on what you want to do here. Looks good though.


http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/MultiRowMutationEndpoint.java


This could be good to put into client.coprocessor, rather than client (when 
you make the move over).



http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java


It would be nice if this stuff was in its own test, _IF_ you are going to 
pull the endpoint out to src/main. It will help make the functionality a bit 
more obvious.


- Jesse


On 2012-02-07 19:41:16, Lars Hofhansl wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3748/
bq.  ---
bq.  
bq.  (Updated 2012-02-07 19:41:16)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This builds on HBASE-3584, HBASE-5203, and HBASE-5304.
bq.  
bq.  Multiple Rows can be locked and applied atomically as long as the 
application ensures that all rows reside in the same Region (by presplitting or 
a custom RegionSplitPolicy).
bq.  At SFDC we can use this to colocate subsets of a tenant's data and allow 
atomic operations over these subsets.
bq.  
bq.  Obviously this is an advanced features and this prominently called out in 
the Javadoc.
bq.  
bq.  
bq.  This addresses bug HBASE-5229.
bq.  https://issues.apache.org/jira/browse/HBASE-5229
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1241350 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 1241350 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/MultiRowMutationEndpoint.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/MultiRowMutationProtocol.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
 1241350 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java
 1241350 
bq.  
bq.  Diff: https://reviews.apache.org/r/3748/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Tests added to TestFromClientSide and TestAtomicOperation
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Lars
bq.  
bq.


  
> Explore building blocks for "multi-row" local transactions.
> ---
>
> Key: HBASE-5229
> URL: https://issues.apache.org/jira/browse/HBASE-5229
> Project: HBase
>  Issue Type: New Feature
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 5229-endpoint.txt, 5229-multiRow-v2.txt, 
> 5229-multiRow.txt, 5229-seekto-v2.txt, 5229-seekto.txt, 5229.txt
>
>
> HBase should provide basic building blocks for multi-row local transactions. 
> Local means that we 

[jira] [Issue Comment Edited] (HBASE-5229) Explore building blocks for "multi-row" local transactions.

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202943#comment-13202943
 ] 

Lars Hofhansl edited comment on HBASE-5229 at 2/8/12 12:16 AM:
---

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4885
---

Ship it!


+1

- Michael


  was (Author: jirapos...@reviews.apache.org):

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4885
---

Ship it!


+1

- Michael


On 2012-02-07 19:41:16, Lars Hofhansl wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3748/
bq.  ---
bq.  
bq.  (Updated 2012-02-07 19:41:16)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This builds on HBASE-3584, HBASE-5203, and HBASE-5304.
bq.  
bq.  Multiple Rows can be locked and applied atomically as long as the 
application ensures that all rows reside in the same Region (by presplitting or 
a custom RegionSplitPolicy).
bq.  At SFDC we can use this to colocate subsets of a tenant's data and allow 
atomic operations over these subsets.
bq.  
bq.  Obviously this is an advanced features and this prominently called out in 
the Javadoc.
bq.  
bq.  
bq.  This addresses bug HBASE-5229.
bq.  https://issues.apache.org/jira/browse/HBASE-5229
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1241350 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 1241350 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/MultiRowMutationEndpoint.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/MultiRowMutationProtocol.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
 1241350 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java
 1241350 
bq.  
bq.  Diff: https://reviews.apache.org/r/3748/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Tests added to TestFromClientSide and TestAtomicOperation
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Lars
bq.  
bq.


  
> Explore building blocks for "multi-row" local transactions.
> ---
>
> Key: HBASE-5229
> URL: https://issues.apache.org/jira/browse/HBASE-5229
> Project: HBase
>  Issue Type: New Feature
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 5229-endpoint.txt, 5229-multiRow-v2.txt, 
> 5229-multiRow.txt, 5229-seekto-v2.txt, 5229-seekto.txt, 5229.txt
>
>
> HBase should provide basic building blocks for multi-row local transactions. 
> Local means that we do this by co-locating the data. Global (cross region) 
> transactions are not discussed here.
> After a bit of discussion two solutions have emerged:
> 1. Keep the row-key for determining grouping and location and allow efficient 
> intra-row scanning. A client application would then model tables as 
> HBase-rows.
> 2. Define a prefix-length in HTableDescriptor that defines a grouping of 
> rows. Regions will then never be split inside a grouping prefix.
> #1 is true to the current storage paradigm of HBase.
> #2 is true to the current client side API.
> I will explore these two with sample patches here.
> 
> Was:
> As discussed (at length) on the dev mailing list with the HBASE-3584 and 
> HBASE-5203 committed, supporting atomic cross row transactions within a 
> region becomes simple.
> I am aware of the hesitation about the usefulness of this feature, but we 
> have to start somewhere.
> Let's use this jira for discussion, I'll attach a patch (with tests) 
> momentarily to make this concrete.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5229) Explore building blocks for "multi-row" local transactions.

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202739#comment-13202739
 ] 

Lars Hofhansl edited comment on HBASE-5229 at 2/7/12 9:05 PM:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4876
---



http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/MultiRowMutationProtocol.java


Please add javadoc.


- Ted


  was (Author: jirapos...@reviews.apache.org):

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4876
---



http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/MultiRowMutationProtocol.java


Please add javadoc.


- Ted


On 2012-02-07 19:41:16, Lars Hofhansl wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3748/
bq.  ---
bq.  
bq.  (Updated 2012-02-07 19:41:16)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This builds on HBASE-3584, HBASE-5203, and HBASE-5304.
bq.  
bq.  Multiple Rows can be locked and applied atomically as long as the 
application ensures that all rows reside in the same Region (by presplitting or 
a custom RegionSplitPolicy).
bq.  At SFDC we can use this to colocate subsets of a tenant's data and allow 
atomic operations over these subsets.
bq.  
bq.  Obviously this is an advanced features and this prominently called out in 
the Javadoc.
bq.  
bq.  
bq.  This addresses bug HBASE-5229.
bq.  https://issues.apache.org/jira/browse/HBASE-5229
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1241350 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 1241350 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/MultiRowMutationEndpoint.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/MultiRowMutationProtocol.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
 1241350 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java
 1241350 
bq.  
bq.  Diff: https://reviews.apache.org/r/3748/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Tests added to TestFromClientSide and TestAtomicOperation
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Lars
bq.  
bq.


  
> Explore building blocks for "multi-row" local transactions.
> ---
>
> Key: HBASE-5229
> URL: https://issues.apache.org/jira/browse/HBASE-5229
> Project: HBase
>  Issue Type: New Feature
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 5229-endpoint.txt, 5229-multiRow-v2.txt, 
> 5229-multiRow.txt, 5229-seekto-v2.txt, 5229-seekto.txt, 5229.txt
>
>
> HBase should provide basic building blocks for multi-row local transactions. 
> Local means that we do this by co-locating the data. Global (cross region) 
> transactions are not discussed here.
> After a bit of discussion two solutions have emerged:
> 1. Keep the row-key for determining grouping and location and allow efficient 
> intra-row scanning. A client application would then model tables as 
> HBase-rows.
> 2. Define a prefix-length in HTableDescriptor that defines a grouping of 
> rows. Regions will then never be split inside a grouping prefix.
> #1 is true to the current storage paradigm of HBase.
> #2 is true to the current client side API.
> I will explore these two with sample patches here.
> 
> Was:
> As discussed (at length) on the dev mailing list with the HBASE-3584 and 
> HBASE-5203 committed, supporting atomic cross row transactions within a 
> region becomes simple.
> I am aware of the hesitation about the usefulness of this feature, but we 
> have to start somewhere.
> Let's use this jira for discussion, I'll attach a patch (with tests) 
> momentari

[jira] [Issue Comment Edited] (HBASE-5229) Explore building blocks for "multi-row" local transactions.

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201556#comment-13201556
 ] 

Lars Hofhansl edited comment on HBASE-5229 at 2/7/12 5:42 AM:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4844
---



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


What if rm contains more than one Mutation ?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


else is not needed considering exception is thrown on line 4170.


- Ted


  was (Author: jirapos...@reviews.apache.org):

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4844
---



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


What if rm contains more than one Mutation ?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


else is not needed considering exception is thrown on line 4170.


- Ted


On 2012-02-06 19:51:58, Lars Hofhansl wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3748/
bq.  ---
bq.  
bq.  (Updated 2012-02-06 19:51:58)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This builds on HBASE-3584, HBASE-5203, and HBASE-5304.
bq.  
bq.  Multiple Rows can be locked and applied atomically as long as the 
application ensures that all rows reside in the same Region (by presplitting or 
a custom RegionSplitPolicy).
bq.  At SFDC we can use this to colocate subsets of a tenant's data and allow 
atomic operations over these subsets.
bq.  
bq.  Obviously this is an advanced features and this prominently called out in 
the Javadoc.
bq.  
bq.  
bq.  This addresses bug HBASE-5229.
bq.  https://issues.apache.org/jira/browse/HBASE-5229
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 1241120 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java
 1241120 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
 1241120 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java
 1241120 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java
 1241120 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java
 1241120 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1241120 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java
 1241120 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java
 1241120 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java
 1241120 
bq.  
bq.  Diff: https://reviews.apache.org/r/3748/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Tests added to TestFromClientSide and TestAtomicOperation
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Lars
bq.  
bq.


  
> Explore building blocks for "multi-row" local transactions.
> ---
>
> Key: HBASE-5229
> URL: https://issues.apache.org/jira/browse/HBASE-5229
> Project: HBase
>  Issue Type: New Feature
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 5229-multiRow-v2.txt, 5229-multiRow.txt, 
> 5229-seekto-v2.txt, 5229-seekto.txt, 5229.txt
>
>
> HBase should provide 

[jira] [Issue Comment Edited] (HBASE-5229) Explore building blocks for "multi-row" local transactions.

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201624#comment-13201624
 ] 

Lars Hofhansl edited comment on HBASE-5229 at 2/7/12 5:42 AM:
--


bq.  On 2012-02-06 20:24:17, Ted Yu wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java,
 line 4152
bq.  > 
bq.  >
bq.  > What if rm contains more than one Mutation ?

Hopefully rm does contain more than one Mutation, otherwise using this API is 
pointless. :)
It is guranteed, though, that all Mutations are for this single row.

Do you see a concern?


bq.  On 2012-02-06 20:24:17, Ted Yu wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java,
 line 4171
bq.  > 
bq.  >
bq.  > else is not needed considering exception is thrown on line 4170.

Right. But this makes the flow clear. Personally I am not a big fan of having 
to look through code and having to piece together the control flow by tracking 
exceptions and return statements.
I don't mind changing it, though.


- Lars


  was (Author: jirapos...@reviews.apache.org):


bq.  On 2012-02-06 20:24:17, Ted Yu wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java,
 line 4152
bq.  > 
bq.  >
bq.  > What if rm contains more than one Mutation ?

Hopefully rm does contain more than one Mutation, otherwise using this API is 
pointless. :)
It is guranteed, though, that all Mutations are for this single row.

Do you see a concern?


bq.  On 2012-02-06 20:24:17, Ted Yu wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java,
 line 4171
bq.  > 
bq.  >
bq.  > else is not needed considering exception is thrown on line 4170.

Right. But this makes the flow clear. Personally I am not a big fan of having 
to look through code and having to piece together the control flow by tracking 
exceptions and return statements.
I don't mind changing it, though.


- Lars


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4844
---


On 2012-02-06 19:51:58, Lars Hofhansl wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3748/
bq.  ---
bq.  
bq.  (Updated 2012-02-06 19:51:58)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This builds on HBASE-3584, HBASE-5203, and HBASE-5304.
bq.  
bq.  Multiple Rows can be locked and applied atomically as long as the 
application ensures that all rows reside in the same Region (by presplitting or 
a custom RegionSplitPolicy).
bq.  At SFDC we can use this to colocate subsets of a tenant's data and allow 
atomic operations over these subsets.
bq.  
bq.  Obviously this is an advanced features and this prominently called out in 
the Javadoc.
bq.  
bq.  
bq.  This addresses bug HBASE-5229.
bq.  https://issues.apache.org/jira/browse/HBASE-5229
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 1241120 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java
 1241120 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
 1241120 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java
 1241120 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java
 1241120 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java
 1241120 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1241120 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java
 1241120 
bq.
http://svn.apache.org/repos/asf/hbas

[jira] [Issue Comment Edited] (HBASE-5229) Explore building blocks for "multi-row" local transactions.

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13200928#comment-13200928
 ] 

Lars Hofhansl edited comment on HBASE-5229 at 2/5/12 9:56 PM:
--


bq.  On 2012-02-05 07:26:08, Jesse Yates wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java,
 line 4161
bq.  > 
bq.  >
bq.  > Suprised this isn't a utility method in HRegion - it seems really 
useful. Maybe worth pulling out for general use.
bq.  
bq.  Lars Hofhansl wrote:
bq.  internalMutate?
bq.  
bq.  Lars Hofhansl wrote:
bq.  Ah misread again. There is a utility method: checkRow, which I should 
indeed use. The first incarnation of this did something different, now checkRow 
should be used.

Remember now. checkRow throws WrongRegionException, which means the client will 
retry. Here I throws DoNotRetryException, because retrying will be wrong.


- Lars


  was (Author: jirapos...@reviews.apache.org):


bq.  On 2012-02-05 07:26:08, Jesse Yates wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java,
 line 4161
bq.  > 
bq.  >
bq.  > Suprised this isn't a utility method in HRegion - it seems really 
useful. Maybe worth pulling out for general use.
bq.  
bq.  Lars Hofhansl wrote:
bq.  internalMutate?
bq.  
bq.  Lars Hofhansl wrote:
bq.  Ah misread again. There is a utility method: checkRow, which I should 
indeed use. The first incarnation of this did something different, now checkRow 
should be used.

Remember now. checkRow throws WrongRegionException, which means the client will 
retry. Here I throws DoNotRetryException, because retrying will be wrong.


- Lars


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4822
---


On 2012-02-03 19:59:55, Lars Hofhansl wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3748/
bq.  ---
bq.  
bq.  (Updated 2012-02-03 19:59:55)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This builds on HBASE-3584, HBASE-5203, and HBASE-5304.
bq.  
bq.  Multiple Rows can be locked and applied atomically as long as the 
application ensures that all rows reside in the same Region (by presplitting or 
a custom RegionSplitPolicy).
bq.  At SFDC we can use this to colocate subsets of a tenant's data and allow 
atomic operations over these subsets.
bq.  
bq.  Obviously this is an advanced features and this prominently called out in 
the Javadoc.
bq.  
bq.  
bq.  This addresses bug HBASE-5229.
bq.  https://issues.apache.org/jira/browse/HBASE-5229
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java
 1239953 
bq.  
bq.  Diff: https://reviews.apache.org/r/3748/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Tests added to TestFromClientSide and TestAtomicOperation
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Lars
bq.  
bq.


  
> Explore building blocks for "multi-row" local transactions.
> --

[jira] [Issue Comment Edited] (HBASE-5229) Explore building blocks for "multi-row" local transactions.

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13200912#comment-13200912
 ] 

Lars Hofhansl edited comment on HBASE-5229 at 2/5/12 9:48 PM:
--


bq.  On 2012-02-03 04:45:54, Ted Yu wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java,
 line 64
bq.  > 
bq.  >
bq.  > Is this method thread-safe ?
bq.  
bq.  Lars Hofhansl wrote:
bq.  Should be. Only uses local state or protected data structures (like 
put, get, etc)

Misread your comment Ted. This method is indeed not thread-safe nor should it 
be. This is a client side object and it up to the application to provide thread 
safety (which is identical to Put/Delete/Append/Increment, and indeed HTable).


- Lars


  was (Author: jirapos...@reviews.apache.org):


bq.  On 2012-02-03 04:45:54, Ted Yu wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java,
 line 64
bq.  > 
bq.  >
bq.  > Is this method thread-safe ?
bq.  
bq.  Lars Hofhansl wrote:
bq.  Should be. Only uses local state or protected data structures (like 
put, get, etc)

Misread your comment Ted. This method is indeed not thread-safe nor should it 
be. This is a client side object and it up to the application to provide thread 
safety (which is identical to Put/Delete/Append/Increment, and indeed HTable).


- Lars


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4788
---


On 2012-02-03 19:59:55, Lars Hofhansl wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3748/
bq.  ---
bq.  
bq.  (Updated 2012-02-03 19:59:55)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This builds on HBASE-3584, HBASE-5203, and HBASE-5304.
bq.  
bq.  Multiple Rows can be locked and applied atomically as long as the 
application ensures that all rows reside in the same Region (by presplitting or 
a custom RegionSplitPolicy).
bq.  At SFDC we can use this to colocate subsets of a tenant's data and allow 
atomic operations over these subsets.
bq.  
bq.  Obviously this is an advanced features and this prominently called out in 
the Javadoc.
bq.  
bq.  
bq.  This addresses bug HBASE-5229.
bq.  https://issues.apache.org/jira/browse/HBASE-5229
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java
 1239953 
bq.  
bq.  Diff: https://reviews.apache.org/r/3748/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Tests added to TestFromClientSide and TestAtomicOperation
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Lars
bq.  
bq.


  
> Explore building blocks for "multi-row" local transactions.
> ---
>
> Key: HBASE-5229
> URL: https://issues.apache.org/jira/browse/HBASE-5229
> Project: HBase
>  Issue Type: New Feature
>  Components: client, regionserver
>Reporter: Lar

[jira] [Issue Comment Edited] (HBASE-5229) Explore building blocks for "multi-row" local transactions.

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13200922#comment-13200922
 ] 

Lars Hofhansl edited comment on HBASE-5229 at 2/5/12 9:49 PM:
--


bq.  On 2012-02-05 07:26:08, Jesse Yates wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java,
 line 4161
bq.  > 
bq.  >
bq.  > Suprised this isn't a utility method in HRegion - it seems really 
useful. Maybe worth pulling out for general use.
bq.  
bq.  Lars Hofhansl wrote:
bq.  internalMutate?

Ah misread again. There is a utility method: checkRow, which I should indeed 
use. The first incarnation of this did something different, now checkRow should 
be used.


- Lars


  was (Author: jirapos...@reviews.apache.org):


bq.  On 2012-02-05 07:26:08, Jesse Yates wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java,
 line 4161
bq.  > 
bq.  >
bq.  > Suprised this isn't a utility method in HRegion - it seems really 
useful. Maybe worth pulling out for general use.
bq.  
bq.  Lars Hofhansl wrote:
bq.  internalMutate?

Ah misread again. There is a utility method: checkRow, which I should indeed 
use. The first incarnation of this did something different, now checkRow should 
be used.


- Lars


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4822
---


On 2012-02-03 19:59:55, Lars Hofhansl wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3748/
bq.  ---
bq.  
bq.  (Updated 2012-02-03 19:59:55)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This builds on HBASE-3584, HBASE-5203, and HBASE-5304.
bq.  
bq.  Multiple Rows can be locked and applied atomically as long as the 
application ensures that all rows reside in the same Region (by presplitting or 
a custom RegionSplitPolicy).
bq.  At SFDC we can use this to colocate subsets of a tenant's data and allow 
atomic operations over these subsets.
bq.  
bq.  Obviously this is an advanced features and this prominently called out in 
the Javadoc.
bq.  
bq.  
bq.  This addresses bug HBASE-5229.
bq.  https://issues.apache.org/jira/browse/HBASE-5229
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java
 1239953 
bq.  
bq.  Diff: https://reviews.apache.org/r/3748/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Tests added to TestFromClientSide and TestAtomicOperation
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Lars
bq.  
bq.


  
> Explore building blocks for "multi-row" local transactions.
> ---
>
> Key: HBASE-5229
> URL: https://issues.apache.org/jira/browse/HBASE-5229
> Project: HBase
>  Issue Type: New Feature
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> At

[jira] [Issue Comment Edited] (HBASE-5229) Explore building blocks for "multi-row" local transactions.

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13200104#comment-13200104
 ] 

Lars Hofhansl edited comment on HBASE-5229 at 2/3/12 10:29 PM:
---


bq.  On 2012-02-03 05:01:34, Jimmy Xiang wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java,
 line 67
bq.  > 
bq.  >
bq.  > Should we check they are for different rows here?
bq.  
bq.  Lars Hofhansl wrote:
bq.  It is still valid to pass the same rows here.
bq.  
bq.  Jimmy Xiang wrote:
bq.  That's right.  Should the server side share the same function to do 
MultiRowMutation and RowMutation?  Looks the server side doesn't have much 
difference for mutateRow() and mutateRows()

Oh... I misunderstood what you were saying.
On the server it is mostly the same code. On the client I would like to keep 
them separate features as RowMutation (hopefully) is a pretty standard feature, 
whereas MultiRowMutation is pretty advanced (need to manage your Splits, etc) 
and really treads new HBase ground.


- Lars


  was (Author: jirapos...@reviews.apache.org):


bq.  On 2012-02-03 05:01:34, Jimmy Xiang wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java,
 line 67
bq.  > 
bq.  >
bq.  > Should we check they are for different rows here?
bq.  
bq.  Lars Hofhansl wrote:
bq.  It is still valid to pass the same rows here.
bq.  
bq.  Jimmy Xiang wrote:
bq.  That's right.  Should the server side share the same function to do 
MultiRowMutation and RowMutation?  Looks the server side doesn't have much 
difference for mutateRow() and mutateRows()

Oh... I misunderstood what you were saying.
On the server it is mostly the same code. On the client I would like to keep 
them separate features as RowMutation (hopefully) is a pretty standard feature, 
whereas MultiRowMutation is pretty advanced (need to manage your Splits, etc) 
and really treads new HBase ground.


- Lars


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4789
---


On 2012-02-03 19:59:55, Lars Hofhansl wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3748/
bq.  ---
bq.  
bq.  (Updated 2012-02-03 19:59:55)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This builds on HBASE-3584, HBASE-5203, and HBASE-5304.
bq.  
bq.  Multiple Rows can be locked and applied atomically as long as the 
application ensures that all rows reside in the same Region (by presplitting or 
a custom RegionSplitPolicy).
bq.  At SFDC we can use this to colocate subsets of a tenant's data and allow 
atomic operations over these subsets.
bq.  
bq.  Obviously this is an advanced features and this prominently called out in 
the Javadoc.
bq.  
bq.  
bq.  This addresses bug HBASE-5229.
bq.  https://issues.apache.org/jira/browse/HBASE-5229
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java
 1239953 
bq.  
bq.

[jira] [Issue Comment Edited] (HBASE-5229) Explore building blocks for "multi-row" local transactions.

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13199933#comment-13199933
 ] 

Lars Hofhansl edited comment on HBASE-5229 at 2/3/12 10:28 PM:
---


bq.  On 2012-02-03 05:07:24, Ted Yu wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java,
 line 3160
bq.  > 
bq.  >
bq.  > My point was we shouldn't throw exception in this case.
bq.  > MultiRowMutation should be delivered to the correct region.
bq.  > 
bq.  > I think we agree on the above.

Oh... MultiRowMutation does not know anything about regions. So we have the use 
the one passed.


- Lars


  was (Author: jirapos...@reviews.apache.org):


bq.  On 2012-02-03 05:07:24, Ted Yu wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java,
 line 3160
bq.  > 
bq.  >
bq.  > My point was we shouldn't throw exception in this case.
bq.  > MultiRowMutation should be delivered to the correct region.
bq.  > 
bq.  > I think we agree on the above.

Oh... MultiRowMutation does not know anything about regions. So we have the use 
the one passed.


- Lars


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4791
---


On 2012-02-03 01:29:58, Lars Hofhansl wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3748/
bq.  ---
bq.  
bq.  (Updated 2012-02-03 01:29:58)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This builds on HBASE-3584, HBASE-5203, and HBASE-5304.
bq.  
bq.  Multiple Rows can be locked and applied atomically as long as the 
application ensures that all rows reside in the same Region (by presplitting or 
a custom RegionSplitPolicy).
bq.  At SFDC we can use this to colocate subsets of a tenant's data and allow 
atomic operations over these subsets.
bq.  
bq.  Obviously this is an advanced features and this prominently called out in 
the Javadoc.
bq.  
bq.  
bq.  This addresses bug HBASE-5229.
bq.  https://issues.apache.org/jira/browse/HBASE-5229
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RowMutation.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java
 1239953 
bq.  
bq.  Diff: https://reviews.apache.org/r/3748/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Tests added to TestFromClientSide and TestAtomicOperation
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Lars
bq.  
bq.


  
> Explore building blocks for "multi-row" local transactions.
> ---
>
> Key: HBASE-5229
> URL: https://issues.apache.org/jira/browse/HBASE-5229
> Project: HBase
>  Issue Type: New Feature
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> 

[jira] [Issue Comment Edited] (HBASE-5229) Explore building blocks for "multi-row" local transactions.

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13199520#comment-13199520
 ] 

Lars Hofhansl edited comment on HBASE-5229 at 2/3/12 10:29 PM:
---

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4789
---



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java


Should we check they are for different rows here?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RowMutation.java


This is called by default.  No need to call it explicitly.



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RowMutation.java


This is called by default.  No need to call it explicitly.


- Jimmy


  was (Author: jirapos...@reviews.apache.org):

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4789
---



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java


Should we check they are for different rows here?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RowMutation.java


This is called by default.  No need to call it explicitly.



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RowMutation.java


This is called by default.  No need to call it explicitly.


- Jimmy


On 2012-02-03 01:29:58, Lars Hofhansl wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3748/
bq.  ---
bq.  
bq.  (Updated 2012-02-03 01:29:58)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This builds on HBASE-3584, HBASE-5203, and HBASE-5304.
bq.  
bq.  Multiple Rows can be locked and applied atomically as long as the 
application ensures that all rows reside in the same Region (by presplitting or 
a custom RegionSplitPolicy).
bq.  At SFDC we can use this to colocate subsets of a tenant's data and allow 
atomic operations over these subsets.
bq.  
bq.  Obviously this is an advanced features and this prominently called out in 
the Javadoc.
bq.  
bq.  
bq.  This addresses bug HBASE-5229.
bq.  https://issues.apache.org/jira/browse/HBASE-5229
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RowMutation.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java
 1239953 
bq.  
bq.  Diff: https://reviews.apache.org/r/3748/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Tests added to TestFromClientSide and TestAtomicOperation
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Lars
bq.  
bq.


  
> Explore 

[jira] [Issue Comment Edited] (HBASE-5229) Explore building blocks for "multi-row" local transactions.

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13199934#comment-13199934
 ] 

Lars Hofhansl edited comment on HBASE-5229 at 2/3/12 10:28 PM:
---


bq.  On 2012-02-03 04:45:54, Ted Yu wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RowMutation.java,
 line 39
bq.  > 
bq.  >
bq.  > From its name, RowMutation seems to refer to single row. It is a 
little confusing RowMutation extends MultiRowMutation.
bq.  
bq.  Lars Hofhansl wrote:
bq.  Yeah. Maybe I'll have a common super class instead.

On 2nd thought. RowMutation *is* specialisation of MultiRowMutation, one that 
only allows a single row.


- Lars




  was (Author: jirapos...@reviews.apache.org):


bq.  On 2012-02-03 04:45:54, Ted Yu wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RowMutation.java,
 line 39
bq.  > 
bq.  >
bq.  > From its name, RowMutation seems to refer to single row. It is a 
little confusing RowMutation extends MultiRowMutation.
bq.  
bq.  Lars Hofhansl wrote:
bq.  Yeah. Maybe I'll have a common super class instead.

On 2nd thought. RowMutation *is* specialisation of MultiRowMutation, one that 
only allows a single row.


- Lars


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4788
---


On 2012-02-03 01:29:58, Lars Hofhansl wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3748/
bq.  ---
bq.  
bq.  (Updated 2012-02-03 01:29:58)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This builds on HBASE-3584, HBASE-5203, and HBASE-5304.
bq.  
bq.  Multiple Rows can be locked and applied atomically as long as the 
application ensures that all rows reside in the same Region (by presplitting or 
a custom RegionSplitPolicy).
bq.  At SFDC we can use this to colocate subsets of a tenant's data and allow 
atomic operations over these subsets.
bq.  
bq.  Obviously this is an advanced features and this prominently called out in 
the Javadoc.
bq.  
bq.  
bq.  This addresses bug HBASE-5229.
bq.  https://issues.apache.org/jira/browse/HBASE-5229
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RowMutation.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java
 1239953 
bq.  
bq.  Diff: https://reviews.apache.org/r/3748/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Tests added to TestFromClientSide and TestAtomicOperation
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Lars
bq.  
bq.


  
> Explore building blocks for "multi-row" local transactions.
> ---
>
> Key: HBASE-5229
> URL: https://issues.apache.org/jira/browse/HBASE-5229
> Project: HBase
>  Issue Type: New Feature
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assign

[jira] [Issue Comment Edited] (HBASE-5229) Explore building blocks for "multi-row" local transactions.

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13199932#comment-13199932
 ] 

Lars Hofhansl edited comment on HBASE-5229 at 2/3/12 10:28 PM:
---


bq.  On 2012-02-03 05:01:34, Jimmy Xiang wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java,
 line 67
bq.  > 
bq.  >
bq.  > Should we check they are for different rows here?

It is still valid to pass the same rows here.


bq.  On 2012-02-03 05:01:34, Jimmy Xiang wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RowMutation.java,
 line 46
bq.  > 
bq.  >
bq.  > This is called by default.  No need to call it explicitly.

rights... will fix.


- Lars


  was (Author: jirapos...@reviews.apache.org):


bq.  On 2012-02-03 05:01:34, Jimmy Xiang wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java,
 line 67
bq.  > 
bq.  >
bq.  > Should we check they are for different rows here?

It is still valid to pass the same rows here.


bq.  On 2012-02-03 05:01:34, Jimmy Xiang wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RowMutation.java,
 line 46
bq.  > 
bq.  >
bq.  > This is called by default.  No need to call it explicitly.

rights... will fix.


- Lars


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4789
---


On 2012-02-03 01:29:58, Lars Hofhansl wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3748/
bq.  ---
bq.  
bq.  (Updated 2012-02-03 01:29:58)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This builds on HBASE-3584, HBASE-5203, and HBASE-5304.
bq.  
bq.  Multiple Rows can be locked and applied atomically as long as the 
application ensures that all rows reside in the same Region (by presplitting or 
a custom RegionSplitPolicy).
bq.  At SFDC we can use this to colocate subsets of a tenant's data and allow 
atomic operations over these subsets.
bq.  
bq.  Obviously this is an advanced features and this prominently called out in 
the Javadoc.
bq.  
bq.  
bq.  This addresses bug HBASE-5229.
bq.  https://issues.apache.org/jira/browse/HBASE-5229
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RowMutation.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java
 1239953 
bq.  
bq.  Diff: https://reviews.apache.org/r/3748/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Tests added to TestFromClientSide and TestAtomicOperation
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Lars
bq.  
bq.


  
> Explore building blocks for "multi-row" local transactions.
> ---

[jira] [Issue Comment Edited] (HBASE-5229) Explore building blocks for "multi-row" local transactions.

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13199523#comment-13199523
 ] 

Lars Hofhansl edited comment on HBASE-5229 at 2/3/12 10:28 PM:
---

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4791
---



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java


My point was we shouldn't throw exception in this case.
MultiRowMutation should be delivered to the correct region.

I think we agree on the above.


- Ted


  was (Author: jirapos...@reviews.apache.org):

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4791
---



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java


My point was we shouldn't throw exception in this case.
MultiRowMutation should be delivered to the correct region.

I think we agree on the above.


- Ted


On 2012-02-03 01:29:58, Lars Hofhansl wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3748/
bq.  ---
bq.  
bq.  (Updated 2012-02-03 01:29:58)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This builds on HBASE-3584, HBASE-5203, and HBASE-5304.
bq.  
bq.  Multiple Rows can be locked and applied atomically as long as the 
application ensures that all rows reside in the same Region (by presplitting or 
a custom RegionSplitPolicy).
bq.  At SFDC we can use this to colocate subsets of a tenant's data and allow 
atomic operations over these subsets.
bq.  
bq.  Obviously this is an advanced features and this prominently called out in 
the Javadoc.
bq.  
bq.  
bq.  This addresses bug HBASE-5229.
bq.  https://issues.apache.org/jira/browse/HBASE-5229
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RowMutation.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java
 1239953 
bq.  
bq.  Diff: https://reviews.apache.org/r/3748/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Tests added to TestFromClientSide and TestAtomicOperation
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Lars
bq.  
bq.


  
> Explore building blocks for "multi-row" local transactions.
> ---
>
> Key: HBASE-5229
> URL: https://issues.apache.org/jira/browse/HBASE-5229
> Project: HBase
>  Issue Type: New Feature
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 5229-multiRow.txt, 5229-seekto-v2.txt, 5229-seekto.txt, 
> 5229.txt
>
>
> HBase should provide basic building blocks for multi-row local transactions. 
> Local means that we do this by co-locating the data. Glob

[jira] [Issue Comment Edited] (HBASE-5229) Explore building blocks for "multi-row" local transactions.

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13199940#comment-13199940
 ] 

Lars Hofhansl edited comment on HBASE-5229 at 2/3/12 10:27 PM:
---


bq.  On 2012-02-03 04:45:54, Ted Yu wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RowMutation.java,
 line 96
bq.  > 
bq.  >
bq.  > version would be read / written twice, right ?
bq.  
bq.  Lars Hofhansl wrote:
bq.  Yes. Need to fix that.

And actually this is what I had in mind too. Both MultiRowMutation and 
RowMution write their version.
MultiRowMutation knowns to serialize mutations, that process needs to be 
versioned.
RowMutation adds some fields on top of that, those also need to be versioned.

Maybe the two should be entirely independent (at the expense of a few 
duplicated lines of code) so that they can evolve independently.


- Lars




  was (Author: jirapos...@reviews.apache.org):


bq.  On 2012-02-03 04:45:54, Ted Yu wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RowMutation.java,
 line 96
bq.  > 
bq.  >
bq.  > version would be read / written twice, right ?
bq.  
bq.  Lars Hofhansl wrote:
bq.  Yes. Need to fix that.

And actually this is what I had in mind too. Both MultiRowMutation and 
RowMution write their version.
MultiRowMutation knowns to serialize mutations, that process needs to be 
versioned.
RowMutation adds some fields on top of that, those also need to be versioned.

Maybe the two should be entirely independent (at the expense of a few 
duplicated lines of code) so that they can evolve independently.


- Lars


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4788
---


On 2012-02-03 01:29:58, Lars Hofhansl wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3748/
bq.  ---
bq.  
bq.  (Updated 2012-02-03 01:29:58)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This builds on HBASE-3584, HBASE-5203, and HBASE-5304.
bq.  
bq.  Multiple Rows can be locked and applied atomically as long as the 
application ensures that all rows reside in the same Region (by presplitting or 
a custom RegionSplitPolicy).
bq.  At SFDC we can use this to colocate subsets of a tenant's data and allow 
atomic operations over these subsets.
bq.  
bq.  Obviously this is an advanced features and this prominently called out in 
the Javadoc.
bq.  
bq.  
bq.  This addresses bug HBASE-5229.
bq.  https://issues.apache.org/jira/browse/HBASE-5229
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RowMutation.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java
 1239953 
bq.  
bq.  Diff: https://reviews.apache.org/r/3748/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Tests added to TestFromClientSide and TestAtomicOperation
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Lars
bq.  
bq.


  
> Expl

[jira] [Issue Comment Edited] (HBASE-5229) Explore building blocks for "multi-row" local transactions.

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13199953#comment-13199953
 ] 

Lars Hofhansl edited comment on HBASE-5229 at 2/3/12 10:27 PM:
---


bq.  On 2012-02-03 05:01:34, Jimmy Xiang wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java,
 line 67
bq.  > 
bq.  >
bq.  > Should we check they are for different rows here?
bq.  
bq.  Lars Hofhansl wrote:
bq.  It is still valid to pass the same rows here.

That's right.  Should the server side share the same function to do 
MultiRowMutation and RowMutation?  Looks the server side doesn't have much 
difference for mutateRow() and mutateRows()


- Jimmy




  was (Author: jirapos...@reviews.apache.org):


bq.  On 2012-02-03 05:01:34, Jimmy Xiang wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java,
 line 67
bq.  > 
bq.  >
bq.  > Should we check they are for different rows here?
bq.  
bq.  Lars Hofhansl wrote:
bq.  It is still valid to pass the same rows here.

That's right.  Should the server side share the same function to do 
MultiRowMutation and RowMutation?  Looks the server side doesn't have much 
difference for mutateRow() and mutateRows()


- Jimmy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4789
---


On 2012-02-03 01:29:58, Lars Hofhansl wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3748/
bq.  ---
bq.  
bq.  (Updated 2012-02-03 01:29:58)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This builds on HBASE-3584, HBASE-5203, and HBASE-5304.
bq.  
bq.  Multiple Rows can be locked and applied atomically as long as the 
application ensures that all rows reside in the same Region (by presplitting or 
a custom RegionSplitPolicy).
bq.  At SFDC we can use this to colocate subsets of a tenant's data and allow 
atomic operations over these subsets.
bq.  
bq.  Obviously this is an advanced features and this prominently called out in 
the Javadoc.
bq.  
bq.  
bq.  This addresses bug HBASE-5229.
bq.  https://issues.apache.org/jira/browse/HBASE-5229
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RowMutation.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java
 1239953 
bq.  
bq.  Diff: https://reviews.apache.org/r/3748/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Tests added to TestFromClientSide and TestAtomicOperation
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Lars
bq.  
bq.


  
> Explore building blocks for "multi-row" local transactions.
> ---
>
> Key: HBASE-5229
> URL: https://issues.apache.org/jira/browse/HBASE-5229
> Project: HBase
>  Issue Type: New Feature
>  Components: client, regionserver
>Reporte

[jira] [Issue Comment Edited] (HBASE-5304) Pluggable split key policy

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196644#comment-13196644
 ] 

Lars Hofhansl edited comment on HBASE-5304 at 1/31/12 2:14 AM:
---

Jesse Yates and I are working pluggable split key policies. Will attach a patch 
momentarily.

  was (Author: lhofhansl):
Jesse Yates and I are working pluggable split key policy. Will attach a 
patch momentarily.
  
> Pluggable split key policy
> --
>
> Key: HBASE-5304
> URL: https://issues.apache.org/jira/browse/HBASE-5304
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 5304.txt
>
>
> We need a way to specify custom policies to determine split keys.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5293) Purge hfile v1 from code base

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195358#comment-13195358
 ] 

Lars Hofhansl edited comment on HBASE-5293 at 1/28/12 2:30 AM:
---

Let's make this issue about adding a script to verify that all HFiles have been 
compacted. We can add that script to 0.94, to be ready for HFile V1 removal in 
0.96.

  was (Author: lhofhansl):
Let's make this issue about adding a script to verify that HFiles have been 
compacted. We can add that script to 0.94, to be ready of 0.96 migration.
  
> Purge hfile v1 from code base
> -
>
> Key: HBASE-5293
> URL: https://issues.apache.org/jira/browse/HBASE-5293
> Project: HBase
>  Issue Type: Task
>Reporter: stack
>
> Remove all hfile v1 references from code base.
> If we do this though, as Matt Corgan suggests up on mailing list, we will 
> need to make sure all hfile v1s in an hbase.rootdir have been compacted out 
> of existence.  We'll probably need to bump the hbase.version to indicate the 
> check for hfile v1s has been run.  A migration script will need to be run 
> that checks the hbase.rootdir for hfile v1s and runs a major compaction if 
> any found.
> I've not put a version on this issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5268) Add delete column prefix delete marker

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192255#comment-13192255
 ] 

Lars Hofhansl edited comment on HBASE-5268 at 1/24/12 4:27 PM:
---

I'll work on a fully correct patch today, and then we can decide whether it is 
worth the extra complexity. I find this first patch compelling, because (1) is 
very small and easy to verify and (2) it keeps column prefix delete markers 
sorted (by timestamp) with the KVs they affect.
In the bigger patch, neither will be true anymore.


  was (Author: lhofhansl):
I'll work on a fully correct patch today, and then we can decide whether it 
is worth the extra complexity. I find this first patch compelling, because (1) 
is very small and easy to verify and (2) it keeps column prefix delete markers 
sorted (by timestamp) with the KVs they affect.
In the bigger patch, neither will be try anymore.

  
> Add delete column prefix delete marker
> --
>
> Key: HBASE-5268
> URL: https://issues.apache.org/jira/browse/HBASE-5268
> Project: HBase
>  Issue Type: Improvement
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 5268.txt
>
>
> This is another part missing in the "wide row challenge".
> Currently entire families of a row can be deleted or individual columns or 
> versions.
> There is no facility to mark multiple columns for deletion by column prefix.
> Turns out that be achieve with very little code (it's possible that I missed 
> some of the new delete bloom filter code, so please review this thoroughly). 
> I'll attach a patch soon, just working on some tests now.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5268) Add delete column prefix delete marker

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192016#comment-13192016
 ] 

Lars Hofhansl edited comment on HBASE-5268 at 1/24/12 8:43 AM:
---

When I muck with KeyValue.KeyComparator to always sort prefix delete marker 
before all columns with the same prefix regardless of TS, then I see the 
opposite scenario that in the setup above the column delete marker is ignored 
and hence the newly put version of column 123 is visible again. This is the 
because the column marker is considered less specific and hence won't override 
the column prefix marker, even though it is newer. Arghh :)

So ScanDeleteTracker would need to be completely refactored in order to keep 
prefix markers separate from column and version markers.
The question is: Is that worth the risk, added code entropy, and effort, or can 
we document that prefix delete markers should not be used together with column 
delete markers (where the column markers column is identical to the prefix 
marker's column prefix, which does not make much sense anyway)?


  was (Author: lhofhansl):
When I muck with KeyValue.KeyComparator to always sort prefix delete marker 
before all columns with the same prefix regardless of TS, then I see the 
opposite scenario that in the setup above the column delete marker is ignored 
and hence the newly put version of column 123 is visible again. This is the 
because the column marker is considered less specific and hence won't override 
the column prefix marker. Arghh :)

So ScanDeleteTracker would need to be completely refactored in order to keep 
prefix marker separate from column and version markers.
The question is: Is that worth the risk, added code entropy, and effort, or can 
we document that prefix delete markers should not be used together with column 
delete markers (where the column markers column is identical to the prefix 
marker's column prefix, which does not make much sense anyway)?

  
> Add delete column prefix delete marker
> --
>
> Key: HBASE-5268
> URL: https://issues.apache.org/jira/browse/HBASE-5268
> Project: HBase
>  Issue Type: Improvement
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 5268.txt
>
>
> This is another part missing in the "wide row challenge".
> Currently entire families of a row can be deleted or individual columns or 
> versions.
> There is no facility to mark multiple columns for deletion by column prefix.
> Turns out that be achieve with very little code (it's possible that I missed 
> some of the new delete bloom filter code, so please review this thoroughly). 
> I'll attach a patch soon, just working on some tests now.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5268) Add delete column prefix delete marker

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191958#comment-13191958
 ] 

Lars Hofhansl edited comment on HBASE-5268 at 1/24/12 8:44 AM:
---

@Stack... Here's an example.
Say you have columns X100, X101, X102, ..., X199, and X200, X201, X202, ..., 
X299, X300, ..., in a row R, all in the same column family.
(I'd have this if I modeled rows inside HBase rows by using a prefix to 
identify the inner rows.)

Now I want to be able to delete all columns that start with X1. Without a 
prefix delete marker I'd have to place 100 column delete markers. With a prefix 
marker I can just place one marker for the X1 prefix.
If the prefix is empty, this would degenerate into the same behavior as a 
family marker (albeit less efficient).


  was (Author: lhofhansl):
@Stack... Here's an example.
Say you can columns X100, X101, X102, ..., X199, and X200, X201, X202, ..., 
X299, X300, ..., in a row R, all in the same column family.
(I'd have this if I modeled rows inside HBase rows by using a prefix to 
identify the inner rows.)

Now I want to be able to delete all columns that start with X1. Without a 
prefix delete marker I'd have to place 100 column delete markers. With a prefix 
marker I can just place one marker for the X1 prefix.
If the prefix is empty, this would degenerate into the same behavior as a 
family marker (albeit less efficient).

  
> Add delete column prefix delete marker
> --
>
> Key: HBASE-5268
> URL: https://issues.apache.org/jira/browse/HBASE-5268
> Project: HBase
>  Issue Type: Improvement
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 5268.txt
>
>
> This is another part missing in the "wide row challenge".
> Currently entire families of a row can be deleted or individual columns or 
> versions.
> There is no facility to mark multiple columns for deletion by column prefix.
> Turns out that be achieve with very little code (it's possible that I missed 
> some of the new delete bloom filter code, so please review this thoroughly). 
> I'll attach a patch soon, just working on some tests now.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5268) Add delete column prefix delete marker

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191958#comment-13191958
 ] 

Lars Hofhansl edited comment on HBASE-5268 at 1/24/12 8:42 AM:
---

@Stack... Here's an example.
Say you can columns X100, X101, X102, ..., X199, and X200, X201, X202, ..., 
X299, X300, ..., in a row R, all in the same column family.
(I'd have this if I modeled rows inside HBase rows by using a prefix to 
identify the inner rows.)

Now I want to be able to delete all columns that start with X1. Without a 
prefix delete marker I'd have to place 100 column delete markers. With a prefix 
marker I can just place one marker for the X1 prefix.
If the prefix is empty, this would degenerate into the same behavior as a 
family marker (albeit less efficient).


  was (Author: lhofhansl):
@Stack... Here's an example.
Say you can columns X100, X101, X102, ..., X199, and X200, X201, X202, ..., 
X299, X300, ..., in a row R, all in the same column family.
(I'd have this if I modeled rows inside HBase by using a prefix to identify the 
inner rows.)

Now I want to be able to delete all columns that start with X1. Without a 
prefix delete marker I'd have to place 100 column delete markers. With a prefix 
marker I can just place one marker for the X1 prefix.
If the prefix is empty, this would degenerate into the same behavior as a 
family marker (albeit less efficient).

  
> Add delete column prefix delete marker
> --
>
> Key: HBASE-5268
> URL: https://issues.apache.org/jira/browse/HBASE-5268
> Project: HBase
>  Issue Type: Improvement
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 5268.txt
>
>
> This is another part missing in the "wide row challenge".
> Currently entire families of a row can be deleted or individual columns or 
> versions.
> There is no facility to mark multiple columns for deletion by column prefix.
> Turns out that be achieve with very little code (it's possible that I missed 
> some of the new delete bloom filter code, so please review this thoroughly). 
> I'll attach a patch soon, just working on some tests now.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5268) Add delete column prefix delete marker

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191997#comment-13191997
 ] 

Lars Hofhansl edited comment on HBASE-5268 at 1/24/12 8:14 AM:
---

Here's a patch. The bulk is testing.

During testing with delete marker types I found one strange scenario:
Say you
# put columns 123, 1234, 12345
# then delete with prefix 123
# then put column 123 again
# now delete 123 with a normal column marker

Now what happens is that the ScanDeleteTracker sees the normal column delete 
marker first, then it will see the new put for column 123. Now it will conclude 
that it is done with all versions of column 123, and thus seeks ahead to the 
next column. During that process the prefix marker with prefix 123 is also 
skipped. And hence 1234 and 12345 are no longer marked as deleted.

This only happens in exactly this scenario.

I cannot fix this without de-optimizing column delete markers or adding 
complicated logic to sort prefix delete marker always before all prefixes they 
affect regardless of the timestamp.

I added this scenario as a unit test.

  was (Author: lhofhansl):
Here's a patch. The bulk is testing.

During testing with deleted marker types I found one strange scenario:
Say you
# put columns 123, 1234, 12345
# then delete with prefix 123
# then put column 123 again
# now delete 123 with a normal column marker

Now what happens is that the ScanDeleteTracker sees the normal column delete 
marker first, then it will see the new put for column 123. Now it will conclude 
that it is done with all versions of column 123, and thus seek ahead to the 
next column. During that process the prefix marker with prefix 123 is also 
skipped. And hence 1234 and 12345 are no longer marked as deleted.

This only happens in exactly this scenario.

I cannot fix this without de-optimizing column delete markers or adding 
complicated logic to sort prefix delete marker always before all prefixes they 
affect regardless of the timestamp.

I added this scenario as a unit test.
  
> Add delete column prefix delete marker
> --
>
> Key: HBASE-5268
> URL: https://issues.apache.org/jira/browse/HBASE-5268
> Project: HBase
>  Issue Type: Improvement
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 5268.txt
>
>
> This is another part missing in the "wide row challenge".
> Currently entire families of a row can be deleted or individual columns or 
> versions.
> There is no facility to mark multiple columns for deletion by column prefix.
> Turns out that be achieve with very little code (it's possible that I missed 
> some of the new delete bloom filter code, so please review this thoroughly). 
> I'll attach a patch soon, just working on some tests now.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5229) Explore building blocks for "multi-row" local transactions.

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190796#comment-13190796
 ] 

Lars Hofhansl edited comment on HBASE-5229 at 1/22/12 9:59 PM:
---

Here's a patch that works. Regions are correctly addressed using the row key. 
Intra-row scans are limited to a single row.
The region scanner will correctly skip all stores for families before the 
seekTo KV.
The first matching store will use the seekTo KV, the others will seek to the 
beginning of the row (since their family sorts after the seekTo key).

The model possible here would be that a "transactional" table ends up being a 
row in HBase (i.e. the row-key is the tablename), and a prefix of the columns 
defines "rows" inside that table, the rest of the columns defines the actual 
"columns" of that row.

With this patch that is possible (set seekTo in Scan.java to seek inside a row, 
and enable batching).

This patch provides no way to set a stop point inside a row. The client would 
need to set batching to reasonable amount (to avoid too many roundtrips and at 
the same not to return too many unnecessary KVs).
Also with a stop point, we could prune all stores whose family is past the stop 
point (just like this patch prunes all stores with families before the stop 
point).
Because RegionScannerImpl and StoreScaner are inherently the row based the 
refactoring would be non-trivial and risky.

I will now explore what it would take to define a grouping prefix in 
HTableDefinition.

  was (Author: lhofhansl):
Here's a patch that works. Regions are correctly addressed using the row 
key.
The region scanner will correctly skip all stores for families before the 
seekTo KV.
The first matching store will use the seekTo KV, the others will seek to the 
beginning of the row (since their family sorts after the seekTo key).

The model possible here would be that a "transactional" table ends up being a 
row in HBase (i.e. the row-key is the tablename), and a prefix of the columns 
defines "rows" inside that table, the rest of the columns defines the actual 
"columns" of that row.

With this patch that is possible (set seekTo in Scan.java to seek inside a row, 
and enable batching).

This patch provides no way to set a stop point inside a row. The client would 
need to set batching to reasonable amount (to avoid too many roundtrips and at 
the same not to return too many unnecessary KVs).
Also with a stop point, we could prune all stores whose family is past the stop 
point (just like this patch prunes all stores with families before the stop 
point).
Because RegionScannerImpl and StoreScaner are inherently the row based the 
refactoring would be non-trivial and risky.

I will now explore what it would take to define a grouping prefix in 
HTableDefinition.
  
> Explore building blocks for "multi-row" local transactions.
> ---
>
> Key: HBASE-5229
> URL: https://issues.apache.org/jira/browse/HBASE-5229
> Project: HBase
>  Issue Type: New Feature
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 5229-seekto-v2.txt, 5229-seekto.txt, 5229.txt
>
>
> HBase should provide basic building blocks for multi-row local transactions. 
> Local means that we do this by co-locating the data. Global (cross region) 
> transactions are not discussed here.
> After a bit of discussion two solutions have emerged:
> 1. Keep the row-key for determining grouping and location and allow efficient 
> intra-row scanning. A client application would then model tables as 
> HBase-rows.
> 2. Define a prefix-length in HTableDescriptor that defines a grouping of 
> rows. Regions will then never be split inside a grouping prefix.
> #1 is true to the current storage paradigm of HBase.
> #2 is true to the current client side API.
> I will explore these two with sample patches here.
> 
> Was:
> As discussed (at length) on the dev mailing list with the HBASE-3584 and 
> HBASE-5203 committed, supporting atomic cross row transactions within a 
> region becomes simple.
> I am aware of the hesitation about the usefulness of this feature, but we 
> have to start somewhere.
> Let's use this jira for discussion, I'll attach a patch (with tests) 
> momentarily to make this concrete.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-2600) Change how we do meta tables; from tablename+STARTROW+randomid to instead, tablename+ENDROW+randomid

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188805#comment-13188805
 ] 

Lars Hofhansl edited comment on HBASE-2600 at 1/18/12 11:37 PM:


Oh? Did not know that.
If that's the case, I agree we should remove that test and create a similar 
test for this issue.

  was (Author: lhofhansl):
Oh? Did now know that.
If that's the case, I agree we should remove that test and create a similar 
test for this issue.
  
> Change how we do meta tables; from tablename+STARTROW+randomid to instead, 
> tablename+ENDROW+randomid
> 
>
> Key: HBASE-2600
> URL: https://issues.apache.org/jira/browse/HBASE-2600
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: Alex Newman
> Attachments: 
> 0001-Changed-regioninfo-format-to-use-endKey-instead-of-s.patch, 
> 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v2.patch, 
> 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v4.patch, 
> 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v6.patch, 
> 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v7.2.patch, 
> 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v8, 
> 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v8.1, 
> 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v9.patch, 
> 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen.patch, 
> 2600-trunk-01-17.txt, jenkins.pdf
>
>
> This is an idea that Ryan and I have been kicking around on and off for a 
> while now.
> If regionnames were made of tablename+endrow instead of tablename+startrow, 
> then in the metatables, doing a search for the region that contains the 
> wanted row, we'd just have to open a scanner using passed row and the first 
> row found by the scan would be that of the region we need (If offlined 
> parent, we'd have to scan to the next row).
> If we redid the meta tables in this format, we'd be using an access that is 
> natural to hbase, a scan as opposed to the perverse, expensive 
> getClosestRowBefore we currently have that has to walk backward in meta 
> finding a containing region.
> This issue is about changing the way we name regions.
> If we were using scans, prewarming client cache would be near costless (as 
> opposed to what we'll currently have to do which is first a 
> getClosestRowBefore and then a scan from the closestrowbefore forward).
> Converting to the new method, we'd have to run a migration on startup 
> changing the content in meta.
> Up to this, the randomid component of a region name has been the timestamp of 
> region creation.   HBASE-2531 "32-bit encoding of regionnames waaay 
> too susceptible to hash clashes" proposes changing the randomid so that it 
> contains actual name of the directory in the filesystem that hosts the 
> region.  If we had this in place, I think it would help with the migration to 
> this new way of doing the meta because as is, the region name in fs is a hash 
> of regionname... changing the format of the regionname would mean we generate 
> a different hash... so we'd need hbase-2531 to be in place before we could do 
> this change.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-2600) Change how we do meta tables; from tablename+STARTROW+randomid to instead, tablename+ENDROW+randomid

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188795#comment-13188795
 ] 

Lars Hofhansl edited comment on HBASE-2600 at 1/18/12 11:17 PM:


@Alex: What is the problem with TestMetaMigrationRemovingHTD? The tar file with 
that version of HBase is in ./src/test/data/hbase-4388-root.dir.tgz.
I am actually not sure what it means if that test fails.
@Stack: That seems to be the last stumbling block to get a successful test run. 
I assume this test failing indicates something bad (that we cannot migrate meta 
with this patch applied).

  was (Author: lhofhansl):
@Alex: What the problem with TestMetaMigrationRemovingHTD? The tar file 
with that version of HBase is in ./src/test/data/hbase-4388-root.dir.tgz.
I am actually not sure what it means if that test fails.
@Stack: That seems to be the last stumbling block to get a successful test run. 
I assume this test failing indicates something bad (that we cannot migrate meta 
with this patch applied).
  
> Change how we do meta tables; from tablename+STARTROW+randomid to instead, 
> tablename+ENDROW+randomid
> 
>
> Key: HBASE-2600
> URL: https://issues.apache.org/jira/browse/HBASE-2600
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: Alex Newman
> Attachments: 
> 0001-Changed-regioninfo-format-to-use-endKey-instead-of-s.patch, 
> 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v2.patch, 
> 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v4.patch, 
> 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v6.patch, 
> 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v7.2.patch, 
> 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v8, 
> 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v8.1, 
> 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v9.patch, 
> 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen.patch, 
> 2600-trunk-01-17.txt, jenkins.pdf
>
>
> This is an idea that Ryan and I have been kicking around on and off for a 
> while now.
> If regionnames were made of tablename+endrow instead of tablename+startrow, 
> then in the metatables, doing a search for the region that contains the 
> wanted row, we'd just have to open a scanner using passed row and the first 
> row found by the scan would be that of the region we need (If offlined 
> parent, we'd have to scan to the next row).
> If we redid the meta tables in this format, we'd be using an access that is 
> natural to hbase, a scan as opposed to the perverse, expensive 
> getClosestRowBefore we currently have that has to walk backward in meta 
> finding a containing region.
> This issue is about changing the way we name regions.
> If we were using scans, prewarming client cache would be near costless (as 
> opposed to what we'll currently have to do which is first a 
> getClosestRowBefore and then a scan from the closestrowbefore forward).
> Converting to the new method, we'd have to run a migration on startup 
> changing the content in meta.
> Up to this, the randomid component of a region name has been the timestamp of 
> region creation.   HBASE-2531 "32-bit encoding of regionnames waaay 
> too susceptible to hash clashes" proposes changing the randomid so that it 
> contains actual name of the directory in the filesystem that hosts the 
> region.  If we had this in place, I think it would help with the migration to 
> this new way of doing the meta because as is, the region name in fs is a hash 
> of regionname... changing the format of the regionname would mean we generate 
> a different hash... so we'd need hbase-2531 to be in place before we could do 
> this change.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5205) Delete handles deleteFamily incorrectly

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13186649#comment-13186649
 ] 

Lars Hofhansl edited comment on HBASE-5205 at 1/16/12 2:17 AM:
---

I also find that the TS of the Delete's constructor is not used as default for 
delete{Column|Columns|Family|, which is *really* confusing. (And Put does the 
right thing, using the constructor TS for all subsequent adds by default).


  was (Author: lhofhansl):
I also find that the TS of the Delete's constructor is not used as default 
for delete{Column|Columns|Family|, which is *really* confusing.

  
> Delete handles deleteFamily incorrectly
> ---
>
> Key: HBASE-5205
> URL: https://issues.apache.org/jira/browse/HBASE-5205
> Project: HBase
>  Issue Type: Bug
>  Components: client
>Reporter: Lars Hofhansl
>Priority: Minor
>
> Delete.deleteFamily clears all other markers for the same family.
> That is not correct as some of these other markers might be for a later time.
> That logic should be removed.
> If (really) needed this can be slightly optimized by keeping track of the max 
> TS so far for each family.
> If both the TS-so-far and the TS of a new deleteFamily request is < 
> LATEST_TIMESTAMP and the TS-so-far is < deleteFamily marker, then the 
> previous delete marker can be removed.
> I think that might be overkill, as most deletes issued from clients are for 
> LATEST_TIMESTAMP (which the server translates to the current time).
> I'll have a (one-line) patch soon. Unless folks insist on the optimization I 
> mentioned above.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-3584) Allow atomic put/delete in one call

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-3584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13186613#comment-13186613
 ] 

Lars Hofhansl edited comment on HBASE-3584 at 1/15/12 10:53 PM:


I noticed that there is a hole in my logic when faced with region server 
failures.

Since the WALEdits are applied to the WAL piece by piece what can happen is 
that the region server applied the WALEdits of some of the mutations, and then 
dies before it can apply the edits of the rest of the mutations.

The region server who is picking up the region will replay those edit and hence 
clients picking up the KVs from the recovering region server will see the 
operation partially applied.
The fact that the MVCC write point was not forwarded by the first region server 
is irrelevant and not known to the recovering region server.

Other operation (Increment/Append) also have correctness issues (albeit 
different) in failure scenarios.

Filed HBASE-5203. Work on this would not be trivial. What's the general 
feeling: Disable this feature until this is fixed? Or leave it enabled, since 
it is in principle not safer than Increment or ICV?


  was (Author: lhofhansl):
I noticed that there is a hole in my logic when faced with region server 
failures.

Since the WALEdits are applied to the WAL piece by piece what can happen is 
that the region server applied the WALEdits of some of the mutations, and then 
dies before it can apply the edits of the other mutations. The fact that the 
MVCC write point was not forwarded by the first region server is irrelevant for 
this.

The region server who is picking up the region will replay those edit and hence 
clients picking up the KVs from the recovering region server will see the 
operation partially applied.

Other operation (Increment/Append) also have correctness issues (albeit 
different) in failure scenarios.

Filed HBASE-5203. Work on this would not be trivial. What's the general 
feeling: Disable this feature until this is fixed? Or leave it is, since it is 
in principle not safer than Increment or ICV?

  
> Allow atomic put/delete in one call
> ---
>
> Key: HBASE-3584
> URL: https://issues.apache.org/jira/browse/HBASE-3584
> Project: HBase
>  Issue Type: New Feature
>  Components: client, coprocessors, regionserver
>Reporter: ryan rawson
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 3584-final.txt, 3584-v1.txt, 3584-v3.txt
>
>
> Right now we have the following calls:
> put(Put)
> delete(Delete)
> increment(Increments)
> But we cannot combine all of the above in a single call, complete with a 
> single row lock.  It would be nice to do that.
> It would also allow us to do a CAS where we could do a put/increment if the 
> check succeeded.
> -
> Amendment:
> Since Increment does not currently support MVCC it cannot be included in an 
> atomic operation.
> So this for Put and Delete only.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5153) HConnection re-creation in HTable after HConnection abort

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13186271#comment-13186271
 ] 

Lars Hofhansl edited comment on HBASE-5153 at 1/14/12 5:33 PM:
---

Thanks Jieshan. This patch is great.
+1 after Ted's comments are fixed.

Are you planning to make a 0.92 and trunk patch as well?
(After Stacks changes in trunk that might be not completely trivial).

  was (Author: lhofhansl):
Thanks Jieshan. This patch is great.
+1 after Ted's comments are fixed.

Are you planning to make a 0.92 and trunk patch as well?
(Aft
  
> HConnection re-creation in HTable after HConnection abort
> -
>
> Key: HBASE-5153
> URL: https://issues.apache.org/jira/browse/HBASE-5153
> Project: HBase
>  Issue Type: Bug
>  Components: client
>Affects Versions: 0.90.4
>Reporter: Jieshan Bean
>Assignee: Jieshan Bean
> Fix For: 0.90.6
>
> Attachments: 5153-trunk.txt, HBASE-5153-V2.patch, 
> HBASE-5153-V3.patch, HBASE-5153-V4-90.patch, HBASE-5153-V5-90.patch, 
> HBASE-5153-trunk-v2.patch, HBASE-5153-trunk.patch, HBASE-5153.patch
>
>
> HBASE-4893 is related to this issue. In that issue, we know, if multi-threads 
> share a same connection, once this connection got abort in one thread, the 
> other threads will got a 
> "HConnectionManager$HConnectionImplementation@18fb1f7 closed" exception.
> It solve the problem of "stale connection can't removed". But the orignal 
> HTable instance cann't be continue to use. The connection in HTable should be 
> recreated.
> Actually, there's two aproach to solve this:
> 1. In user code, once catch an IOE, close connection and re-create HTable 
> instance. We can use this as a workaround.
> 2. In HBase Client side, catch this exception, and re-create connection.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-2376) Add special SnapshotScanner which presents view of all data at some time in the past

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185167#comment-13185167
 ] 

Lars Hofhansl edited comment on HBASE-2376 at 1/12/12 9:18 PM:
---

Is this addressed with HBASE-4536?

  was (Author: lhofhansl):
Is this fixed with HBASE-4536
  
> Add special SnapshotScanner which presents view of all data at some time in 
> the past
> 
>
> Key: HBASE-2376
> URL: https://issues.apache.org/jira/browse/HBASE-2376
> Project: HBase
>  Issue Type: New Feature
>  Components: client, regionserver
>Affects Versions: 0.20.3
>Reporter: Jonathan Gray
>
> In order to support a particular kind of database "snapshot" feature which 
> doesn't require copying data, we came up with the idea for a special 
> SnapshotScanner that would present a view of your data at some point in the 
> past.  The primary use case for this would be to be able to recover 
> particular data/rows (but not all data, like a global rollback) should they 
> have somehow been messed up (application fault, application bug, user error, 
> etc.).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5088) A concurrency issue on SoftValueSortedMap

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13181515#comment-13181515
 ] 

Lars Hofhansl edited comment on HBASE-5088 at 1/6/12 7:00 PM:
--

I think we should check in the latest change. Since we cannot reliably 
reproduce the issue, we have to verify by inspection. The latest change 
(5088-syncObject.txt) will be least amount of change from the original code and 
adds the least extra synchronization needed.

I'll fix up the patch slightly (pull in the interface cleanups from my other 
patch).

Ted, Stack, etc, please weigh in.


  was (Author: lhofhansl):
I think we should check in the latest change. Since we cannot reliably 
reproduce the issue, we have verify by inspection. The latest change 
(5088-syncObject.txt) will be least amount of change from the original code and 
adds the lowest least extra synchronization needed.

I'll fix up the patch slightly (pull in the interface cleanups from my other 
patch).

Ted, Stack, etc, please weigh in.

  
> A concurrency issue on SoftValueSortedMap
> -
>
> Key: HBASE-5088
> URL: https://issues.apache.org/jira/browse/HBASE-5088
> Project: HBase
>  Issue Type: Bug
>  Components: client
>Affects Versions: 0.90.4, 0.94.0
>Reporter: Jieshan Bean
>Assignee: Jieshan Bean
>Priority: Critical
> Fix For: 0.92.0, 0.90.6
>
> Attachments: 5088-syncObj.txt, 5088-useMapInterfaces.txt, 
> 5088.generics.txt, HBase-5088-90.patch, HBase-5088-trunk.patch, 
> HBase5088-90-replaceSoftValueSortedMap.patch, 
> HBase5088-90-replaceTreeMap.patch, HBase5088-trunk-replaceTreeMap.patch, 
> HBase5088Reproduce.java, PerformanceTestResults.png
>
>
> SoftValueSortedMap is backed by a TreeMap. All the methods in this class are 
> synchronized. If we use this method to add/delete elements, it's ok.
> But in HConnectionManager#getCachedLocation, it use headMap to get a view 
> from SoftValueSortedMap#internalMap. Once we operate 
> on this view map(like add/delete) in other threads, a concurrency issue may 
> occur.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5088) A concurrency issue on SoftValueSortedMap

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13181178#comment-13181178
 ] 

Lars Hofhansl edited comment on HBASE-5088 at 1/6/12 7:24 AM:
--

Here's a patch for sync object approach. It uses the internalMap itself to 
synchronize. When a new SoftValueSortedMap is created via {head|tail|sub}Map 
the parent's sync object is passed to the newly created map.
This way no memory is wasted as the {head|tail|sub}map hold a reference to 
their parents anyway.

  was (Author: lhofhansl):
Here's a patch for sync object approach. It uses the internalMap itself to 
synchronize. When a new SoftValueSortedMap is created via {head|tail|sub}Map 
the parent's sync object is passed to the newly created map.
This way to more memory is wasted as the {head|tail|sub}map hold a reference to 
their parents anyway.
  
> A concurrency issue on SoftValueSortedMap
> -
>
> Key: HBASE-5088
> URL: https://issues.apache.org/jira/browse/HBASE-5088
> Project: HBase
>  Issue Type: Bug
>  Components: client
>Affects Versions: 0.90.4, 0.94.0
>Reporter: Jieshan Bean
>Assignee: Jieshan Bean
>Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 5088-syncObj.txt, 5088-useMapInterfaces.txt, 
> 5088.generics.txt, HBase-5088-90.patch, HBase-5088-trunk.patch, 
> HBase5088-90-replaceSoftValueSortedMap.patch, 
> HBase5088-90-replaceTreeMap.patch, HBase5088-trunk-replaceTreeMap.patch, 
> HBase5088Reproduce.java, PerformanceTestResults.png
>
>
> SoftValueSortedMap is backed by a TreeMap. All the methods in this class are 
> synchronized. If we use this method to add/delete elements, it's ok.
> But in HConnectionManager#getCachedLocation, it use headMap to get a view 
> from SoftValueSortedMap#internalMap. Once we operate 
> on this view map(like add/delete) in other threads, a concurrency issue may 
> occur.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5084) Allow different HTable instances to share one ExecutorService

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13181108#comment-13181108
 ] 

Lars Hofhansl edited comment on HBASE-5084 at 1/6/12 5:09 AM:
--

The difference is that with
{code}
public HTable(final byte[] tableName, final HConnection connection, 
  final ExecutorService pool) throws IOException {
{code}
that I added in HBASE-4805 HTable will not need to do any cleanup.
It is by no means a big a deal, just that if ExecutorService is externally 
managed while HConnection is not, we need to maintain more state in HTable in 
order to do partial cleanup (close the - managed - HConnection, which decreases 
its reference count, but leave the ExecutorService alone).

We can add this logic, but I feel we should not over-engineer this... More 
variations lead to more bugs or unexpected behavior for new users.

In my fairy tale world HConnectionImplementation would have the ExecutorService 
as well, and you'd get an HTable via HConnection.getHTable().


  was (Author: lhofhansl):
The difference is that with the 
{code}
public HTable(final byte[] tableName, final HConnection connection, 
  final ExecutorService pool) throws IOException {
{code}
that I added in HBASE-4805 HTable will not need to do any cleanup.
It is by no means a big a deal, just that if ExecutorService is externally 
managed while HConnection is not, we need to maintain more state in HTable in 
order to do partial cleanup (close the - managed - HConnection, which decreases 
its reference count, but leave the ExecutorService alone).

We can add this logic, but I feel we should not over-engineer this... More 
variations lead to more bugs or unexpected behavior for new users.

In my fairy tale world HConnectionImplementation would have the ExecutorService 
as well, and you'd get an HTable via HConnection.getHTable().

  
> Allow different HTable instances to share one ExecutorService
> -
>
> Key: HBASE-5084
> URL: https://issues.apache.org/jira/browse/HBASE-5084
> Project: HBase
>  Issue Type: Task
>Reporter: Zhihong Yu
>
> This came out of Lily 1.1.1 release:
> Use a shared ExecutorService for all HTable instances, leading to better (or 
> actual) thread reuse

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-2947) MultiIncrement/MultiAppend (MultiGet functionality for increments and appends)

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179685#comment-13179685
 ] 

Lars Hofhansl edited comment on HBASE-2947 at 1/4/12 5:53 PM:
--

I ran the four failing tests locally and they all pass.
I would like to commit this today (the change itself is pretty uncontentious I 
think).


  was (Author: lhofhansl):
I ran the four failing tests locally and they all pass.
I would like to commit this today (the change it pretty uncontentious I think).

  
> MultiIncrement/MultiAppend (MultiGet functionality for increments and appends)
> --
>
> Key: HBASE-2947
> URL: https://issues.apache.org/jira/browse/HBASE-2947
> Project: HBase
>  Issue Type: New Feature
>  Components: client, regionserver
>Reporter: Jonathan Gray
>Assignee: Lars Hofhansl
>Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 2947-v2.txt, HBASE-2947-v1.patch
>
>
> HBASE-1845 introduced MultiGet and other cross-row/cross-region batch 
> operations.  We should add a way to do that with increments.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5120) Timeout monitor races with table disable handler

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179660#comment-13179660
 ] 

Lars Hofhansl edited comment on HBASE-5120 at 1/4/12 5:17 PM:
--

Hey Ram, I know you said this is not done, yet...
{code}
if (t instanceof NullPointerException) {
  removeRegionInTransition(region);
...
{code}

Could we instead add a null check at the relevant point and deal with it there 
(or maybe throw another exception)? (Dealing with this after the NPE occurred 
leaves a bad taste... What if we introduce another bug later that also causes 
an NPE? That would go unnoticed for a while.)


  was (Author: lhofhansl):
Hey Ram, I know you said this is not done, yet...
{code}
if (t instanceof NullPointerException) {
  removeRegionInTransition(region);
...
{code}

Could we instead add a null check at the relevant point and deal with it there 
(or maybe throw another exception)? (Dealing with this after the NPE occurred 
leaves a bad taste... What we introduce another bug later that also causes an 
NPE, that will go unnoticed for a while.)

  
> Timeout monitor races with table disable handler
> 
>
> Key: HBASE-5120
> URL: https://issues.apache.org/jira/browse/HBASE-5120
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Zhihong Yu
>Priority: Blocker
> Attachments: HBASE-5120.patch
>
>
> Here is what J-D described here:
> https://issues.apache.org/jira/browse/HBASE-5119?focusedCommentId=13179176&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13179176
> I think I will retract from my statement that it "used to be extremely racy 
> and caused more troubles than it fixed", on my first test I got a stuck 
> region in transition instead of being able to recover. The timeout was set to 
> 2 minutes to be sure I hit it.
> First the region gets closed
> {quote}
> 2012-01-04 00:16:25,811 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Sent CLOSE to 
> sv4r5s38,62023,1325635980913 for region 
> test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
> {quote}
> 2 minutes later it times out:
> {quote}
> 2012-01-04 00:18:30,026 INFO 
> org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed 
> out:  test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791. 
> state=PENDING_CLOSE, ts=1325636185810, server=null
> 2012-01-04 00:18:30,026 INFO 
> org.apache.hadoop.hbase.master.AssignmentManager: Region has been 
> PENDING_CLOSE for too long, running forced unassign again on 
> region=test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
> 2012-01-04 00:18:30,027 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of 
> region test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791. 
> (offlining)
> {quote}
> 100ms later the master finally gets the event:
> {quote}
> 2012-01-04 00:18:30,129 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=RS_ZK_REGION_CLOSED, server=sv4r5s38,62023,1325635980913, 
> region=1a4b111bcc228043e89f59c4c3f6a791, which is more than 15 seconds late
> 2012-01-04 00:18:30,129 DEBUG 
> org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED 
> event for 1a4b111bcc228043e89f59c4c3f6a791
> 2012-01-04 00:18:30,129 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Table being disabled so 
> deleting ZK node and removing from regions in transition, skipping assignment 
> of region test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
> 2012-01-04 00:18:30,129 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:62003-0x134589d3db03587 Deleting existing unassigned node for 
> 1a4b111bcc228043e89f59c4c3f6a791 that is in expected state RS_ZK_REGION_CLOSED
> 2012-01-04 00:18:30,166 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:62003-0x134589d3db03587 Successfully deleted unassigned node for 
> region 1a4b111bcc228043e89f59c4c3f6a791 in expected state RS_ZK_REGION_CLOSED
> {quote}
> At this point everything is fine, the region was processed as closed. But 
> wait, remember that line where it said it was going to force an unassign?
> {quote}
> 2012-01-04 00:18:30,322 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:62003-0x134589d3db03587 Creating unassigned node for 
> 1a4b111bcc228043e89f59c4c3f6a791 in a CLOSING state
> 2012-01-04 00:18:30,328 INFO 
> org.apache.hadoop.hbase.master.AssignmentManager: Server null returned 
> java.lang.NullPointerException: Passed server is null for 
> 1a4b111bcc228043e89f59c4c3f6a791
> {quote}
> Now the master is confused, it recreated the RIT znode but the region doesn't 
> even exist anymore. It even tries to shut it down but is block

[jira] [Issue Comment Edited] (HBASE-4218) Data Block Encoding of KeyValues (aka delta encoding / prefix compression)

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179057#comment-13179057
 ] 

Lars Hofhansl edited comment on HBASE-4218 at 1/3/12 9:59 PM:
--

+1 on avoiding different encoding on disk vs cache.
However, since we have all this framework in place, why not also allow it for 
disk only encoding?
It is in principle different from the current block based compression, as it 
can easily take the shape of KeyValues into account.

Could we have ENCODING, ENCODE_IN_CACHE, and ENCODE_ON_DISK?

  was (Author: lhofhansl):
+1 on avoiding different encoding on disk vs cache.
However, since we have all this framework in place, why not also allow it for 
disk only encoding.
It is in principle different from the current block based compression, as it 
can easily take the shape of KeyValues into account.

Could we have ENCODING, ENCODE_IN_CACHE, and ENCODE_ON_DISK?
  
> Data Block Encoding of KeyValues  (aka delta encoding / prefix compression)
> ---
>
> Key: HBASE-4218
> URL: https://issues.apache.org/jira/browse/HBASE-4218
> Project: HBase
>  Issue Type: Improvement
>  Components: io
>Affects Versions: 0.94.0
>Reporter: Jacek Migdal
>Assignee: Mikhail Bautin
>  Labels: compression
> Fix For: 0.94.0
>
> Attachments: 0001-Delta-encoding-fixed-encoded-scanners.patch, 
> 0001-Delta-encoding.patch, 4218-v16.txt, D447.1.patch, D447.10.patch, 
> D447.11.patch, D447.12.patch, D447.13.patch, D447.14.patch, D447.15.patch, 
> D447.16.patch, D447.17.patch, D447.2.patch, D447.3.patch, D447.4.patch, 
> D447.5.patch, D447.6.patch, D447.7.patch, D447.8.patch, D447.9.patch, 
> Data-block-encoding-2011-12-23.patch, 
> Delta-encoding.patch-2011-12-22_11_52_07.patch, 
> Delta_encoding_with_memstore_TS.patch, open-source.diff
>
>
> A compression for keys. Keys are sorted in HFile and they are usually very 
> similar. Because of that, it is possible to design better compression than 
> general purpose algorithms,
> It is an additional step designed to be used in memory. It aims to save 
> memory in cache as well as speeding seeks within HFileBlocks. It should 
> improve performance a lot, if key lengths are larger than value lengths. For 
> example, it makes a lot of sense to use it when value is a counter.
> Initial tests on real data (key length = ~ 90 bytes , value length = 8 bytes) 
> shows that I could achieve decent level of compression:
>  key compression ratio: 92%
>  total compression ratio: 85%
>  LZO on the same data: 85%
>  LZO after delta encoding: 91%
> While having much better performance (20-80% faster decompression ratio than 
> LZO). Moreover, it should allow far more efficient seeking which should 
> improve performance a bit.
> It seems that a simple compression algorithms are good enough. Most of the 
> savings are due to prefix compression, int128 encoding, timestamp diffs and 
> bitfields to avoid duplication. That way, comparisons of compressed data can 
> be much faster than a byte comparator (thanks to prefix compression and 
> bitfields).
> In order to implement it in HBase two important changes in design will be 
> needed:
> -solidify interface to HFileBlock / HFileReader Scanner to provide seeking 
> and iterating; access to uncompressed buffer in HFileBlock will have bad 
> performance
> -extend comparators to support comparison assuming that N first bytes are 
> equal (or some fields are equal)
> Link to a discussion about something similar:
> http://search-hadoop.com/m/5aqGXJEnaD1/hbase+windows&subj=Re+prefix+compression

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-4218) Data Block Encoding of KeyValues (aka delta encoding / prefix compression)

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179057#comment-13179057
 ] 

Lars Hofhansl edited comment on HBASE-4218 at 1/3/12 9:59 PM:
--

+1 on avoiding different encoding on disk vs cache.
However, since we have all this framework in place, why not also allow it for 
disk only encoding.
It is in principle different from the current block based compression, as it 
can easily take the shape of KeyValues into account.

Could we have ENCODING, ENCODE_IN_CACHE, and ENCODE_ON_DISK?

  was (Author: lhofhansl):
+1 on avoid different encoding on disk vs cache.
However, since we have all this framework in place, why not also allow it for 
disk only encoding.
It is in principle different from the current block based compression, as it 
can easily take the shape of KeyValues into account.

Could we have ENCODING, ENCODE_IN_CACHE, and ENCODE_ON_DISK?
  
> Data Block Encoding of KeyValues  (aka delta encoding / prefix compression)
> ---
>
> Key: HBASE-4218
> URL: https://issues.apache.org/jira/browse/HBASE-4218
> Project: HBase
>  Issue Type: Improvement
>  Components: io
>Affects Versions: 0.94.0
>Reporter: Jacek Migdal
>Assignee: Mikhail Bautin
>  Labels: compression
> Fix For: 0.94.0
>
> Attachments: 0001-Delta-encoding-fixed-encoded-scanners.patch, 
> 0001-Delta-encoding.patch, 4218-v16.txt, D447.1.patch, D447.10.patch, 
> D447.11.patch, D447.12.patch, D447.13.patch, D447.14.patch, D447.15.patch, 
> D447.16.patch, D447.17.patch, D447.2.patch, D447.3.patch, D447.4.patch, 
> D447.5.patch, D447.6.patch, D447.7.patch, D447.8.patch, D447.9.patch, 
> Data-block-encoding-2011-12-23.patch, 
> Delta-encoding.patch-2011-12-22_11_52_07.patch, 
> Delta_encoding_with_memstore_TS.patch, open-source.diff
>
>
> A compression for keys. Keys are sorted in HFile and they are usually very 
> similar. Because of that, it is possible to design better compression than 
> general purpose algorithms,
> It is an additional step designed to be used in memory. It aims to save 
> memory in cache as well as speeding seeks within HFileBlocks. It should 
> improve performance a lot, if key lengths are larger than value lengths. For 
> example, it makes a lot of sense to use it when value is a counter.
> Initial tests on real data (key length = ~ 90 bytes , value length = 8 bytes) 
> shows that I could achieve decent level of compression:
>  key compression ratio: 92%
>  total compression ratio: 85%
>  LZO on the same data: 85%
>  LZO after delta encoding: 91%
> While having much better performance (20-80% faster decompression ratio than 
> LZO). Moreover, it should allow far more efficient seeking which should 
> improve performance a bit.
> It seems that a simple compression algorithms are good enough. Most of the 
> savings are due to prefix compression, int128 encoding, timestamp diffs and 
> bitfields to avoid duplication. That way, comparisons of compressed data can 
> be much faster than a byte comparator (thanks to prefix compression and 
> bitfields).
> In order to implement it in HBase two important changes in design will be 
> needed:
> -solidify interface to HFileBlock / HFileReader Scanner to provide seeking 
> and iterating; access to uncompressed buffer in HFileBlock will have bad 
> performance
> -extend comparators to support comparison assuming that N first bytes are 
> equal (or some fields are equal)
> Link to a discussion about something similar:
> http://search-hadoop.com/m/5aqGXJEnaD1/hbase+windows&subj=Re+prefix+compression

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5088) A concurrency issue on SoftValueSortedMap

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13178295#comment-13178295
 ] 

Lars Hofhansl edited comment on HBASE-5088 at 1/2/12 6:04 AM:
--

@Jieshan: What do you think of my attached 5088.generics.txt patch?

It switches TreeMap with ConcurrentSkipListMap and also:
* fixes all generics warnings
* uses proper interfaces instead of concrete classes
* removes all synchronization (should not be necessary with the inner map being 
a ConcurrentSkipListMap).


  was (Author: lhofhansl):
@Jieshan: What do you think of my attached 5088.generics.txt patch?

It switches TreeMap with ConcurrentSkipListMap and also:
* fixes are generics warnings
* uses proper interface instead of concrete classes
* remove all synchronization (should not be necessary with the inner map being 
ConcurrentSkipListMap).

  
> A concurrency issue on SoftValueSortedMap
> -
>
> Key: HBASE-5088
> URL: https://issues.apache.org/jira/browse/HBASE-5088
> Project: HBase
>  Issue Type: Bug
>  Components: client
>Affects Versions: 0.90.4, 0.94.0
>Reporter: Jieshan Bean
>Assignee: Jieshan Bean
> Attachments: 5088-useMapInterfaces.txt, 5088.generics.txt, 
> HBase-5088-90.patch, HBase-5088-trunk.patch, HBase5088Reproduce.java
>
>
> SoftValueSortedMap is backed by a TreeMap. All the methods in this class are 
> synchronized. If we use this method to add/delete elements, it's ok.
> But in HConnectionManager#getCachedLocation, it use headMap to get a view 
> from SoftValueSortedMap#internalMap. Once we operate 
> on this view map(like add/delete) in other threads, a concurrency issue may 
> occur.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5088) A concurrency issue on SoftValueSortedMap

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175632#comment-13175632
 ] 

Lars Hofhansl edited comment on HBASE-5088 at 12/24/11 12:58 AM:
-

Here's a version that does that.
o SoftValueSortedMap implements NavigableMap
o All extra NavigableMap methods are delegated to internalMap
o None of its methods are synchronized
o All generics warnings are fixed.

Please have a look.

  was (Author: lhofhansl):
Here's a version that does that.
o SoftValueSortedMap implements NavigableMap
o All extra NavigableMap are delegated to internalMap
o None of its methods are synchronized
o All generics warnings are fixed.

Please have a look.
  
> A concurrency issue on SoftValueSortedMap
> -
>
> Key: HBASE-5088
> URL: https://issues.apache.org/jira/browse/HBASE-5088
> Project: HBase
>  Issue Type: Bug
>  Components: client
>Affects Versions: 0.90.4, 0.94.0
>Reporter: Jieshan Bean
>Assignee: Jieshan Bean
> Attachments: 5088-useMapInterfaces.txt, 5088.generics.txt, 
> HBase-5088-90.patch, HBase-5088-trunk.patch, HBase5088Reproduce.java
>
>
> SoftValueSortedMap is backed by a TreeMap. All the methods in this class are 
> synchronized. If we use this method to add/delete elements, it's ok.
> But in HConnectionManager#getCachedLocation, it use headMap to get a view 
> from SoftValueSortedMap#internalMap. Once we operate 
> on this view map(like add/delete) in other threads, a concurrency issue may 
> occur.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5088) A concurrency issue on SoftValueSortedMap

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13174646#comment-13174646
 ] 

Lars Hofhansl edited comment on HBASE-5088 at 12/22/11 6:35 AM:


ConcurrentSkipListMap does not appear to have this problem as a {head|tail}Map 
just maps put/get operations back to the original map. Maybe we can do the same 
approach here.

Cannot use ConcurrentSkipListMap directly as SoftValueSortedMap specifically 
stores SoftReferences to the values... Although I am personally not a big fan 
of this. Either we cache stuff or not. If there is memory pressure and the JVM 
removes the object referred to by the soft references, we generate even more 
new garbage when we miss the cache and have to reacquire the object.

Looks like SoftValueSortedMap.headMap() is buggy, we should throw 
UnsupportedOperation or fix it. Fixing it would mean to keep a reference to the 
original map and use that synchronize in the headMap (which means we need to 
return a special inner class), or to materialize the headmap in a copy, or do 
what ConcurrentSkipListMap does and map the API operations back to the original 
map.


  was (Author: lhofhansl):
ConcurrentSkipListMap does not appear to have this problem as a 
{head|tail}Map just maps put/get operations back to the original map. Maybe we 
can do the same approach here.

Cannot use ConcurrentSkipListMap as SoftValueSortedMap specifically stores 
SoftReferences to the values... Although I am personally not a big fan of this. 
Either we cache stuff or not. If there is memory pressure and the JVM removes 
the object referred to by the soft references, we generate even more new 
garbage when we miss the cache and have to reacquire the object.

Looks like SoftValueSortedMap.headMap() is buggy, we should throw 
UnsupportedOperation or fix it. Fixing it would mean to keep a reference to the 
original map and use that synchronize in the headMap (which means we need to 
return a special inner class), or to materialize the headmap in a copy, or do 
what ConcurrentSkipListMap does and map the API operations back to the original 
map.

  
> A concurrency issue on SoftValueSortedMap
> -
>
> Key: HBASE-5088
> URL: https://issues.apache.org/jira/browse/HBASE-5088
> Project: HBase
>  Issue Type: Bug
>  Components: client
>Affects Versions: 0.90.4, 0.94.0
>Reporter: Jieshan Bean
>Assignee: Jieshan Bean
>
> SoftValueSortedMap is backed by a TreeMap. All the methods in this class are 
> synchronized. If we use this method to add/delete elements, it's ok.
> But in HConnectionManager#getCachedLocation, it use headMap to get a view 
> from SoftValueSortedMap#internalMap. Once we operate 
> on this view map(like add/delete) in other threads, a concurrency issue may 
> occur.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5058) Allow HBaseAmin to use an existing connection

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173364#comment-13173364
 ] 

Lars Hofhansl edited comment on HBASE-5058 at 12/20/11 6:12 PM:


@Stack: I think that about sums it up. The complexity of layers and timeout 
stories are alleviated somewhat by parent HBASE-4805 (no per HTable threadpool, 
HTablePool no longer needed).
I had a brief look at the first issue, unless I am missing something this would 
require a nontrivial amount of refactoring. The simplest would be to do all 
network IO from the Connection thread rather than the application thread (as 
described in HBASE-4956). Would need allow for the client to synchronize and 
retrieve exceptions on/from a Future.

Short term, should we take HBASE-4805 all the way and add a getTable(...) 
method to HConnection? (Or even further and add put/get/scan/etc methods that 
take a table name to HConnection?)

Long term a design based on asynchhbase with a thin synchronous layer on top is 
probably the best option.

  was (Author: lhofhansl):
@Stack: I think that about sums it up. The complexity of layers and timeout 
stories are alleviated somewhat by parent HBASE-4805 (no per HTable threadpool, 
HTablePool no longer needed).
I had a brief look at the first issue, unless I am missing something this would 
require a nontrivial amount of refactoring. The simplest would be to do all 
network IO from the Connection thread rather than the application thread (as 
described in HBASE-4956). Would need allow for the client to synchronize and 
retrieved exceptions on/from a Future.

Short term, should we take HBASE-4805 all the way and a getTable(...) method to 
HConnection? (Or even further and add put/get/scan/etc methods that take a 
table name to HConnection?)

Long term a design based on asynchhbase with a thin synchronous layer on top is 
probably the best option.
  
> Allow HBaseAmin to use an existing connection
> -
>
> Key: HBASE-5058
> URL: https://issues.apache.org/jira/browse/HBASE-5058
> Project: HBase
>  Issue Type: Sub-task
>  Components: client
>Affects Versions: 0.94.0
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
>Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 5058-v2.txt, 5058-v3.txt, 5058-v3.txt, 5058.txt
>
>
> What HBASE-4805 does for HTables, this should do for HBaseAdmin.
> Along with this the shared error handling and retrying between HBaseAdmin and 
> HConnectionManager can also be improved. I'll attach a first pass patch soon.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5001) Improve the performance of block cache keys

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167933#comment-13167933
 ] 

Lars Hofhansl edited comment on HBASE-5001 at 12/12/11 11:14 PM:
-

Here's a patch against trunk.

Wasn't as bad as I thought. Made a bunch of things nicer (all the evict by 
prefix huh-hah is gone, was just used to evict by hfileName)

I ran all the tests that I needed to modify. Will now get a full run.

  was (Author: lhofhansl):
Here's a patch against trunk.

Wasn't as bad as I though. Made a bunch of things nicer (all the evict by 
prefix huh-hah is gone, was just used to evict by hfileName)

I ran all the tests that I needed to modify. Will now get a full run.
  
> Improve the performance of block cache keys
> ---
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.90.4
>Reporter: Jean-Daniel Cryans
>Assignee: Lars Hofhansl
>Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 5001-v1.txt
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we 
> are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x7fe0501ff800 
> nid=0x6c87 runnable [0x7fe0577f6000]
>java.lang.Thread.State: RUNNABLE
>   at java.util.Arrays.copyOf(Arrays.java:2882)
>   at 
> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
>   at 
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
>   at java.lang.StringBuilder.append(StringBuilder.java:119)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:80)
>   at 
> org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be 
> possible to allocate exactly what we need. Maybe use byte[] as the key and 
> drop the separator too.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5000) Speed up simultaneous reads of a block when block caching is turned off

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167360#comment-13167360
 ] 

Lars Hofhansl edited comment on HBASE-5000 at 12/12/11 5:39 AM:


+1 on documenting. Could state there that block caching can be disabled on a 
per scan bases, and in fact should be disabled if the data scanned is known (or 
likely to) not to fit in the cache.

Generally shouldn't aim for loading the same block only once per JVM if 
requested by multiple threads at the same time? Could do some kind of reference 
counting to keep track of other requesters, or register a call back when the 
block is loaded...


  was (Author: lhofhansl):
+1 on documenting. Could state there, that block caching can be disabled on 
a per scan bases, and in fact should be disable if the data scanned is known 
(or likely to) not to fit in the cache.

Generally shouldn't aim for loading the same block only once per JVM if 
requested by multiple threads at the same time? Could do some kind of reference 
counting to keep track of other requesters, or register a call back when the 
block is loaded...

  
> Speed up simultaneous reads of a block when block caching is turned off
> ---
>
> Key: HBASE-5000
> URL: https://issues.apache.org/jira/browse/HBASE-5000
> Project: HBase
>  Issue Type: Bug
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>Priority: Minor
>
> With block caching, when one client starts reading a block and another one 
> comes around asking for the same block, the second client waits for the first 
> one to finish reading and returns the block from cache. This is achieved by 
> locking on the block offset using IdLock, a "sparse lock" primitive allowing 
> to lock on arbitrary long numbers. However, in case there is no block 
> caching, there is no reason to wait for other clients that are reading the 
> same block. One challenge optimizing this that we don't necessary have 
> accurate information about whether other HFile API clients interested in the 
> block would cache it.
> Setting priority as minor, as it is very unusual to turn off block caching.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5000) Speed up simultaneous reads of a block when block caching is turned off

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167360#comment-13167360
 ] 

Lars Hofhansl edited comment on HBASE-5000 at 12/12/11 5:40 AM:


+1 on documenting. Could state there that block caching can be disabled on a 
per scan bases, and in fact should be disabled if the data scanned is known (or 
likely to) not to fit in the cache.

Generally shouldn't we aim for loading the same block only once per JVM if 
requested by multiple threads at the same time? Could do some kind of reference 
counting to keep track of other requesters, or register a call back when the 
block is loaded...


  was (Author: lhofhansl):
+1 on documenting. Could state there that block caching can be disabled on 
a per scan bases, and in fact should be disabled if the data scanned is known 
(or likely to) not to fit in the cache.

Generally shouldn't aim for loading the same block only once per JVM if 
requested by multiple threads at the same time? Could do some kind of reference 
counting to keep track of other requesters, or register a call back when the 
block is loaded...

  
> Speed up simultaneous reads of a block when block caching is turned off
> ---
>
> Key: HBASE-5000
> URL: https://issues.apache.org/jira/browse/HBASE-5000
> Project: HBase
>  Issue Type: Bug
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>Priority: Minor
>
> With block caching, when one client starts reading a block and another one 
> comes around asking for the same block, the second client waits for the first 
> one to finish reading and returns the block from cache. This is achieved by 
> locking on the block offset using IdLock, a "sparse lock" primitive allowing 
> to lock on arbitrary long numbers. However, in case there is no block 
> caching, there is no reason to wait for other clients that are reading the 
> same block. One challenge optimizing this that we don't necessary have 
> accurate information about whether other HFile API clients interested in the 
> block would cache it.
> Setting priority as minor, as it is very unusual to turn off block caching.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-4682) Support deleted rows using Import/Export

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13165029#comment-13165029
 ] 

Lars Hofhansl edited comment on HBASE-4682 at 12/8/11 6:47 AM:
---

Straight forward patch.

I found there's actually a bug in the Scan(Scan) constructor, which did not 
copy the passed Scan's attributes.

Will add some tests soon, but want to get a test run in tonight.

  was (Author: lhofhansl):
Straight forward patch.

I found there's actually a bug in the Scan(Scan) constructor, which did not 
copy the attributes passed Scan's.

Will add some tests soon, but want to get a test run in tonight.
  
> Support deleted rows using Import/Export
> 
>
> Key: HBASE-4682
> URL: https://issues.apache.org/jira/browse/HBASE-4682
> Project: HBase
>  Issue Type: Sub-task
>  Components: mapreduce
>Affects Versions: 0.94.0
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 4682-v1.txt
>
>
> Parent allow keeping deleted rows around. Would be nice if those could be 
> exported and imported as well.
> All the building blocks are there.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-4956) Control direct memory buffer consumption by HBaseClient

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164199#comment-13164199
 ] 

Lars Hofhansl edited comment on HBASE-4956 at 12/7/11 7:18 AM:
---

We might not have to go all the way to use netty (although that would be nice).
If we find that it is possible to avoid calling 
HBaseClient.Connection.sendParam from the client thread, but have the call 
actually be made from the Connection thread (after the callable representing 
the operation was queued), we have limited the number of threads that will have 
cached DirectBuffer on their behalf.
Stack suggested the only reason for the direct call might be to pass errors 
back to the client. We could hand the client a Deferred or Future that will 
eventually hold any encountered exception, the client could (and would by 
default to keep the current synchronous behavior) also wait on that object.

  was (Author: lhofhansl):
We might not have to go all the way to use netty (although that would be 
nice).
If we find that it is possible to avoid calling 
HBaseClient.Connection.sendParam from the client thread, but have the call 
actually be made from the Connection thread (after the callable representing 
the operation was queued), we have limited the number of threads that will have 
cached DirectBuffer on their behalf.
Stack suggested the only reason for the direct call might be to pass errors 
back to the client. We could hand the client a Deferred or Future that will 
eventually hold any encountered exception, the client could (and would be 
default to keep the current synchronous behavior) also wait on that object.
  
> Control direct memory buffer consumption by HBaseClient
> ---
>
> Key: HBASE-4956
> URL: https://issues.apache.org/jira/browse/HBASE-4956
> Project: HBase
>  Issue Type: New Feature
>Reporter: Ted Yu
>
> As Jonathan explained here 
> https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1
>  , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-4838) Port 2856 (TestAcidGuarantee is failing) to 0.92

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158086#comment-13158086
 ] 

Lars Hofhansl edited comment on HBASE-4838 at 11/28/11 12:52 AM:
-

Whoops, missed adding MultiVersionConsistencyControl.java and removing 
ReadWriteConsistencyControl.java.

Fixed in addendum patch.

  was (Author: lhofhansl):
Whoops, missing adding MultiVersionConsistencyControl.java and removing 
ReadWriteConsistencyControl.java.

Fixed in addendum patch.
  
> Port 2856 (TestAcidGuarantee is failing) to 0.92
> 
>
> Key: HBASE-4838
> URL: https://issues.apache.org/jira/browse/HBASE-4838
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4838-v1.txt, 4838-v3.txt
>
>
> Moving back port into a separate issue (as suggested by JonH), because this 
> not trivial.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-4874) TestHCM#testClosing fails if not enough entropy is available

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13157378#comment-13157378
 ] 

Lars Hofhansl edited comment on HBASE-4874 at 11/26/11 5:54 AM:


This happens at least on Linux when SecureRandom is used.
SecureRandom uses /dev/random in Linux, which will block if not enough entropy 
has been gathered by system events (disk, network, keyboard, mouse movements, 
etc).

It's *not* only testClosing. Every time a test cluster is started it might 
hang, if the system happens to run out of entropy bits to generate secure 
random numbers. I've seen the same with TestFromClient and other tests that 
start an actual cluster for the test.

The fix I found was to add "-Djava.security.egd=file:/dev/./urandom" (note the 
extra ./ in path is needed for JDK 1.5 or newer) to the command line. 
/dev/urandom never blocks, but if not enough entropy has been gathered the 
random numbers generated won't be truly random (which is fine for the tests).

  was (Author: lhofhansl):
This happens at least on Linux when SecureRandom is used.
SecureRandom uses /dev/random in Linux, which will block if not enough entropy 
has been gathered by system events (disk, network, keyboard, mouse movements, 
etc).

It's *not* only testClosing. Every time a test cluster is started it might 
hang, if the system happens to run out of entropy bits to generate secure 
random numbers. I've seen the same with TestFromClient and other tests that 
start an actual cluster for the test.

The fix I found was to add "-Djava.security.egd=file:/dev/./urandom" (note the 
extra ./ in path is needed for JDK 1.5 or newer) to the command line.

  
> TestHCM#testClosing fails if not enough entropy is available
> 
>
> Key: HBASE-4874
> URL: https://issues.apache.org/jira/browse/HBASE-4874
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Ted Yu
>
> TestHCM#testClosing fails if not enough entropy is available

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-4874) TestHCM#testClosing fails if not enough entropy is available

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13157378#comment-13157378
 ] 

Lars Hofhansl edited comment on HBASE-4874 at 11/26/11 5:53 AM:


This happens at least on Linux when SecureRandom is used.
SecureRandom uses /dev/random in Linux, which will block if not enough entropy 
has been gathered by system events (disk, network, keyboard, mouse movements, 
etc).

It's *not* only testClosing. Every time a test cluster is started it might 
hang, if the system happens to run out of entropy bits to generate secure 
random numbers. I've seen the same with TestFromClient and other tests that 
start an actual cluster for the test.

The fix I found was to add "-Djava.security.egd=file:/dev/./urandom" (note the 
extra ./ in path is needed for JDK 1.5 or newer) to the command line.


  was (Author: lhofhansl):
This happens at least on Linux when SecureRandom is used.
SecureRandom used /dev/random in Linux, which will block if not enough entropy 
has been gathered by system events (disk, network, keyboard, mouse movements, 
etc).

It's *not* only testClosing. Every time a test cluster is started it might 
hang, if the system happens to run out of entropy its to generated secure 
random numbers. I've seen the same with TestFromClient and other tests that 
start an actual cluster for the test.

The fix I found was to add "-Djava.security.egd=file:/dev/./urandom" (note the 
extra ./ in path is needed for JDK 1.5 or newer) to the command line.

  
> TestHCM#testClosing fails if not enough entropy is available
> 
>
> Key: HBASE-4874
> URL: https://issues.apache.org/jira/browse/HBASE-4874
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Ted Yu
>
> TestHCM#testClosing fails if not enough entropy is available

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-4838) Port 2856 (TestAcidGuarantee is failing) to 0.92

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13157345#comment-13157345
 ] 

Lars Hofhansl edited comment on HBASE-4838 at 11/26/11 12:30 AM:
-

Finally found the problem!!
It was indeed a problem in HalfStoreFileReader (as Todd and Stack have 
suggested all along). HalfStoreFileReader did not have a getScanner(final 
boolean cacheBlocks, final boolean pread, final boolean isCompaction) method, 
and hence the super method was called, returning a "normal" ScannerV2, instead 
of the inner HFileScanner instance.

bq. Reference.java and HalfStoreFileReader.java are identical between 0.92 and 
trunk

I don't know what I was comparing here (must have been late in the night), but 
they are different and that difference was exactly the problem. Could have 
saved myself about 16 hours of debugging.


  was (Author: lhofhansl):
Finally found the problem!!
It was indeed a problem in HalfStoreFileReader (as Todd and Stack have 
suggested all along). HalfStoreFileReader did not have a getScanner(final 
boolean cacheBlocks, final boolean pread, final boolean isCompaction) method, 
and hence the super method was called returning a normal reader.

bq. Reference.java and HalfStoreFileReader.java are identical between 0.92 and 
trunk

I don't know what I was comparing here (must have been late in the night), but 
they are different and that difference was exactly the problem. Could have 
saved myself about 16 hours of debugging.

  
> Port 2856 (TestAcidGuarantee is failing) to 0.92
> 
>
> Key: HBASE-4838
> URL: https://issues.apache.org/jira/browse/HBASE-4838
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4838-v1.txt, 4838-v3.txt
>
>
> Moving back port into a separate issue (as suggested by JonH), because this 
> not trivial.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-4787) Make corePool as a configurable parameter in HTable

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156449#comment-13156449
 ] 

Lars Hofhansl edited comment on HBASE-4787 at 11/24/11 1:39 AM:


@Nicolas: Correct. It also add the ability to create "unmanaged" HConnections.

This allows for a philosophical change of managing HConnections rather than 
HTables. Together with HBASE-3767 that obviates (for us at least) the need of 
HTablePool, HTable objects can be created whenever needed. I think that is a 
better way of looking at the client, especially in a AppServer setting.


  was (Author: lhofhansl):
@Nicolas: Correct. It also add the ability to create "unmanaged" 
HConnections.

This allows for a philosophical change of managing HConnections rather than 
HTables. Together with HBASE-3767 that obviates (for us at least) the need of 
HTablePool, HTable object can be created whenever needed. I think that is a 
better way of looking at the client, especially in a AppServer setting.

  
> Make corePool as a configurable parameter in HTable
> ---
>
> Key: HBASE-4787
> URL: https://issues.apache.org/jira/browse/HBASE-4787
> Project: HBase
>  Issue Type: Improvement
>Reporter: Nicolas Spiegelberg
>Priority: Trivial
> Fix For: 0.94.0
>
> Attachments: HBASE-4787.patch
>
>
> Make the corePool a configurable parameter in HTable. So we can tune this 
> parameter in the config file.  While at it, change the core pool name so we 
> can distinguish it from other AppServer pools.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-4787) Make corePool as a configurable parameter in HTable

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156449#comment-13156449
 ] 

Lars Hofhansl edited comment on HBASE-4787 at 11/24/11 1:39 AM:


@Nicolas: Correct. It also add the ability to create "unmanaged" HConnections.

This allows for a philosophical change of managing HConnections rather than 
HTables. Together with HBASE-3767 that obviates (for us at least) the need for 
HTablePool; HTable objects can be created whenever needed. I think that is a 
better way of looking at the client, especially in a AppServer setting.


  was (Author: lhofhansl):
@Nicolas: Correct. It also add the ability to create "unmanaged" 
HConnections.

This allows for a philosophical change of managing HConnections rather than 
HTables. Together with HBASE-3767 that obviates (for us at least) the need of 
HTablePool, HTable objects can be created whenever needed. I think that is a 
better way of looking at the client, especially in a AppServer setting.

  
> Make corePool as a configurable parameter in HTable
> ---
>
> Key: HBASE-4787
> URL: https://issues.apache.org/jira/browse/HBASE-4787
> Project: HBase
>  Issue Type: Improvement
>Reporter: Nicolas Spiegelberg
>Priority: Trivial
> Fix For: 0.94.0
>
> Attachments: HBASE-4787.patch
>
>
> Make the corePool a configurable parameter in HTable. So we can tune this 
> parameter in the config file.  While at it, change the core pool name so we 
> can distinguish it from other AppServer pools.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-2856) TestAcidGuarantee broken on trunk

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13153852#comment-13153852
 ] 

Lars Hofhansl edited comment on HBASE-2856 at 11/20/11 11:38 PM:
-

@Nicolas and @Amit, could you review the 0.92 patch? It turned out to be much 
more manual than I had wished or expected, so it is very possible that I missed 
something.

(I tried to upload the 0.92 patch to review board for easier verification, but 
apparently that does not work for branches other than trunk.)

  was (Author: lhofhansl):
@Nicolas and @Amit, could you review the 0.92 patch? I turned out to be 
much more manual than I had wished or expected, so it is very possible that I 
missed something.

(I tried to upload the 0.92 patch to review board for easier verification, but 
apparently that does not work for branches other than trunk.)
  
> TestAcidGuarantee broken on trunk 
> --
>
> Key: HBASE-2856
> URL: https://issues.apache.org/jira/browse/HBASE-2856
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.89.20100621
>Reporter: ryan rawson
>Assignee: Amitanand Aiyer
>Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 2856-0.92.txt, 2856-v2.txt, 2856-v3.txt, 2856-v4.txt, 
> 2856-v5.txt, 2856-v6.txt, 2856-v7.txt, 2856-v8.txt, 
> 2856-v9-all-inclusive.txt, acid.txt
>
>
> TestAcidGuarantee has a test whereby it attempts to read a number of columns 
> from a row, and every so often the first column of N is different, when it 
> should be the same.  This is a bug deep inside the scanner whereby the first 
> peek() of a row is done at time T then the rest of the read is done at T+1 
> after a flush, thus the memstoreTS data is lost, and previously 'uncommitted' 
> data becomes committed and flushed to disk.
> One possible solution is to introduce the memstoreTS (or similarly equivalent 
> value) to the HFile thus allowing us to preserve read consistency past 
> flushes.  Another solution involves fixing the scanners so that peek() is not 
> destructive (and thus might return different things at different times alas).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-2856) TestAcidGuarantee broken on trunk

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13153852#comment-13153852
 ] 

Lars Hofhansl edited comment on HBASE-2856 at 11/20/11 7:20 PM:


@Nicolas and @Amit, could you review the 0.92 patch? I turned out to be much 
more manual than I had wished or expected, so it is very possible that I missed 
something.

(I tried to upload the 0.92 patch to review board for easier verification, but 
apparently that does not work for branches other than trunk.)

  was (Author: lhofhansl):
@Nicolas and @Amit, could you review the 0.92 patch? I turned out to be 
much more manual than I had wished or expected, so it is very possible, that I 
missed something.

(I tried to upload the 0.92 patch to review board for easier verification but 
apparently that does not work for branches other than trunk.)
  
> TestAcidGuarantee broken on trunk 
> --
>
> Key: HBASE-2856
> URL: https://issues.apache.org/jira/browse/HBASE-2856
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.89.20100621
>Reporter: ryan rawson
>Assignee: Amitanand Aiyer
>Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 2856-0.92.txt, 2856-v2.txt, 2856-v3.txt, 2856-v4.txt, 
> 2856-v5.txt, 2856-v6.txt, 2856-v7.txt, 2856-v8.txt, 
> 2856-v9-all-inclusive.txt, acid.txt
>
>
> TestAcidGuarantee has a test whereby it attempts to read a number of columns 
> from a row, and every so often the first column of N is different, when it 
> should be the same.  This is a bug deep inside the scanner whereby the first 
> peek() of a row is done at time T then the rest of the read is done at T+1 
> after a flush, thus the memstoreTS data is lost, and previously 'uncommitted' 
> data becomes committed and flushed to disk.
> One possible solution is to introduce the memstoreTS (or similarly equivalent 
> value) to the HFile thus allowing us to preserve read consistency past 
> flushes.  Another solution involves fixing the scanners so that peek() is not 
> destructive (and thus might return different things at different times alas).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-3025) Coprocessor based simple access control

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13153410#comment-13153410
 ] 

Lars Hofhansl edited comment on HBASE-3025 at 11/19/11 6:31 AM:


Is there a step-by-step guide somewhere on how to set this up?

  was (Author: lhofhansl):
Is there a step-by-step guide somewhere on to exactly set this up?
  
> Coprocessor based simple access control
> ---
>
> Key: HBASE-3025
> URL: https://issues.apache.org/jira/browse/HBASE-3025
> Project: HBase
>  Issue Type: Sub-task
>  Components: coprocessors
>Reporter: Andrew Purtell
>Priority: Critical
> Fix For: 0.92.0
>
> Attachments: HBASE-3025.1.patch, HBASE-3025_5.patch, 
> HBASE-3025_6.patch
>
>
> Thanks for the clarification Jeff which reminds me to edit this issue.
> Goals of this issue
> # Client access to HBase is authenticated
> # User data is private unless access has been granted
> # Access to data can be granted at a table or per column family basis. 
> Non-Goals of this issue
> The following items will be left out of the initial implementation for 
> simplicity:
> # Row-level or per value (cell) This would require broader changes for 
> storing the ACLs inline with rows. It's still a future goal, but would slow 
> down the initial implementation considerably.
> # Push down of file ownership to HDFS While table ownership seems like a 
> useful construct to start with (at least to lay the groundwork for future 
> changes), making HBase act as table owners when interacting with HDFS would 
> require more changes. In additional, while HDFS file ownership would make 
> applying quotas easy, and possibly make bulk imports more straightforward, 
> it's not clean it would offer a more secure setup. We'll leave this to 
> evaluate in a later phase.
> # HBase managed "roles" as collections of permissions We will not model 
> "roles" internally in HBase to begin with. We will instead allow group names 
> to be granted permissions, which will allow some external modeling of roles 
> via group memberships. Groups will be created and manipulated externally to 
> HBase. 
> While the assignment of permissions to roles and roles to users (or other 
> roles) allows a great deal of flexibility in security policy, it would add 
> complexity to the initial implementation. 
> After the initial implementation, which will appear on this issue, we will 
> evaluate the addition of role definitions internal to HBase in a new JIRA. In 
> this scheme, administrators could assign permissions specifying HDFS groups, 
> and additionally HBase roles. HBase roles would be created and manipulated 
> internally to HBase, and would appear distinct from HDFS groups via some 
> syntactic sugar. HBase role definitions will be allowed to reference other 
> HBase role definitions. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-4748) Race between creating recovered edits for META and master assigning ROOT and META.

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13147929#comment-13147929
 ] 

Lars Hofhansl edited comment on HBASE-4748 at 11/10/11 7:23 PM:


Makes sense. +1 for patch.

Ram, can you validate that your scenarios are all fixed with this change?


  was (Author: lhofhansl):
Makes +1 for patch.

Ram, can you validate that your scenarios are all fixed with this change?

  
> Race between creating recovered edits for META and master assigning ROOT and 
> META.
> --
>
> Key: HBASE-4748
> URL: https://issues.apache.org/jira/browse/HBASE-4748
> Project: HBase
>  Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: 4748.txt
>
>
> 1. Start a cluster.
> 2. Alter a table
> 3. Restart the master using ./hbase-daemon.sh restart master
> 4. Kill the RS after master restarts.
> 5. Start RS again.
> 6. No table operations can be performed on the table that was altered but 
> admin.listTables() is able to list the altered table.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-4721) Retain Delete Markers after Major Compaction

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145688#comment-13145688
 ] 

Lars Hofhansl edited comment on HBASE-4721 at 11/7/11 6:44 PM:
---

Is this not covered with HBASE-4536? I.e. just keeping the delete markers will 
not get you very far, you also have to keep the "deleted" rows (KVs affected by 
the delete markers).

  was (Author: lhofhansl):
Is this not covered with HBASE-4536?
  
> Retain Delete Markers after Major Compaction
> 
>
> Key: HBASE-4721
> URL: https://issues.apache.org/jira/browse/HBASE-4721
> Project: HBase
>  Issue Type: New Feature
>Reporter: Prakash Khemani
>Assignee: Prakash Khemani
>
> There is a need to provide long TTLs for delete markers. This is useful when 
> replicating hbase logs from one cluster to another. The receiving cluster 
> shouldn't compact away the delete markers because the affected key-values 
> might still be on the way.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-4583) Integrate RWCC with Append and Increment operations

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139550#comment-13139550
 ] 

Lars Hofhansl edited comment on HBASE-4583 at 10/30/11 7:06 AM:


Here's a *sketch* of a patch.
May need some clean up but should work.

Basically the KVs of Append and Increment are applied to memstore through a 
call to the "standard" applyFamilyMapToMemstore(...); then duplicates are 
removed after the rwcc is moved forward.
Still a lot of boilerplate shared between Append and Increment.

I think the dup removal does not need to hold the regions updatesLock.readLock 
(just the lock.readLock), but I am not entirely sure.

Comments/suggestions/flames/praise are welcome. :)


  was (Author: lhofhansl):
Here a *sketch* of a patch.
May need some clean up but should work.

Basically the KVs of Append and Increment are applied to memstore through a 
call to the "standard" applyFamilyMapToMemstore(...); then duplicates are 
removed after the rwcc is moved forward.
Still a lot of boilerplate shared between Append and Increment.

I think the dup removal does not need to hold the regions updatesLock.readLock 
(just the lock.readLock), but I am not entirely sure.

Comments/suggestions/flames/praise are welcome. :)

  
> Integrate RWCC with Append and Increment operations
> ---
>
> Key: HBASE-4583
> URL: https://issues.apache.org/jira/browse/HBASE-4583
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 4583.txt
>
>
> Currently Increment and Append operations do not work with RWCC and hence a 
> client could see the results of multiple such operation mixed in the same 
> Get/Scan.
> The semantics might be a bit more interesting here as upsert adds and removes 
> to and from the memstore.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-4583) Integrate RWCC with Append and Increment operations

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139551#comment-13139551
 ] 

Lars Hofhansl edited comment on HBASE-4583 at 10/30/11 7:08 AM:


Also changes some calls to take  "? extends Collection"  rather than List, so 
that I can use guava's MultiMap. If all callers would be changed the  "? 
extends"  could be removed.


  was (Author: lhofhansl):
Also changes some calls to take  ? extends Collection  rather than List, so 
that I can use guava's MultiMap. If all callers would be changed the  ? extends 
 could be removed.

  
> Integrate RWCC with Append and Increment operations
> ---
>
> Key: HBASE-4583
> URL: https://issues.apache.org/jira/browse/HBASE-4583
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 4583.txt
>
>
> Currently Increment and Append operations do not work with RWCC and hence a 
> client could see the results of multiple such operation mixed in the same 
> Get/Scan.
> The semantics might be a bit more interesting here as upsert adds and removes 
> to and from the memstore.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-4648) Bytes.toBigDecimal() doesn't use offset

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13135371#comment-13135371
 ] 

Lars Hofhansl edited comment on HBASE-4648 at 10/25/11 7:56 PM:


Patch looks good to me.
I'm fine applying to 0.90, 0.92, and trunk. Nobody should use 
toBigDecimal(byte[], int).

If nobody objects I'll commit this later today.

  was (Author: lhofhansl):
Patch looks good to me.
I'm find applying to 0.90, 0.92, and trunk. Nobody should use 
toBigDecimal(byte[], int).

If nobody objects I'll commit this later today.
  
> Bytes.toBigDecimal() doesn't use offset
> ---
>
> Key: HBASE-4648
> URL: https://issues.apache.org/jira/browse/HBASE-4648
> Project: HBase
>  Issue Type: Bug
>  Components: util
>Affects Versions: 0.90.4
> Environment: Java 1.6.0_26, Mac OS X 10.7 and CentOS 6
>Reporter: Bryan Keller
> Attachments: bigdec.patch, bigdec2.patch
>
>
> The Bytes.toBigDecimal(byte[], offset, len) method does not use the offset, 
> thus you will get an incorrect result for the BigDecimal unless the 
> BigDecimal's bytes are at the beginning of the byte array.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-4562) When split doing offlineParentInMeta encounters error, it'll cause data loss

[Lars Hofhansl (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13129457#comment-13129457
 ] 

Lars Hofhansl edited comment on HBASE-4562 at 10/18/11 3:40 AM:


No problem. Thanks for the patch :)

  was (Author: lhofhansl):
No problem. Thanks for patch :)
  
> When split doing offlineParentInMeta encounters error, it'll cause data loss
> 
>
> Key: HBASE-4562
> URL: https://issues.apache.org/jira/browse/HBASE-4562
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.90.4
>Reporter: bluedavy
>Assignee: bluedavy
>Priority: Blocker
> Fix For: 0.90.5
>
> Attachments: HBASE-4562-0.90.4.patch, HBASE-4562-0.90.patch, 
> HBASE-4562-0.92.patch, HBASE-4562-trunk.patch, test-4562-0.90.4.txt, 
> test-4562-0.90.txt, test-4562-0.92.txt, test-4562-trunk.txt
>
>
> Follow below steps to replay the problem:
> 1. change the SplitTransaction.java as below,just like mock the timeout error.
>{code:title=SplitTransaction.java|borderStyle=solid}
>   if (!testing) {
> MetaEditor.offlineParentInMeta(server.getCatalogTracker(),
>this.parent.getRegionInfo(), a.getRegionInfo(), b.getRegionInfo());
> throw new IOException("some unexpected error in split");
>   }
>{code} 
> 2. update the regionserver code,restart;
> 3. create a table & put some data to the table;
> 4. split the table;
> 5. kill the regionserver hosted the table;
> 6. wait some time after master ServerShutdownHandler.process execute,then 
> scan the table,u'll find the data wrote before lost.
> We can fix the bug just use the patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira