date:20111021

2011-10-21 Thread ramkrishna.s.vasudevan (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13132371#comment-13132371
 ] 

Hudson commented on HBASE-4626:
---

Integrated in HBase-TRUNK #2348 (See 
[https://builds.apache.org/job/HBase-TRUNK/2348/])
HBASE-4626  Filters unnecessarily copy byte arrays

larsh : 
Files : 
* /hbase/trunk/CHANGES.txt
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/BinaryComparator.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/BinaryPrefixComparator.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/BitComparator.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/CompareFilter.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/NullComparator.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/RegexStringComparator.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueFilter.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/SubstringComparator.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/filter/WritableByteArrayComparable.java


 Filters unnecessarily copy byte arrays...
 -

 Key: HBASE-4626
 URL: https://issues.apache.org/jira/browse/HBASE-4626
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.92.0, 0.94.0

 Attachments: 4626-v2.txt, 4626-v3.txt, 4626.txt


 Just looked at SingleCol and ValueFilter... And on every column compared they 
 create a copy of the column and/or value portion of the KV.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-3929) Add option to HFile tool to produce basic stats

2011-10-21 Thread Matteo Bertozzi (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Bertozzi updated HBASE-3929:
---

Attachment: HBASE-3929-v3.patch

Added v3 that applies to trunk after HBASE-4595 integration.

 Add option to HFile tool to produce basic stats
 ---

 Key: HBASE-3929
 URL: https://issues.apache.org/jira/browse/HBASE-3929
 Project: HBase
  Issue Type: New Feature
  Components: io
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
 Fix For: 0.94.0

 Attachments: HBASE-3929-v2.patch, HBASE-3929-v3.patch, 
 hbase-3929-draft.patch, hbase-3929-draft.txt


 In looking at HBASE-3421 I wrote a small tool to scan an HFile and produce 
 some basic statistics about it:
 - min/mean/max key size, value size (uncompressed)
 - min/mean/max number of columns per row (uncompressed)
 - min/mean/max number of bytes per row (uncompressed)
 - the key of the largest row

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog

[
https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13132523#comment-13132523
]

ramkrishna.s.vasudevan commented on HBASE-4528:
---

@Dhruba
I found one more thing using the testcase TestParallelPut.java. If the
LogSyncer thread starts running first time the unflushedEntries will be 0 and
syncedTillHere will also be 0.
Now first put will make this unFlushedEntries to be 1. Now the LogSyncer
thread will check for the condition
{code}
try {
if (unflushedEntries.get() = syncedTillHere) {
Thread.sleep(this.optionalFlushInterval);
}
sync();
}
{code}
now there is a chance that just after the append is completed to HLog before
the lock gets released the entry may be synced because of LogFlusher Thread.
Is it ok ? Pls correct me if am wrong. I think it should not be a problem but
syncing will happen in 2 different places.

The put operation can release the rowlock before sync-ing the Hlog
--

Key: HBASE-4528
URL: https://issues.apache.org/jira/browse/HBASE-4528
Project: HBase
Issue Type: Improvement
Components: regionserver
Reporter: dhruba borthakur
Assignee: dhruba borthakur
Fix For: 0.94.0

Attachments: appendNoSync5.txt, appendNoSyncPut1.txt,
appendNoSyncPut2.txt, appendNoSyncPut3.txt, appendNoSyncPut4.txt,
appendNoSyncPut5.txt, appendNoSyncPut6.txt

This allows for better throughput when there are hot rows. A single row
update improves from 100 puts/sec/server to 5000 puts/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4585) Avoid next operations (and instead reseek) when current kv is deleted


[ 
https://issues.apache.org/jira/browse/HBASE-4585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13132563#comment-13132563
 ] 

Hudson commented on HBASE-4585:
---

Integrated in HBase-0.92 #75 (See 
[https://builds.apache.org/job/HBase-0.92/75/])
HBASE-4585 Revert for 0.92, patch is for TRUNK only
HBASE-4585 - Avoid seek operation when current kv is deleted (Liyin Tang)

tedyu : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/DeleteTracker.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/ScanDeleteTracker.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
* 
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java
* 
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanDeleteTracker.java

ramkrishna : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/DeleteTracker.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/ScanDeleteTracker.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
* 
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java
* 
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanDeleteTracker.java


 Avoid next operations (and instead reseek) when current kv is deleted
 -

 Key: HBASE-4585
 URL: https://issues.apache.org/jira/browse/HBASE-4585
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang
 Fix For: 0.94.0

 Attachments: hbase-4585-89.patch, hbase-4585-apache-trunk.patch


 When the current kv is deleted during the matching in the ScanQueryMatcher, 
 currently the matcher will return skip and continue to do next() instead of 
 reseeking to the next column of interest.
 Actually, if the current kv is deleted because of family deleted or column 
 deleted, the matcher should seek to next col.
 If the current kv is deleted because of version deleted, the matcher should 
 just return skip.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4459) HbaseObjectWritable code is a byte, we will eventually run out of codes


[ 
https://issues.apache.org/jira/browse/HBASE-4459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13132564#comment-13132564
 ] 

Hudson commented on HBASE-4459:
---

Integrated in HBase-0.92 #75 (See 
[https://builds.apache.org/job/HBase-0.92/75/])
HBASE-4459 HbaseObjectWritable code is a byte, we will eventually run out 
of codes

ramkrishna : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java
* 
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/io/TestHbaseObjectWritable.java


 HbaseObjectWritable code is a byte, we will eventually run out of codes
 ---

 Key: HBASE-4459
 URL: https://issues.apache.org/jira/browse/HBASE-4459
 Project: HBase
  Issue Type: Bug
  Components: io
Reporter: Jonathan Gray
Assignee: ramkrishna.s.vasudevan
Priority: Critical
 Fix For: 0.92.0

 Attachments: 4459-v2.patch.txt, 4459.txt


 There are about 90 classes/codes in HbaseObjectWritable currently and 
 Byte.MAX_VALUE is 127.  In addition, anyone wanting to add custom classes but 
 not break compatibility might want to leave a gap before using codes and 
 that's difficult in such limited space.
 Eventually we should get rid of this pattern that makes compatibility 
 difficult (better client/server protocol handshake) but we should probably at 
 least bump this to a short for 0.94.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4621) TestAvroServer fails quite often intermittently


[ 
https://issues.apache.org/jira/browse/HBASE-4621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13132565#comment-13132565
 ] 

Hudson commented on HBASE-4621:
---

Integrated in HBase-0.92 #75 (See 
[https://builds.apache.org/job/HBase-0.92/75/])
HBASE-4621 TestAvroServer fails quite often intermittently

stack : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/avro/TestAvroServer.java


 TestAvroServer fails quite often intermittently
 ---

 Key: HBASE-4621
 URL: https://issues.apache.org/jira/browse/HBASE-4621
 Project: HBase
  Issue Type: Bug
Reporter: Akash Ashok
Assignee: Akash Ashok
 Fix For: 0.92.0




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4620) I broke the build when I submitted HBASE-3581 (Send length of the rpc response)


[ 
https://issues.apache.org/jira/browse/HBASE-4620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13132566#comment-13132566
 ] 

Hudson commented on HBASE-4620:
---

Integrated in HBase-0.92 #75 (See 
[https://builds.apache.org/job/HBase-0.92/75/])
HBASE-4620 I broke the build when I submitted HBASE-3581 (Send length of 
the rpc response)

stack : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/ResponseFlag.java


 I broke the build when I submitted HBASE-3581 (Send length of the rpc 
 response)
 ---

 Key: HBASE-4620
 URL: https://issues.apache.org/jira/browse/HBASE-4620
 Project: HBase
  Issue Type: Bug
Reporter: stack
Assignee: stack
Priority: Blocker
 Fix For: 0.92.0

 Attachments: 4620.txt


 Thanks to Ted, Ram and Gao for figuring my messup.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4486) Improve Javadoc for HTableDescriptor


[ 
https://issues.apache.org/jira/browse/HBASE-4486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13132568#comment-13132568
 ] 

Hudson commented on HBASE-4486:
---

Integrated in HBase-0.92 #75 (See 
[https://builds.apache.org/job/HBase-0.92/75/])
HBASE-4486 Improve Javadoc for HTableDescriptor

stack : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java


 Improve Javadoc for HTableDescriptor
 

 Key: HBASE-4486
 URL: https://issues.apache.org/jira/browse/HBASE-4486
 Project: HBase
  Issue Type: Improvement
  Components: client, documentation
Reporter: Akash Ashok
Assignee: Akash Ashok
Priority: Minor
 Fix For: 0.92.0

 Attachments: HBase-4486-v2.patch, HBase-4486-v3.patch, 
 HBase-4486.patch, HTableDescriptor-v2.html, HTableDescriptor.html




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4378) [hbck] Does not complain about regions with startkey==endkey.


[ 
https://issues.apache.org/jira/browse/HBASE-4378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13132571#comment-13132571
 ] 

Hudson commented on HBASE-4378:
---

Integrated in HBase-0.92 #75 (See 
[https://builds.apache.org/job/HBase-0.92/75/])
HBASE-4378 [hbck] Does not complain about regions with startkey==endkey.

stack : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
* 
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java


 [hbck] Does not complain about regions with startkey==endkey.
 -

 Key: HBASE-4378
 URL: https://issues.apache.org/jira/browse/HBASE-4378
 Project: HBase
  Issue Type: Bug
  Components: hbck
Affects Versions: 0.90.4
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Fix For: 0.92.0, 0.90.5

 Attachments: 
 0001-HBASE-4378-hbck-Does-not-complain-about-regions-with.patch


 hbck doesn't seem to complain or have an error condition if there is a region 
 where startkey==endkey.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4430) Disable TestSlabCache and TestSingleSizedCache temporarily to see if these are cause of build box failure though all tests pass


[ 
https://issues.apache.org/jira/browse/HBASE-4430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13132572#comment-13132572
 ] 

Hudson commented on HBASE-4430:
---

Integrated in HBase-0.92 #75 (See 
[https://builds.apache.org/job/HBase-0.92/75/])
HBASE-4430 Disable TestSlabCache and TestSingleSizedCache temporarily to 
see if these are cause of build box failure though all tests pass
HBASE-4430 Disable TestSlabCache and TestSingleSizedCache temporarily to see if 
these are cause of build box failure though all tests pass

stack : 
Files : 
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemActionWatcher.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java

stack : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java


 Disable TestSlabCache and TestSingleSizedCache temporarily to see if these 
 are cause of build box failure though all tests pass
 ---

 Key: HBASE-4430
 URL: https://issues.apache.org/jira/browse/HBASE-4430
 Project: HBase
  Issue Type: Task
  Components: test
Reporter: stack
Assignee: Li Pi
 Fix For: 0.92.0

 Attachments: HBase-4430.txt, HBase-4430v2.txt, HBase-4430v3.txt, 
 TestSlabCache.trace




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4580) Some invalid zk nodes were created when a clean cluster restarts


[ 
https://issues.apache.org/jira/browse/HBASE-4580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13132574#comment-13132574
 ] 

Hudson commented on HBASE-4580:
---

Integrated in HBase-0.92 #75 (See 
[https://builds.apache.org/job/HBase-0.92/75/])
HBASE-4580 Some invalid zk nodes were created when a clean cluster restarts

stack : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/HMaster.java


 Some invalid zk nodes were created when a clean cluster restarts
 

 Key: HBASE-4580
 URL: https://issues.apache.org/jira/browse/HBASE-4580
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.92.0
Reporter: gaojinchao
Assignee: gaojinchao
 Fix For: 0.92.0

 Attachments: HBASE-4580_TrunkV1.patch, HBASE-4580_TrunkV2.patch, 
 HBASE-4580_TrunkV3.patch


 The below logs said that we created a invalid zk node when restarted a 
 cluster.
 it mistakenly believed that the regions belong to a dead server.
 2011-10-11 05:05:29,127 INFO org.apache.hadoop.hbase.master.HMaster: Meta 
 updated status = true
 2011-10-11 05:05:29,127 INFO org.apache.hadoop.hbase.master.HMaster: 
 ROOT/Meta already up-to date with new HRI.
 2011-10-11 05:05:29,151 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
 master:58198-0x132f23a9a38 Creating (or updating) unassigned node for 
 771d63e9327383159553619a4f2dc74f with OFFLINE state
 2011-10-11 05:05:29,161 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
 master:58198-0x132f23a9a38 Creating (or updating) unassigned node for 
 3cf860dd323fe6360f571aeafc129f95 with OFFLINE state
 2011-10-11 05:05:29,170 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
 master:58198-0x132f23a9a38 Creating (or updating) unassigned node for 
 4065350214452a9d5c55243c734bef08 with OFFLINE state
 2011-10-11 05:05:29,178 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
 master:58198-0x132f23a9a38 Creating (or updating) unassigned node for 
 4e81613f82a39fc6e5e89f96e7b3ccc4 with OFFLINE state
 2011-10-11 05:05:29,187 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
 master:58198-0x132f23a9a38 Creating (or updating) unassigned node for 
 e21b9e1545a28953aba0098fda5c9cd9 with OFFLINE state
 2011-10-11 05:05:29,195 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
 master:58198-0x132f23a9a38 Creating (or updating) unassigned node for 
 5cd9f55eecd43d088bbd505f6795131f with OFFLINE state
 2011-10-11 05:05:29,229 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
 master:58198-0x132f23a9a38 Creating (or updating) unassigned node for 
 db5f641452a70b09b85a92970e4198c7 with OFFLINE state
 2011-10-11 05:05:29,237 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
 master:58198-0x132f23a9a38 Creating (or updating) unassigned node for 
 a7b20a653919e7f41bfb2ed349af7d21 with OFFLINE state
 2011-10-11 05:05:29,253 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
 master:58198-0x132f23a9a38 Creating (or updating) unassigned node for 
 c9385619425f737eab1a6624d2e097a8 with OFFLINE state
 // we cleaned all zk nodes.
 2011-10-11 05:05:29,262 INFO 
 org.apache.hadoop.hbase.master.AssignmentManager: Clean cluster startup. 
 Assigning userregions
 2011-10-11 05:05:29,262 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
 master:58198-0x132f23a9a38 Deleting any existing unassigned nodes
 2011-10-11 05:05:29,367 INFO 
 org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning 9 region(s) 
 across 1 server(s), retainAssignment=true
 2011-10-11 05:05:29,369 DEBUG 
 org.apache.hadoop.hbase.master.AssignmentManager: Timeout-on-RIT=9000
 2011-10-11 05:05:29,369 DEBUG 
 org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning 9 region(s) 
 to C3S3,54366,1318323920153
 2011-10-11 05:05:29,369 INFO 
 org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning done
 2011-10-11 05:05:29,371 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
 master:58198-0x132f23a9a38 Async create of unassigned node for 
 771d63e9327383159553619a4f2dc74f with OFFLINE state
 2011-10-11 05:05:29,371 INFO org.apache.hadoop.hbase.master.HMaster: Master 
 has completed initialization
 2011-10-11 05:05:29,371 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
 master:58198-0x132f23a9a38 Async create of unassigned node for 
 3cf860dd323fe6360f571aeafc129f95 with OFFLINE state
 2011-10-11 05:05:29,371 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
 master:58198-0x132f23a9a38 Async create of unassigned node for 
 4065350214452a9d5c55243c734bef08 with OFFLINE state
 2011-10-11 05:05:29,371 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:

[jira] [Commented] (HBASE-4604) hbase.client.TestHTablePool could start a single cluster instead of one per method


[ 
https://issues.apache.org/jira/browse/HBASE-4604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13132569#comment-13132569
 ] 

Hudson commented on HBASE-4604:
---

Integrated in HBase-0.92 #75 (See 
[https://builds.apache.org/job/HBase-0.92/75/])
HBASE-4604 hbase.client.TestHTablePool could start a single cluster instead 
of one per method

stack : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/client/TestHTablePool.java


 hbase.client.TestHTablePool could start a single cluster instead of one per 
 method
 --

 Key: HBASE-4604
 URL: https://issues.apache.org/jira/browse/HBASE-4604
 Project: HBase
  Issue Type: Improvement
  Components: test
Affects Versions: 0.92.0
 Environment: all
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
 Fix For: 0.92.0

 Attachments: 20111017_4604_TestHTablePool.patch


 This tests starts/stops one cluster per method, while is would be possible to 
 start it for the whole class.
 Using a single cluster allows the test to take 20s instead of 175s (after 
 HBASE-4603, much more before).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4510) Check and workaround usage of internal HDFS APIs in HBase


[ 
https://issues.apache.org/jira/browse/HBASE-4510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13132570#comment-13132570
 ] 

Hudson commented on HBASE-4510:
---

Integrated in HBase-0.92 #75 (See 
[https://builds.apache.org/job/HBase-0.92/75/])
HBASE-4510 Check and workaround usage of internal HDFS APIs in HBase

stack : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java


 Check and workaround usage of internal HDFS APIs in HBase
 -

 Key: HBASE-4510
 URL: https://issues.apache.org/jira/browse/HBASE-4510
 Project: HBase
  Issue Type: Task
Affects Versions: 0.94.0
Reporter: Harsh J
Assignee: Harsh J
 Fix For: 0.92.0

 Attachments: HBASE-4510.patch


 HBase isn't seemingly compiling anymore on 0.23 after the HDFS-1620 naming 
 refactorings were carried out.
 Two solutions:
 * We use new classnames. This breaks HBase's backward compatibility with 
 older Hadoop releases (is that a concern with future releases?)
 * HBase gets its own sets of constants as the upstream one is not marked for 
 public usage. This needs a little more maintenance on HBases' side.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4595) HFilePrettyPrinter Scanned kv count always 0

2011-10-21 Thread ramkrishna.s.vasudevan (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13132573#comment-13132573
 ] 

Hudson commented on HBASE-4595:
---

Integrated in HBase-0.92 #75 (See 
[https://builds.apache.org/job/HBase-0.92/75/])
HBASE-4595 HFilePrettyPrinter Scanned kv count always 0

stack : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java


 HFilePrettyPrinter Scanned kv count always 0
 

 Key: HBASE-4595
 URL: https://issues.apache.org/jira/browse/HBASE-4595
 Project: HBase
  Issue Type: Bug
  Components: io
Affects Versions: 0.92.0, 0.94.0, 0.92.1
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
 Fix For: 0.92.0

 Attachments: HBASE-4595.patch


 The count variable used to print the Scanned kv count is never 
 incremented.
 A local count variable in scanKeysValues() method is updated instead.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4219) Add Per-Column Family Metrics


[ 
https://issues.apache.org/jira/browse/HBASE-4219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13132594#comment-13132594
 ] 

ramkrishna.s.vasudevan commented on HBASE-4219:
---

Committed the addendum. Thanks Gao

 Add Per-Column Family Metrics
 -

 Key: HBASE-4219
 URL: https://issues.apache.org/jira/browse/HBASE-4219
 Project: HBase
  Issue Type: New Feature
Affects Versions: 0.92.0
Reporter: Nicolas Spiegelberg
Assignee: David Goode
 Fix For: 0.92.0

 Attachments: 4219-v2.txt, 4219-v3.txt, HBASE-4219-v4.patch, 
 HBASE-4219-v5.patch, HBASE-4219_fixtestcasefail.patch, 
 HBASE-4219_percfmetrics_1.patch


 Right now, we have region server level statistics.  However, the read/write 
 flow varies a lot based on the column family involved.  We should add 
 dynamic, per column family metrics to JMX so we can track each column family 
 individually.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4644) LoadIncrementalHFiles ignores additional configurations

2011-10-21 Thread Alexey Zotov (Created) (JIRA)

LoadIncrementalHFiles ignores additional configurations
---

 Key: HBASE-4644
 URL: https://issues.apache.org/jira/browse/HBASE-4644
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.3
 Environment: Centos 5.5, Cloudera cdh3u1 distribution.
Reporter: Alexey Zotov


Method run ignores configuration if LoadIncrementalHFiles was runned from 
user's application:
{code}
LoadIncrementalHFiles hFilesMergeTask = new LoadIncrementalHFiles(conf);
ToolRunner.run(hFilesMergeTask, args);
{code}

Quick fix:
{code}
--- LoadIncrementalHFiles.java  2011-07-18 08:20:38.0 +0400
+++ LoadIncrementalHFiles.java  2011-10-19 18:08:31.228972054 +0400
@@ -447,14 +446,20 @@
 if (!tableExists) this.createTable(tableName,dirPath);
 
 Path hfofDir = new Path(dirPath);
-HTable table = new HTable(tableName);
+HTable table;
+Configuration configuration = getConf();
+if (configuration != null) {
+  table = new HTable(configuration, tableName);
+} else {
+  table = new HTable(tableName);
+}
 
 doBulkLoad(hfofDir, table);
 return 0;
   }
{code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4644) LoadIncrementalHFiles ignores additional configurations

2011-10-21 Thread Alexey Zotov (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexey Zotov updated HBASE-4644:


Description: 
Method run ignores configuration, which was passed in as constructor argument:
{code}
LoadIncrementalHFiles hFilesMergeTask = new LoadIncrementalHFiles(conf);
ToolRunner.run(hFilesMergeTask, args); 
{code}

This happens because HTable creation (_new HTable(tableName);_ in 
LoadIncrementalHFiles.run() method) skips existing configuration and tries to 
create a new one for HTable. If there is no hbase-site.xml in classpath, 
previously loaded properties (via -conf configuration file) will be missed. 

Quick fix:
{code}
--- LoadIncrementalHFiles.java  2011-07-18 08:20:38.0 +0400
+++ LoadIncrementalHFiles.java  2011-10-19 18:08:31.228972054 +0400
@@ -447,14 +446,20 @@
 if (!tableExists) this.createTable(tableName,dirPath);
 
 Path hfofDir = new Path(dirPath);
-HTable table = new HTable(tableName);
+HTable table;
+Configuration configuration = getConf();
+if (configuration != null) {
+  table = new HTable(configuration, tableName);
+} else {
+  table = new HTable(tableName);
+}
 
 doBulkLoad(hfofDir, table);
 return 0;
   }
{code}

  was:
Method run ignores configuration if LoadIncrementalHFiles was runned from 
user's application:
{code}
LoadIncrementalHFiles hFilesMergeTask = new LoadIncrementalHFiles(conf);
ToolRunner.run(hFilesMergeTask, args);
{code}

Quick fix:
{code}
--- LoadIncrementalHFiles.java  2011-07-18 08:20:38.0 +0400
+++ LoadIncrementalHFiles.java  2011-10-19 18:08:31.228972054 +0400
@@ -447,14 +446,20 @@
 if (!tableExists) this.createTable(tableName,dirPath);
 
 Path hfofDir = new Path(dirPath);
-HTable table = new HTable(tableName);
+HTable table;
+Configuration configuration = getConf();
+if (configuration != null) {
+  table = new HTable(configuration, tableName);
+} else {
+  table = new HTable(tableName);
+}
 
 doBulkLoad(hfofDir, table);
 return 0;
   }
{code}

   Priority: Minor  (was: Major)

 LoadIncrementalHFiles ignores additional configurations
 ---

 Key: HBASE-4644
 URL: https://issues.apache.org/jira/browse/HBASE-4644
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.3
 Environment: Centos 5.5, Cloudera cdh3u1 distribution.
Reporter: Alexey Zotov
Priority: Minor
  Labels: Configuration, LoadIncrementalHFiles

 Method run ignores configuration, which was passed in as constructor argument:
 {code}
 LoadIncrementalHFiles hFilesMergeTask = new LoadIncrementalHFiles(conf);
 ToolRunner.run(hFilesMergeTask, args); 
 {code}
 This happens because HTable creation (_new HTable(tableName);_ in 
 LoadIncrementalHFiles.run() method) skips existing configuration and tries to 
 create a new one for HTable. If there is no hbase-site.xml in classpath, 
 previously loaded properties (via -conf configuration file) will be missed. 
 Quick fix:
 {code}
 --- LoadIncrementalHFiles.java2011-07-18 08:20:38.0 +0400
 +++ LoadIncrementalHFiles.java2011-10-19 18:08:31.228972054 +0400
 @@ -447,14 +446,20 @@
  if (!tableExists) this.createTable(tableName,dirPath);
  
  Path hfofDir = new Path(dirPath);
 -HTable table = new HTable(tableName);
 +HTable table;
 +Configuration configuration = getConf();
 +if (configuration != null) {
 +  table = new HTable(configuration, tableName);
 +} else {
 +  table = new HTable(tableName);
 +}
  
  doBulkLoad(hfofDir, table);
  return 0;
}
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-4644) LoadIncrementalHFiles ignores additional configurations

2011-10-21 Thread Ted Yu (Assigned) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu reassigned HBASE-4644:
-

Assignee: Alexey Zotov

 LoadIncrementalHFiles ignores additional configurations
 ---

 Key: HBASE-4644
 URL: https://issues.apache.org/jira/browse/HBASE-4644
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.3
 Environment: Centos 5.5, Cloudera cdh3u1 distribution.
Reporter: Alexey Zotov
Assignee: Alexey Zotov
Priority: Minor
  Labels: Configuration, LoadIncrementalHFiles

 Method run ignores configuration, which was passed in as constructor argument:
 {code}
 LoadIncrementalHFiles hFilesMergeTask = new LoadIncrementalHFiles(conf);
 ToolRunner.run(hFilesMergeTask, args); 
 {code}
 This happens because HTable creation (_new HTable(tableName);_ in 
 LoadIncrementalHFiles.run() method) skips existing configuration and tries to 
 create a new one for HTable. If there is no hbase-site.xml in classpath, 
 previously loaded properties (via -conf configuration file) will be missed. 
 Quick fix:
 {code}
 --- LoadIncrementalHFiles.java2011-07-18 08:20:38.0 +0400
 +++ LoadIncrementalHFiles.java2011-10-19 18:08:31.228972054 +0400
 @@ -447,14 +446,20 @@
  if (!tableExists) this.createTable(tableName,dirPath);
  
  Path hfofDir = new Path(dirPath);
 -HTable table = new HTable(tableName);
 +HTable table;
 +Configuration configuration = getConf();
 +if (configuration != null) {
 +  table = new HTable(configuration, tableName);
 +} else {
 +  table = new HTable(tableName);
 +}
  
  doBulkLoad(hfofDir, table);
  return 0;
}
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4644) LoadIncrementalHFiles ignores additional configurations


[ 
https://issues.apache.org/jira/browse/HBASE-4644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13132663#comment-13132663
 ] 

Ted Yu commented on HBASE-4644:
---

@Alexey:
Can you put the suggested fix into a patch file ?

If the patch for latest 0.90 branch doesn't apply cleanly to 0.92/TRUNK, 
separate patches should be attached.

 LoadIncrementalHFiles ignores additional configurations
 ---

 Key: HBASE-4644
 URL: https://issues.apache.org/jira/browse/HBASE-4644
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.3
 Environment: Centos 5.5, Cloudera cdh3u1 distribution.
Reporter: Alexey Zotov
Assignee: Alexey Zotov
Priority: Minor
  Labels: Configuration, LoadIncrementalHFiles

 Method run ignores configuration, which was passed in as constructor argument:
 {code}
 LoadIncrementalHFiles hFilesMergeTask = new LoadIncrementalHFiles(conf);
 ToolRunner.run(hFilesMergeTask, args); 
 {code}
 This happens because HTable creation (_new HTable(tableName);_ in 
 LoadIncrementalHFiles.run() method) skips existing configuration and tries to 
 create a new one for HTable. If there is no hbase-site.xml in classpath, 
 previously loaded properties (via -conf configuration file) will be missed. 
 Quick fix:
 {code}
 --- LoadIncrementalHFiles.java2011-07-18 08:20:38.0 +0400
 +++ LoadIncrementalHFiles.java2011-10-19 18:08:31.228972054 +0400
 @@ -447,14 +446,20 @@
  if (!tableExists) this.createTable(tableName,dirPath);
  
  Path hfofDir = new Path(dirPath);
 -HTable table = new HTable(tableName);
 +HTable table;
 +Configuration configuration = getConf();
 +if (configuration != null) {
 +  table = new HTable(configuration, tableName);
 +} else {
 +  table = new HTable(tableName);
 +}
  
  doBulkLoad(hfofDir, table);
  return 0;
}
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4219) Add Per-Column Family Metrics

2011-10-21 Thread Ted Yu (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4219:
--

  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

 Add Per-Column Family Metrics
 -

 Key: HBASE-4219
 URL: https://issues.apache.org/jira/browse/HBASE-4219
 Project: HBase
  Issue Type: New Feature
Affects Versions: 0.92.0
Reporter: Nicolas Spiegelberg
Assignee: David Goode
 Fix For: 0.92.0

 Attachments: 4219-v2.txt, 4219-v3.txt, HBASE-4219-v4.patch, 
 HBASE-4219-v5.patch, HBASE-4219_fixtestcasefail.patch, 
 HBASE-4219_percfmetrics_1.patch


 Right now, we have region server level statistics.  However, the read/write 
 flow varies a lot based on the column family involved.  We should add 
 dynamic, per column family metrics to JMX so we can track each column family 
 individually.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4219) Add Per-Column Family Metrics

2011-10-21 Thread Ted Yu (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4219:
--

Fix Version/s: (was: 0.92.0)
   0.94.0

 Add Per-Column Family Metrics
 -

 Key: HBASE-4219
 URL: https://issues.apache.org/jira/browse/HBASE-4219
 Project: HBase
  Issue Type: New Feature
Affects Versions: 0.92.0
Reporter: Nicolas Spiegelberg
Assignee: David Goode
 Fix For: 0.94.0

 Attachments: 4219-v2.txt, 4219-v3.txt, HBASE-4219-v4.patch, 
 HBASE-4219-v5.patch, HBASE-4219_fixtestcasefail.patch, 
 HBASE-4219_percfmetrics_1.patch


 Right now, we have region server level statistics.  However, the read/write 
 flow varies a lot based on the column family involved.  We should add 
 dynamic, per column family metrics to JMX so we can track each column family 
 individually.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4643) Consider reverting HBASE-451 (change HRI to remove HTD) in 0.92

2011-10-21 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13132684#comment-13132684
 ] 

Ted Yu commented on HBASE-4643:
---

So HBASE-4389 would be abandoned ?
I think more effort should be put on that JIRA.

 Consider reverting HBASE-451 (change HRI to remove HTD) in 0.92
 ---

 Key: HBASE-4643
 URL: https://issues.apache.org/jira/browse/HBASE-4643
 Project: HBase
  Issue Type: Brainstorming
Affects Versions: 0.92.0
Reporter: Todd Lipcon
 Attachments: revert.txt


 I've been chatting with some folks recently about this thought: it seems 
 like, if you enumerate the larger changes in 0.92, this is probably the one 
 that is the most destabilizing that hasn't been through a lot of baking 
 yet. You can see this evidenced by the very high number of followup commits 
 it generated: looks like somewhere around 15 of them, plus some bugs still 
 open.
 I've done a patch to revert this and the related followup changes on the 0.92 
 branch. Do we want to consider doing this?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4645) Edits Log recovery losing data across column families

2011-10-21 Thread Amitanand Aiyer (Created) (JIRA)

Edits Log recovery losing data across column families
-

 Key: HBASE-4645
 URL: https://issues.apache.org/jira/browse/HBASE-4645
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.89.20100924, 0.92.0
Reporter: Amitanand Aiyer
Assignee: Amitanand Aiyer


There is a data loss happening (for some of the column families) when we do the 
replay logs.

The bug seems to be from the fact that during replay-logs we only choose to 
replay
the logs from the maximumSequenceID across *ALL* the stores. This is wrong. If a
column family is ahead of others (because the crash happened before all the 
column
families were flushed), then we lose data for the column families that have not 
yet
caught up.

The correct logic for replay should begin the replay from the minimum across the
maximum in each store. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4645) Edits Log recovery losing data across column families

[
https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13132726#comment-13132726
]

jirapos...@reviews.apache.org commented on HBASE-4645:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2524/
---

Review request for Ted Yu, Michael Stack, Jonathan Gray, Lars Hofhansl,
Amitanand Aiyer, Kannan Muthukkaruppan, Karthik Ranganathan, and Nicolas
Spiegelberg.

Summary
---

There is a data loss happening (for some of the column families) when we do the
replay logs.

The bug seems to be from the fact that during replay-logs we only choose to
replay
the logs from the maximumSequenceID across ALL the stores. This is wrong. If a
column family is ahead of others (because the crash happened before all the
column
families were flushed), then we lose data for the column families that have not
yet
caught up.

The correct logic for replay should begin the replay from the minimum across the
maximum in each store.

This addresses bug hbase-4645.
https://issues.apache.org/jira/browse/hbase-4645

Diffs
-

src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8c32839

Diff: https://reviews.apache.org/r/2524/diff

Testing
---

Initial patch. v1.

mvn test (running).

TBD: add a test case to repro the issue and make sure it fixes.

Thanks,

Amitanand

Edits Log recovery losing data across column families
-

Key: HBASE-4645
URL: https://issues.apache.org/jira/browse/HBASE-4645
Project: HBase
Issue Type: Bug
Affects Versions: 0.89.20100924, 0.92.0
Reporter: Amitanand Aiyer
Assignee: Amitanand Aiyer

There is a data loss happening (for some of the column families) when we do
the replay logs.
The bug seems to be from the fact that during replay-logs we only choose to
replay
the logs from the maximumSequenceID across *ALL* the stores. This is wrong.
If a
column family is ahead of others (because the crash happened before all the
column
families were flushed), then we lose data for the column families that have
not yet
caught up.
The correct logic for replay should begin the replay from the minimum across
the
maximum in each store.

[jira] [Commented] (HBASE-4485) Eliminate window of missing Data

2011-10-21 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13132733#comment-13132733
 ] 

jirapos...@reviews.apache.org commented on HBASE-4485:
--



bq.  On 2011-10-21 06:13:51, Lars Hofhansl wrote:
bq.   src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java, line 
645
bq.   https://reviews.apache.org/r/2481/diff/1/?file=51676#file51676line645
bq.  
bq.   Can these be private?
bq.   volatile because you did not want synchronous in the MemstoreScanner 
constructor?

actually, I had this volatile because the kvset in the Memstore is volatile.

But, on second thought, it seems like we can get rid of that. While the kvset 
and snapshot in Memstore 
can be accessed from different threads (running different Memstorescanners). 
The reference in the
MemstoreScanner should only be used in the Thread that is performing the scan.

Will update the diff and send it again.


bq.  On 2011-10-21 06:13:51, Lars Hofhansl wrote:
bq.   src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java, line 
712
bq.   https://reviews.apache.org/r/2481/diff/1/?file=51676#file51676line712
bq.  
bq.   Cool.
bq.   
bq.   Only concern is that while the scanner exists we may need more 
memory than before.

you are right. 

Are we guaranteed to do a seek() only once? If so, we probably can reset 
kvset/snapshot to null, 
once we have evaluated kvTail and snapshotTail.


bq.  On 2011-10-21 06:13:51, Lars Hofhansl wrote:
bq.   src/main/java/org/apache/hadoop/hbase/regionserver/Store.java, line 1335
bq.   https://reviews.apache.org/r/2481/diff/1/?file=51677#file51677line1335
bq.  
bq.   Why can you move this up?

Here is my understanding/assumption:
https://issues.apache.org/jira/browse/HBASE-4485?focusedCommentId=13131844page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13131844

Let me know if that is wrong :-)


- Amitanand


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2481/#review2736
---


On 2011-10-20 19:13:56, Amitanand Aiyer wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2481/
bq.  ---
bq.  
bq.  (Updated 2011-10-20 19:13:56)
bq.  
bq.  
bq.  Review request for Ted Yu, Michael Stack, Jonathan Gray, Lars Hofhansl, 
Amitanand Aiyer, Kannan Muthukkaruppan, Karthik Ranganathan, and Nicolas 
Spiegelberg.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Part of the 2856 diff split into 3 parts for easier review
bq.  
bq.  The first part is v6 of the patch submitted to:
bq.  https://reviews.apache.org/r/2224/
bq.  
bq.  This is the fix for HBase-4485
bq.  
bq.  
bq.  This addresses bug hbase-4485.
bq.  https://issues.apache.org/jira/browse/hbase-4485
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 7761c42 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 
f5b5c4c 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java 34263e4 
bq.  
bq.  Diff: https://reviews.apache.org/r/2481/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  running mvn test with all 3 patches together.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Amitanand
bq.  
bq.



 Eliminate window of missing Data
 

 Key: HBASE-4485
 URL: https://issues.apache.org/jira/browse/HBASE-4485
 Project: HBase
  Issue Type: Sub-task
Reporter: Amitanand Aiyer
Assignee: Amitanand Aiyer
 Fix For: 0.94.0

 Attachments: 4485-v1.diff, 4485-v2.diff, 4485-v3.diff, 4485-v4.diff, 
 repro_bug-4485.diff


 After incorporating v11 of the 2856 fix, we discovered that we are still 
 having some ACID violations.
 This time, however, the problem is not about including newer updates; but, 
 about missing older updates
 that should be including. 
 Here is what seems to be happening.
 There is a race condition in the StoreScanner.getScanners()
   private ListKeyValueScanner getScanners(Scan scan,
   final NavigableSetbyte[] columns) throws IOException {
 // First the store file scanners
 ListStoreFileScanner sfScanners = StoreFileScanner
   .getScannersForStoreFiles(store.getStorefiles(), cacheBlocks,
 isGet, false);
 ListKeyValueScanner scanners =
   new ArrayListKeyValueScanner(sfScanners.size()+1);
 // include only those scan files which pass all filters
 for (StoreFileScanner sfs : sfScanners) {
   if (sfs.shouldSeek(scan, columns)) {

[jira] [Commented] (HBASE-4485) Eliminate window of missing Data

2011-10-21 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13132740#comment-13132740
 ] 

jirapos...@reviews.apache.org commented on HBASE-4485:
--



bq.  On 2011-10-21 06:13:51, Lars Hofhansl wrote:
bq.   src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java, 
line 216
bq.   https://reviews.apache.org/r/2481/diff/1/?file=51678#file51678line216
bq.  
bq.   I'm not usually a fan of instanceof - unless necessary of course.
bq.   Why did you need to fold this into one loop?

The reason for having one loop is because we want to get the
Scanners for the storefiles and the scanners for memstore at
the same time. This is required for ensuring that the view of
the StoreScanner is consistent with the state of the store,
at some point in the execution history.

I agree, instance of is kinda ugly. Will try to get rid of that.


- Amitanand


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2481/#review2736
---


On 2011-10-20 19:13:56, Amitanand Aiyer wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2481/
bq.  ---
bq.  
bq.  (Updated 2011-10-20 19:13:56)
bq.  
bq.  
bq.  Review request for Ted Yu, Michael Stack, Jonathan Gray, Lars Hofhansl, 
Amitanand Aiyer, Kannan Muthukkaruppan, Karthik Ranganathan, and Nicolas 
Spiegelberg.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Part of the 2856 diff split into 3 parts for easier review
bq.  
bq.  The first part is v6 of the patch submitted to:
bq.  https://reviews.apache.org/r/2224/
bq.  
bq.  This is the fix for HBase-4485
bq.  
bq.  
bq.  This addresses bug hbase-4485.
bq.  https://issues.apache.org/jira/browse/hbase-4485
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 7761c42 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 
f5b5c4c 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java 34263e4 
bq.  
bq.  Diff: https://reviews.apache.org/r/2481/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  running mvn test with all 3 patches together.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Amitanand
bq.  
bq.



 Eliminate window of missing Data
 

 Key: HBASE-4485
 URL: https://issues.apache.org/jira/browse/HBASE-4485
 Project: HBase
  Issue Type: Sub-task
Reporter: Amitanand Aiyer
Assignee: Amitanand Aiyer
 Fix For: 0.94.0

 Attachments: 4485-v1.diff, 4485-v2.diff, 4485-v3.diff, 4485-v4.diff, 
 repro_bug-4485.diff


 After incorporating v11 of the 2856 fix, we discovered that we are still 
 having some ACID violations.
 This time, however, the problem is not about including newer updates; but, 
 about missing older updates
 that should be including. 
 Here is what seems to be happening.
 There is a race condition in the StoreScanner.getScanners()
   private ListKeyValueScanner getScanners(Scan scan,
   final NavigableSetbyte[] columns) throws IOException {
 // First the store file scanners
 ListStoreFileScanner sfScanners = StoreFileScanner
   .getScannersForStoreFiles(store.getStorefiles(), cacheBlocks,
 isGet, false);
 ListKeyValueScanner scanners =
   new ArrayListKeyValueScanner(sfScanners.size()+1);
 // include only those scan files which pass all filters
 for (StoreFileScanner sfs : sfScanners) {
   if (sfs.shouldSeek(scan, columns)) {
 scanners.add(sfs);
   }
 }
 // Then the memstore scanners
 if (this.store.memstore.shouldSeek(scan)) {
   scanners.addAll(this.store.memstore.getScanners());
 }
 return scanners;
   }
 If for example there is a call to Store.updateStorefiles() that happens 
 between
 the store.getStorefiles() and this.store.memstore.getScanners(); then
 it is possible that there was a new HFile created, that is not seen by the
 StoreScanner, and the data is not present in the Memstore.snapshot either.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4645) Edits Log recovery losing data across column families

2011-10-21 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13132741#comment-13132741
]

jirapos...@reviews.apache.org commented on HBASE-4645:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2524/#review2745
---

Amit-- thanks for getting a fix out soon. Comments inline. As you mentioned, I
am assuming you are planning on adding a test as part of the commit itself.

src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
https://reviews.apache.org/r/2524/#comment6180

Comment:

Get minimum seqid across all store

should read:

Get minimum of the max seq id for each store.

src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
https://reviews.apache.org/r/2524/#comment6181

trailing white space

src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
https://reviews.apache.org/r/2524/#comment6183

yes, makes sense as a future optimization.

src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
https://reviews.apache.org/r/2524/#comment6184

as Madhu pointed out in a later email, if now edits were were there for
this region, then this function will simply return minSeqId back as its return
value.

Like before, we should also track the maxSeqId for all stores in the above
loop.

And, then change this line to something like:

maxSeqId = Math.max(maxSeqId, replayRecoveredEditsIfAny(...));

- Kannan

On 2011-10-21 15:01:11, Amitanand Aiyer wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2524/
bq. ---
bq.
bq. (Updated 2011-10-21 15:01:11)
bq.
bq.
bq. Review request for Ted Yu, Michael Stack, Jonathan Gray, Lars Hofhansl,
Amitanand Aiyer, Kannan Muthukkaruppan, Karthik Ranganathan, and Nicolas
Spiegelberg.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. There is a data loss happening (for some of the column families) when we
do the replay logs.
bq.
bq. The bug seems to be from the fact that during replay-logs we only choose
to replay
bq. the logs from the maximumSequenceID across ALL the stores. This is wrong.
If a
bq. column family is ahead of others (because the crash happened before all
the column
bq. families were flushed), then we lose data for the column families that
have not yet
bq. caught up.
bq.
bq. The correct logic for replay should begin the replay from the minimum
across the
bq. maximum in each store.
bq.
bq.
bq. This addresses bug hbase-4645.
bq. https://issues.apache.org/jira/browse/hbase-4645
bq.
bq.
bq. Diffs
bq. -
bq.
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8c32839
bq.
bq. Diff: https://reviews.apache.org/r/2524/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Initial patch. v1.
bq.
bq. mvn test (running).
bq.
bq. TBD: add a test case to repro the issue and make sure it fixes.
bq.
bq.
bq. Thanks,
bq.
bq. Amitanand
bq.
bq.

Edits Log recovery losing data across column families
-

Key: HBASE-4645
URL: https://issues.apache.org/jira/browse/HBASE-4645
Project: HBase
Issue Type: Bug
Affects Versions: 0.89.20100924, 0.92.0
Reporter: Amitanand Aiyer
Assignee: Amitanand Aiyer

[jira] [Commented] (HBASE-4645) Edits Log recovery losing data across column families

2011-10-21 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13132745#comment-13132745
]

jirapos...@reviews.apache.org commented on HBASE-4645:
--

bq. On 2011-10-21 15:20:37, Kannan Muthukkaruppan wrote:
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 424
bq. https://reviews.apache.org/r/2524/diff/1/?file=52115#file52115line424
bq.
bq. as Madhu pointed out in a later email, if now edits were were there
for this region, then this function will simply return minSeqId back as its
return value.
bq.
bq. Like before, we should also track the maxSeqId for all stores in the
above loop.
bq.
bq. And, then change this line to something like:
bq.
bq. maxSeqId = Math.max(maxSeqId, replayRecoveredEditsIfAny(...));
bq.
bq.

typo: s/if now edits were/if no edits were

- Kannan

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2524/#review2745
---

Edits Log recovery losing data across column families
-

Key: HBASE-4645
URL: https://issues.apache.org/jira/browse/HBASE-4645
Project: HBase
Issue Type: Bug
Affects Versions: 0.89.20100924, 0.92.0
Reporter: Amitanand Aiyer
Assignee: Amitanand Aiyer

[jira] [Commented] (HBASE-4644) LoadIncrementalHFiles ignores additional configurations

2011-10-21 Thread Todd Lipcon (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13132746#comment-13132746
 ] 

Todd Lipcon commented on HBASE-4644:


Dup of HBASE-3714?

 LoadIncrementalHFiles ignores additional configurations
 ---

 Key: HBASE-4644
 URL: https://issues.apache.org/jira/browse/HBASE-4644
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.3
 Environment: Centos 5.5, Cloudera cdh3u1 distribution.
Reporter: Alexey Zotov
Assignee: Alexey Zotov
Priority: Minor
  Labels: Configuration, LoadIncrementalHFiles

 Method run ignores configuration, which was passed in as constructor argument:
 {code}
 LoadIncrementalHFiles hFilesMergeTask = new LoadIncrementalHFiles(conf);
 ToolRunner.run(hFilesMergeTask, args); 
 {code}
 This happens because HTable creation (_new HTable(tableName);_ in 
 LoadIncrementalHFiles.run() method) skips existing configuration and tries to 
 create a new one for HTable. If there is no hbase-site.xml in classpath, 
 previously loaded properties (via -conf configuration file) will be missed. 
 Quick fix:
 {code}
 --- LoadIncrementalHFiles.java2011-07-18 08:20:38.0 +0400
 +++ LoadIncrementalHFiles.java2011-10-19 18:08:31.228972054 +0400
 @@ -447,14 +446,20 @@
  if (!tableExists) this.createTable(tableName,dirPath);
  
  Path hfofDir = new Path(dirPath);
 -HTable table = new HTable(tableName);
 +HTable table;
 +Configuration configuration = getConf();
 +if (configuration != null) {
 +  table = new HTable(configuration, tableName);
 +} else {
 +  table = new HTable(tableName);
 +}
  
  doBulkLoad(hfofDir, table);
  return 0;
}
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4645) Edits Log recovery losing data across column families


[ 
https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13132794#comment-13132794
 ] 

Ted Yu commented on HBASE-4645:
---

The TODO should refer to HBASE-3149
I will add a cross reference in HBASE-3149 to this JIRA.

Nice job, Amit.

 Edits Log recovery losing data across column families
 -

 Key: HBASE-4645
 URL: https://issues.apache.org/jira/browse/HBASE-4645
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.89.20100924, 0.92.0
Reporter: Amitanand Aiyer
Assignee: Amitanand Aiyer

 There is a data loss happening (for some of the column families) when we do 
 the replay logs.
 The bug seems to be from the fact that during replay-logs we only choose to 
 replay
 the logs from the maximumSequenceID across *ALL* the stores. This is wrong. 
 If a
 column family is ahead of others (because the crash happened before all the 
 column
 families were flushed), then we lose data for the column families that have 
 not yet
 caught up.
 The correct logic for replay should begin the replay from the minimum across 
 the
 maximum in each store. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4485) Eliminate window of missing Data


[ 
https://issues.apache.org/jira/browse/HBASE-4485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13132809#comment-13132809
 ] 

Ted Yu commented on HBASE-4485:
---

In order to avoid using instanceof in useRWCC(), we have at least two options:

1. create a tuple for each KeyValueScanner discovered with boolean flag 
indicating whether the KeyValueScanner is StoreFileScanner or not.
Thus scanners would be ListKeyValueScannerWrapper

2. Make scanners a MapKeyValueScanner, Boolean

 Eliminate window of missing Data
 

 Key: HBASE-4485
 URL: https://issues.apache.org/jira/browse/HBASE-4485
 Project: HBase
  Issue Type: Sub-task
Reporter: Amitanand Aiyer
Assignee: Amitanand Aiyer
 Fix For: 0.94.0

 Attachments: 4485-v1.diff, 4485-v2.diff, 4485-v3.diff, 4485-v4.diff, 
 repro_bug-4485.diff


 After incorporating v11 of the 2856 fix, we discovered that we are still 
 having some ACID violations.
 This time, however, the problem is not about including newer updates; but, 
 about missing older updates
 that should be including. 
 Here is what seems to be happening.
 There is a race condition in the StoreScanner.getScanners()
   private ListKeyValueScanner getScanners(Scan scan,
   final NavigableSetbyte[] columns) throws IOException {
 // First the store file scanners
 ListStoreFileScanner sfScanners = StoreFileScanner
   .getScannersForStoreFiles(store.getStorefiles(), cacheBlocks,
 isGet, false);
 ListKeyValueScanner scanners =
   new ArrayListKeyValueScanner(sfScanners.size()+1);
 // include only those scan files which pass all filters
 for (StoreFileScanner sfs : sfScanners) {
   if (sfs.shouldSeek(scan, columns)) {
 scanners.add(sfs);
   }
 }
 // Then the memstore scanners
 if (this.store.memstore.shouldSeek(scan)) {
   scanners.addAll(this.store.memstore.getScanners());
 }
 return scanners;
   }
 If for example there is a call to Store.updateStorefiles() that happens 
 between
 the store.getStorefiles() and this.store.memstore.getScanners(); then
 it is possible that there was a new HFile created, that is not seen by the
 StoreScanner, and the data is not present in the Memstore.snapshot either.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4469) Avoid top row seek by looking up ROWCOL bloomfilter

2011-10-21 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liyin Tang updated HBASE-4469:
--

Summary: Avoid top row seek by looking up ROWCOL bloomfilter  (was: Avoid 
top row seek by looking up bloomfilter)

 Avoid top row seek by looking up ROWCOL bloomfilter
 ---

 Key: HBASE-4469
 URL: https://issues.apache.org/jira/browse/HBASE-4469
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang
 Fix For: 0.94.0

 Attachments: HBASE-4469_1.patch


 The problem is that when seeking for the row/col in the hfile, we will go to 
 top of the row in order to check for row delete marker (delete family). 
 However, if the bloomfilter is enabled for the column family, then if a 
 delete family operation is done on a row, the row is already being added to 
 bloomfilter. We can take advantage of this factor to avoid seeking to the top 
 of row.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4641) Block cache can be mistakenly instantiated on Master

2011-10-21 Thread Jonathan Gray (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13132854#comment-13132854
 ] 

Jonathan Gray commented on HBASE-4641:
--

Thanks Ted.  You see any others?

 Block cache can be mistakenly instantiated on Master
 

 Key: HBASE-4641
 URL: https://issues.apache.org/jira/browse/HBASE-4641
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.92.0, 0.94.0
Reporter: Jonathan Gray
Assignee: Jonathan Gray
Priority: Critical
 Fix For: 0.92.0

 Attachments: HBASE-4641-v1.patch, HBASE-4641-v2.patch


 After changes in the block cache instantiation over in HBASE-4422, it looks 
 like the HMaster can now end up with a block cache instantiated.  Not a huge 
 deal but prevents the process from shutting down properly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4643) Consider reverting HBASE-451 (change HRI to remove HTD) in 0.92

2011-10-21 Thread Jonathan Gray (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13132865#comment-13132865
 ] 

Jonathan Gray commented on HBASE-4643:
--

I've had a few pretty horrible experiences moving an 0.90 cluster to 0.92 so 
far, so I agree that this is definitely the most unbaked part of 0.92.

Now that I've got 92 clusters, I'm going to have to figure out a reverting plan 
for them if we back this out now.  It will also become a barrier between 0.92 
and 0.94 which will make my life difficult as well (since we have been pulling 
94 changes into a local 92 branch).

I'd like to see if Stack's next changes do the trick before abandoning this.

 Consider reverting HBASE-451 (change HRI to remove HTD) in 0.92
 ---

 Key: HBASE-4643
 URL: https://issues.apache.org/jira/browse/HBASE-4643
 Project: HBase
  Issue Type: Brainstorming
Affects Versions: 0.92.0
Reporter: Todd Lipcon
 Attachments: revert.txt


 I've been chatting with some folks recently about this thought: it seems 
 like, if you enumerate the larger changes in 0.92, this is probably the one 
 that is the most destabilizing that hasn't been through a lot of baking 
 yet. You can see this evidenced by the very high number of followup commits 
 it generated: looks like somewhere around 15 of them, plus some bugs still 
 open.
 I've done a patch to revert this and the related followup changes on the 0.92 
 branch. Do we want to consider doing this?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4622) Remove trivial 0.90 deprecated code from 0.92 and trunk.

2011-10-21 Thread Jonathan Hsieh (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-4622:
--

Status: Patch Available  (was: Open)

I've run full tests suites of the respective versions on trunk and 0.92 
branches.  All pass or known flakey tests fails intermittently. 
(TestMasterFailover.testMasterFailoverWithMockedRIT)

 Remove trivial 0.90 deprecated code from 0.92 and trunk.
 

 Key: HBASE-4622
 URL: https://issues.apache.org/jira/browse/HBASE-4622
 Project: HBase
  Issue Type: Sub-task
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Fix For: 0.92.0

 Attachments: 
 0001-HBASE-4622.-Remove-trivial-0.90-deprecated-code-from.0.92.patch, 
 0001-HBASE-4622.-Remove-trivial-0.90-deprecated-code-from.trunk.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4622) Remove trivial 0.90 deprecated code from 0.92 and trunk.

2011-10-21 Thread Jonathan Hsieh (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-4622:
--

Affects Version/s: 0.94.0
   0.92.0
Fix Version/s: (was: 0.92.0)

 Remove trivial 0.90 deprecated code from 0.92 and trunk.
 

 Key: HBASE-4622
 URL: https://issues.apache.org/jira/browse/HBASE-4622
 Project: HBase
  Issue Type: Sub-task
Affects Versions: 0.92.0, 0.94.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Attachments: 
 0001-HBASE-4622.-Remove-trivial-0.90-deprecated-code-from.0.92.patch, 
 0001-HBASE-4622.-Remove-trivial-0.90-deprecated-code-from.trunk.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4645) Edits Log recovery losing data across column families

[
https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13132877#comment-13132877
]

jirapos...@reviews.apache.org commented on HBASE-4645:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2524/
---

(Updated 2011-10-21 18:03:54.336799)

Review request for Ted Yu, Michael Stack, Jonathan Gray, Lars Hofhansl,
Amitanand Aiyer, Kannan Muthukkaruppan, Karthik Ranganathan, and Nicolas
Spiegelberg.

Changes
---

fix white space issue.

Summary
---

There is a data loss happening (for some of the column families) when we do the
replay logs.

The correct logic for replay should begin the replay from the minimum across the
maximum in each store.

This addresses bug hbase-4645.
https://issues.apache.org/jira/browse/hbase-4645

Diffs (updated)
-

src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8c32839
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
966262b

Diff: https://reviews.apache.org/r/2524/diff

Testing
---

Initial patch. v1.

mvn test (running).

TBD: add a test case to repro the issue and make sure it fixes.

Thanks,

Amitanand

Edits Log recovery losing data across column families
-

Key: HBASE-4645
URL: https://issues.apache.org/jira/browse/HBASE-4645
Project: HBase
Issue Type: Bug
Affects Versions: 0.89.20100924, 0.92.0
Reporter: Amitanand Aiyer
Assignee: Amitanand Aiyer

[jira] [Commented] (HBASE-4645) Edits Log recovery losing data across column families

2011-10-21 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13132879#comment-13132879
]

jirapos...@reviews.apache.org commented on HBASE-4645:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2524/
---

(Updated 2011-10-21 18:06:16.651615)

Review request for Ted Yu, Michael Stack, Jonathan Gray, Lars Hofhansl,
Amitanand Aiyer, Kannan Muthukkaruppan, Karthik Ranganathan, and Nicolas
Spiegelberg.

Changes
---

missed the white space errors in TestWALReplay. fixed.

Summary
---

There is a data loss happening (for some of the column families) when we do the
replay logs.

The correct logic for replay should begin the replay from the minimum across the
maximum in each store.

This addresses bug hbase-4645.
https://issues.apache.org/jira/browse/hbase-4645

Diffs (updated)
-

src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8c32839
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
966262b

Diff: https://reviews.apache.org/r/2524/diff

Testing
---

Initial patch. v1.

mvn test (running).

TBD: add a test case to repro the issue and make sure it fixes.

Thanks,

Amitanand

Edits Log recovery losing data across column families
-

Key: HBASE-4645
URL: https://issues.apache.org/jira/browse/HBASE-4645
Project: HBase
Issue Type: Bug
Affects Versions: 0.89.20100924, 0.92.0
Reporter: Amitanand Aiyer
Assignee: Amitanand Aiyer

[jira] [Commented] (HBASE-4643) Consider reverting HBASE-451 (change HRI to remove HTD) in 0.92

2011-10-21 Thread Todd Lipcon (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13132884#comment-13132884
 ] 

Todd Lipcon commented on HBASE-4643:


Jonathan: you have clusters on 0.92 with real data? you're an adventurous guy!

 Consider reverting HBASE-451 (change HRI to remove HTD) in 0.92
 ---

 Key: HBASE-4643
 URL: https://issues.apache.org/jira/browse/HBASE-4643
 Project: HBase
  Issue Type: Brainstorming
Affects Versions: 0.92.0
Reporter: Todd Lipcon
 Attachments: revert.txt


 I've been chatting with some folks recently about this thought: it seems 
 like, if you enumerate the larger changes in 0.92, this is probably the one 
 that is the most destabilizing that hasn't been through a lot of baking 
 yet. You can see this evidenced by the very high number of followup commits 
 it generated: looks like somewhere around 15 of them, plus some bugs still 
 open.
 I've done a patch to revert this and the related followup changes on the 0.92 
 branch. Do we want to consider doing this?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4645) Edits Log recovery losing data across column families

2011-10-21 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13132894#comment-13132894
]

jirapos...@reviews.apache.org commented on HBASE-4645:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2524/#review2756
---

Amit-- code changes look solid.

It is unfortunate that there was a nice test TestWALReplay.java to test this
exact case-- but was broken in catching this issue.

Furthermore, the test writes to multiple CFs in separate puts (transactions).
It will be good to enhance it to do cross-CF puts (in a single txn).

src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
https://reviews.apache.org/r/2524/#comment6197

looks like the original test is busted. It expects to flush only one region
(in line 271); but here region.close() will cause it to flush all stores :)

- Kannan

On 2011-10-21 18:06:16, Amitanand Aiyer wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2524/
bq. ---
bq.
bq. (Updated 2011-10-21 18:06:16)
bq.
bq.
bq. Review request for Ted Yu, Michael Stack, Jonathan Gray, Lars Hofhansl,
Amitanand Aiyer, Kannan Muthukkaruppan, Karthik Ranganathan, and Nicolas
Spiegelberg.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. There is a data loss happening (for some of the column families) when we
do the replay logs.
bq.
bq. The bug seems to be from the fact that during replay-logs we only choose
to replay
bq. the logs from the maximumSequenceID across ALL the stores. This is wrong.
If a
bq. column family is ahead of others (because the crash happened before all
the column
bq. families were flushed), then we lose data for the column families that
have not yet
bq. caught up.
bq.
bq. The correct logic for replay should begin the replay from the minimum
across the
bq. maximum in each store.
bq.
bq.
bq. This addresses bug hbase-4645.
bq. https://issues.apache.org/jira/browse/hbase-4645
bq.
bq.
bq. Diffs
bq. -
bq.
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8c32839
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
966262b
bq.
bq. Diff: https://reviews.apache.org/r/2524/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Initial patch. v1.
bq.
bq. mvn test (running).
bq.
bq. TBD: add a test case to repro the issue and make sure it fixes.
bq.
bq.
bq. Thanks,
bq.
bq. Amitanand
bq.
bq.

Edits Log recovery losing data across column families
-

Key: HBASE-4645
URL: https://issues.apache.org/jira/browse/HBASE-4645
Project: HBase
Issue Type: Bug
Affects Versions: 0.89.20100924, 0.92.0
Reporter: Amitanand Aiyer
Assignee: Amitanand Aiyer

[jira] [Commented] (HBASE-4645) Edits Log recovery losing data across column families

2011-10-21 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13132903#comment-13132903
]

jirapos...@reviews.apache.org commented on HBASE-4645:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2524/#review2755
---

Wow... Losing data?! And that went unnoticed for so long.
I guess I don't understand when the store's maxid go out of sync and why this
does not happen all the time.

Nice find!!

src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
https://reviews.apache.org/r/2524/#comment6196

Good comment. Makes sense.

Does that mean that up to now we could have failed to replay some of the
logs and actually lose data?!
(I guess, yes, that's how you describe the problem)

src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
https://reviews.apache.org/r/2524/#comment6198

Is the because before your change it would not need to replay any logs, but
now it does?

- Lars

Edits Log recovery losing data across column families
-

Key: HBASE-4645
URL: https://issues.apache.org/jira/browse/HBASE-4645
Project: HBase
Issue Type: Bug
Affects Versions: 0.89.20100924, 0.92.0
Reporter: Amitanand Aiyer
Assignee: Amitanand Aiyer

[jira] [Updated] (HBASE-3929) Add option to HFile tool to produce basic stats

2011-10-21 Thread Todd Lipcon (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon updated HBASE-3929:
---

  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Committed to trunk and 92. Thanks Matteo

 Add option to HFile tool to produce basic stats
 ---

 Key: HBASE-3929
 URL: https://issues.apache.org/jira/browse/HBASE-3929
 Project: HBase
  Issue Type: New Feature
  Components: io
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
 Fix For: 0.94.0

 Attachments: HBASE-3929-v2.patch, HBASE-3929-v3.patch, 
 hbase-3929-draft.patch, hbase-3929-draft.txt


 In looking at HBASE-3421 I wrote a small tool to scan an HFile and produce 
 some basic statistics about it:
 - min/mean/max key size, value size (uncompressed)
 - min/mean/max number of columns per row (uncompressed)
 - min/mean/max number of bytes per row (uncompressed)
 - the key of the largest row

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-3929) Add option to HFile tool to produce basic stats

2011-10-21 Thread Todd Lipcon (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon updated HBASE-3929:
---

Fix Version/s: (was: 0.94.0)
   0.92.0

 Add option to HFile tool to produce basic stats
 ---

 Key: HBASE-3929
 URL: https://issues.apache.org/jira/browse/HBASE-3929
 Project: HBase
  Issue Type: New Feature
  Components: io
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Assignee: Matteo Bertozzi
 Fix For: 0.92.0

 Attachments: HBASE-3929-v2.patch, HBASE-3929-v3.patch, 
 hbase-3929-draft.patch, hbase-3929-draft.txt


 In looking at HBASE-3421 I wrote a small tool to scan an HFile and produce 
 some basic statistics about it:
 - min/mean/max key size, value size (uncompressed)
 - min/mean/max number of columns per row (uncompressed)
 - min/mean/max number of bytes per row (uncompressed)
 - the key of the largest row

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-3929) Add option to HFile tool to produce basic stats

2011-10-21 Thread Todd Lipcon (Assigned) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon reassigned HBASE-3929:
--

Assignee: Matteo Bertozzi  (was: Todd Lipcon)

 Add option to HFile tool to produce basic stats
 ---

 Key: HBASE-3929
 URL: https://issues.apache.org/jira/browse/HBASE-3929
 Project: HBase
  Issue Type: New Feature
  Components: io
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Assignee: Matteo Bertozzi
 Fix For: 0.92.0

 Attachments: HBASE-3929-v2.patch, HBASE-3929-v3.patch, 
 hbase-3929-draft.patch, hbase-3929-draft.txt


 In looking at HBASE-3421 I wrote a small tool to scan an HFile and produce 
 some basic statistics about it:
 - min/mean/max key size, value size (uncompressed)
 - min/mean/max number of columns per row (uncompressed)
 - min/mean/max number of bytes per row (uncompressed)
 - the key of the largest row

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4645) Edits Log recovery losing data across column families

2011-10-21 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13132919#comment-13132919
]

jirapos...@reviews.apache.org commented on HBASE-4645:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2524/
---

(Updated 2011-10-21 18:49:45.916474)

Review request for Ted Yu, Michael Stack, Jonathan Gray, Lars Hofhansl,
Amitanand Aiyer, Kannan Muthukkaruppan, Karthik Ranganathan, and Nicolas
Spiegelberg.

Changes
---

ensure that when we are testing region.close does not flush the stores.
pass abort=true.

Summary
---

There is a data loss happening (for some of the column families) when we do the
replay logs.

The correct logic for replay should begin the replay from the minimum across the
maximum in each store.

This addresses bug hbase-4645.
https://issues.apache.org/jira/browse/hbase-4645

Diffs (updated)
-

src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8c32839
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
966262b

Diff: https://reviews.apache.org/r/2524/diff

Testing
---

Initial patch. v1.

mvn test (running).

TBD: add a test case to repro the issue and make sure it fixes.

Thanks,

Amitanand

Edits Log recovery losing data across column families
-

Key: HBASE-4645
URL: https://issues.apache.org/jira/browse/HBASE-4645
Project: HBase
Issue Type: Bug
Affects Versions: 0.89.20100924, 0.92.0
Reporter: Amitanand Aiyer
Assignee: Amitanand Aiyer

[jira] [Resolved] (HBASE-4626) Filters unnecessarily copy byte arrays...

2011-10-21 Thread Lars Hofhansl (Resolved) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl resolved HBASE-4626.
--

   Resolution: Fixed
Fix Version/s: (was: 0.92.0)
 Hadoop Flags: Reviewed

I prefer to keep this in trunk only for now.
Please pull back in for 0.92 if you feel otherwise (the patch is simple and 
safe).

 Filters unnecessarily copy byte arrays...
 -

 Key: HBASE-4626
 URL: https://issues.apache.org/jira/browse/HBASE-4626
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.94.0

 Attachments: 4626-v2.txt, 4626-v3.txt, 4626.txt


 Just looked at SingleCol and ValueFilter... And on every column compared they 
 create a copy of the column and/or value portion of the KV.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4532) Avoid top row seek by dedicated bloom filter for delete family bloom filter

2011-10-21 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13132995#comment-13132995
]

jirapos...@reviews.apache.org commented on HBASE-4532:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2393/
---

(Updated 2011-10-21 19:49:16.120589)

Review request for hbase, Dhruba Borthakur, Michael Stack, Mikhail Bautin,
Pritam Damania, Prakash Khemani, Amitanand Aiyer, Kannan Muthukkaruppan, Jerry
Chen, Liyin Tang, Karthik Ranganathan, and Nicolas Spiegelberg.

Changes
---

Thanks Kannan:) Update the diff to address Kannan's review

Summary
---

HBASE-4469 avoids the top row seek operation if row-col bloom filter is
enabled.
This jira tries to avoid top row seek for all the cases by creating a dedicated
bloom filter only for delete family

The only subtle use case is when we are interested in the top row with empty
column.

For example,
we are interested in row1/cf1:/1/put.
So we seek to the top row: row1/cf1:/MAX_TS/MAXIMUM. And the delete family
bloom filter will say there is NO delete family.
Then it will avoid the top row seek and return a fake kv, which is the last kv
for this row (createLastOnRowCol).
In this way, we have already missed the real kv we are interested in.

The solution for the above problem is to disable this optimization if we are
trying to GET/SCAN a row with empty column.

This patch is rebased on 0.89-fb. But it should be the same for apache-trunk as
well. I will submit the patch for apache-trunk later.

This addresses bug HBASE-4532.
https://issues.apache.org/jira/browse/HBASE-4532

Diffs (updated)
-

src/main/java/org/apache/hadoop/hbase/KeyValue.java 93538bb
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockType.java 9a79a74
src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 5d9b518
src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java
6cf7cce
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 1f78dd4
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 3c34f86
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java 2e1d23a
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java c4b60e9
src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
92070b3
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java e4dfc2e
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
ebb360c
src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 8814812
src/main/java/org/apache/hadoop/hbase/util/BloomFilterFactory.java fb4f2df
src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java
b8bcc65

src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java
48e9163
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 0eca9b8

Diff: https://reviews.apache.org/r/2393/diff

Testing
---

Passed all the unit tests

Thanks,

Liyin

Avoid top row seek by dedicated bloom filter for delete family bloom filter
---

Key: HBASE-4532
URL: https://issues.apache.org/jira/browse/HBASE-4532
Project: HBase
Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang
Attachments: D27.1.patch, D27.1.patch

HBASE-4469 avoids the top row seek operation if row-col bloom filter is
enabled.
This jira tries to avoid top row seek for all the cases by creating a
dedicated bloom filter only for delete family
The only subtle use case is when we are interested in the top row with empty
column.
For example,
we are interested in row1/cf1:/1/put.
So we seek to the top row: row1/cf1:/MAX_TS/MAXIMUM. And the delete family
bloom filter will say there is NO delete family.
Then it will avoid the top row seek and return a fake kv, which is the last
kv for this row (createLastOnRowCol).
In this way, we have already missed the real kv we are interested in.
The solution for the above problem is to disable this optimization if we are
trying to GET/SCAN a row with empty column.

[jira] [Updated] (HBASE-4532) Avoid top row seek by dedicated bloom filter for delete family bloom filter

2011-10-21 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Liyin Tang updated HBASE-4532:
--

Description:
The previous jira, HBASE-4469, is to avoid the top row seek operation if
row-col bloom filter is enabled.
This jira tries to avoid top row seek for all the cases by creating a dedicated
bloom filter only for delete family

The only subtle use case is when we are interested in the top row with empty
column.

The solution for the above problem is to disable this optimization if we are
trying to GET/SCAN a row with empty column.

Evaluation from TestSeekOptimization:
Previously:
For bloom=NONE, compr=NONE total seeks without optimization: 2506, with
optimization: 1714 (68.40%), savings: 31.60%
For bloom=ROW, compr=NONE total seeks without optimization: 2506, with
optimization: 1714 (68.40%), savings: 31.60%
For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with
optimization: 1458 (58.18%), savings: 41.82%
For bloom=NONE, compr=GZ total seeks without optimization: 2506, with
optimization: 1714 (68.40%), savings: 31.60%
For bloom=ROW, compr=GZ total seeks without optimization: 2506, with
optimization: 1714 (68.40%), savings: 31.60%
For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with
optimization: 1458 (58.18%), savings: 41.82%

So we can only get 10% more seek savings if the ROWCOL bloom filter is
enabled.[HBASE-4469]

After this change:
For bloom=NONE, compr=NONE total seeks without optimization: 2506, with
optimization: 1458 (58.18%), savings: 41.82%
For bloom=ROW, compr=NONE total seeks without optimization: 2506, with
optimization: 1458 (58.18%), savings: 41.82%
For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with
optimization: 1458 (58.18%), savings: 41.82%
For bloom=NONE, compr=GZ total seeks without optimization: 2506, with
optimization: 1458 (58.18%), savings: 41.82%
For bloom=ROW, compr=GZ total seeks without optimization: 2506, with
optimization: 1458 (58.18%), savings: 41.82%
For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with
optimization: 1458 (58.18%), savings: 41.82%

So we can get 10% more seek savings for ALL kinds of bloom filter.

was:
HBASE-4469 avoids the top row seek operation if row-col bloom filter is
enabled.
This jira tries to avoid top row seek for all the cases by creating a dedicated
bloom filter only for delete family

The only subtle use case is when we are interested in the top row with empty
column.

The solution for the above problem is to disable this optimization if we are
trying to GET/SCAN a row with empty column.

Avoid top row seek by dedicated bloom filter for delete family bloom filter
---

Key: HBASE-4532
URL: https://issues.apache.org/jira/browse/HBASE-4532
Project: HBase
Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang
Attachments: D27.1.patch, D27.1.patch

The previous jira, HBASE-4469, is to avoid the top row seek operation if
row-col bloom filter is enabled.
This jira tries to avoid top row seek for all the cases by creating a
dedicated bloom filter only for delete family
The only subtle use case is when we are interested in the top row with empty
column.
For example,
we are interested in row1/cf1:/1/put.
So we seek to the top row: row1/cf1:/MAX_TS/MAXIMUM. And the delete family
bloom filter will say there is NO delete family.
Then it will avoid the top row seek and return a fake kv, which is the last
kv for this row (createLastOnRowCol).
In this way, we have already missed the real kv we are interested in.
The solution for the above problem is to disable this optimization if we are
trying to GET/SCAN a row with empty column.
Evaluation from TestSeekOptimization:
Previously:
For bloom=NONE, compr=NONE total seeks without optimization: 2506, with
optimization: 1714 (68.40%), savings: 31.60%
For bloom=ROW, compr=NONE total seeks without optimization: 2506, with
optimization:

[jira] [Commented] (HBASE-4532) Avoid top row seek by dedicated bloom filter for delete family bloom filter


[ 
https://issues.apache.org/jira/browse/HBASE-4532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13133002#comment-13133002
 ] 

jirapos...@reviews.apache.org commented on HBASE-4532:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2393/
---

(Updated 2011-10-21 19:58:05.693922)


Review request for hbase, Dhruba Borthakur, Michael Stack, Mikhail Bautin, 
Pritam Damania, Prakash Khemani, Amitanand Aiyer, Kannan Muthukkaruppan, Jerry 
Chen, Liyin Tang, Karthik Ranganathan, and Nicolas Spiegelberg.


Summary (updated)
---

The previous jira, HBASE-4469, is to avoid the top row seek operation if 
row-col bloom filter is enabled. 
This jira tries to avoid top row seek for all the cases by creating a dedicated 
bloom filter only for delete family

The only subtle use case is when we are interested in the top row with empty 
column.

For example, 
we are interested in row1/cf1:/1/put.
So we seek to the top row: row1/cf1:/MAX_TS/MAXIMUM. And the delete family 
bloom filter will say there is NO delete family.
Then it will avoid the top row seek and return a fake kv, which is the last kv 
for this row (createLastOnRowCol).
In this way, we have already missed the real kv we are interested in.


The solution for the above problem is to disable this optimization if we are 
trying to GET/SCAN a row with empty column.


Evaluation from TestSeekOptimization:
Previously:
For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
optimization: 1714 (68.40%), savings: 31.60%
For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
optimization: 1714 (68.40%), savings: 31.60%
For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
optimization: 1458 (58.18%), savings: 41.82%

For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
optimization: 1714 (68.40%), savings: 31.60%
For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
optimization: 1714 (68.40%), savings: 31.60%
For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
optimization: 1458 (58.18%), savings: 41.82%

So we can get about 10% more seek savings ONLY if the ROWCOL bloom filter is 
enabled.[HBASE-4469]



After this change:
For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
optimization: 1458 (58.18%), savings: 41.82%
For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
optimization: 1458 (58.18%), savings: 41.82%
For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
optimization: 1458 (58.18%), savings: 41.82%

For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
optimization: 1458 (58.18%), savings: 41.82%
For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
optimization: 1458 (58.18%), savings: 41.82%
For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
optimization: 1458 (58.18%), savings: 41.82%

So we can get about 10% more seek savings for ALL kinds of bloom filter.


This addresses bug HBASE-4532.
https://issues.apache.org/jira/browse/HBASE-4532


Diffs
-

  src/main/java/org/apache/hadoop/hbase/KeyValue.java 93538bb 
  src/main/java/org/apache/hadoop/hbase/io/hfile/BlockType.java 9a79a74 
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 5d9b518 
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java 
6cf7cce 
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 1f78dd4 
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 3c34f86 
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java 2e1d23a 
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java c4b60e9 
  src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 
92070b3 
  src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java e4dfc2e 
  src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java 
ebb360c 
  src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 8814812 
  src/main/java/org/apache/hadoop/hbase/util/BloomFilterFactory.java fb4f2df 
  src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 
b8bcc65 
  
src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java 
48e9163 
  src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 0eca9b8 

Diff: https://reviews.apache.org/r/2393/diff


Testing
---

Passed all the unit tests


Thanks,

Liyin



 Avoid top row seek by dedicated bloom filter for delete family bloom filter
 ---

 Key: HBASE-4532
 URL:

[jira] [Commented] (HBASE-4532) Avoid top row seek by dedicated bloom filter for delete family bloom filter

2011-10-21 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13133003#comment-13133003
 ] 

jirapos...@reviews.apache.org commented on HBASE-4532:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2393/#review2758
---

Ship it!


+1. One typo below.


src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
https://reviews.apache.org/r/2393/#comment6199

exits - exist


- Kannan


On 2011-10-21 19:58:05, Liyin Tang wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2393/
bq.  ---
bq.  
bq.  (Updated 2011-10-21 19:58:05)
bq.  
bq.  
bq.  Review request for hbase, Dhruba Borthakur, Michael Stack, Mikhail Bautin, 
Pritam Damania, Prakash Khemani, Amitanand Aiyer, Kannan Muthukkaruppan, Jerry 
Chen, Liyin Tang, Karthik Ranganathan, and Nicolas Spiegelberg.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  The previous jira, HBASE-4469, is to avoid the top row seek operation if 
row-col bloom filter is enabled. 
bq.  This jira tries to avoid top row seek for all the cases by creating a 
dedicated bloom filter only for delete family
bq.  
bq.  The only subtle use case is when we are interested in the top row with 
empty column.
bq.  
bq.  For example, 
bq.  we are interested in row1/cf1:/1/put.
bq.  So we seek to the top row: row1/cf1:/MAX_TS/MAXIMUM. And the delete family 
bloom filter will say there is NO delete family.
bq.  Then it will avoid the top row seek and return a fake kv, which is the 
last kv for this row (createLastOnRowCol).
bq.  In this way, we have already missed the real kv we are interested in.
bq.  
bq.  
bq.  The solution for the above problem is to disable this optimization if we 
are trying to GET/SCAN a row with empty column.
bq.  
bq.  
bq.  Evaluation from TestSeekOptimization:
bq.  Previously:
bq.  For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
optimization: 1714 (68.40%), savings: 31.60%
bq.  For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
optimization: 1714 (68.40%), savings: 31.60%
bq.  For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
optimization: 1458 (58.18%), savings: 41.82%
bq.  
bq.  For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
optimization: 1714 (68.40%), savings: 31.60%
bq.  For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
optimization: 1714 (68.40%), savings: 31.60%
bq.  For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
optimization: 1458 (58.18%), savings: 41.82%
bq.  
bq.  So we can get about 10% more seek savings ONLY if the ROWCOL bloom filter 
is enabled.[HBASE-4469]
bq.  
bq.  
bq.  
bq.  After this change:
bq.  For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
optimization: 1458 (58.18%), savings: 41.82%
bq.  For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
optimization: 1458 (58.18%), savings: 41.82%
bq.  For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
optimization: 1458 (58.18%), savings: 41.82%
bq.  
bq.  For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
optimization: 1458 (58.18%), savings: 41.82%
bq.  For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
optimization: 1458 (58.18%), savings: 41.82%
bq.  For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
optimization: 1458 (58.18%), savings: 41.82%
bq.  
bq.  So we can get about 10% more seek savings for ALL kinds of bloom filter.
bq.  
bq.  
bq.  This addresses bug HBASE-4532.
bq.  https://issues.apache.org/jira/browse/HBASE-4532
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/KeyValue.java 93538bb 
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockType.java 9a79a74 
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 5d9b518 
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java 
6cf7cce 
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 
1f78dd4 
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 
3c34f86 
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java 
2e1d23a 
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 
c4b60e9 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 
92070b3 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 
e4dfc2e 
bq.

[jira] [Commented] (HBASE-4532) Avoid top row seek by dedicated bloom filter for delete family bloom filter

2011-10-21 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13133008#comment-13133008
 ] 

jirapos...@reviews.apache.org commented on HBASE-4532:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2393/
---

(Updated 2011-10-21 20:01:23.160793)


Review request for hbase, Dhruba Borthakur, Michael Stack, Mikhail Bautin, 
Pritam Damania, Prakash Khemani, Amitanand Aiyer, Kannan Muthukkaruppan, Jerry 
Chen, Liyin Tang, Karthik Ranganathan, and Nicolas Spiegelberg.


Changes
---

Fix the typo:)


Summary
---

The previous jira, HBASE-4469, is to avoid the top row seek operation if 
row-col bloom filter is enabled. 
This jira tries to avoid top row seek for all the cases by creating a dedicated 
bloom filter only for delete family

The only subtle use case is when we are interested in the top row with empty 
column.

For example, 
we are interested in row1/cf1:/1/put.
So we seek to the top row: row1/cf1:/MAX_TS/MAXIMUM. And the delete family 
bloom filter will say there is NO delete family.
Then it will avoid the top row seek and return a fake kv, which is the last kv 
for this row (createLastOnRowCol).
In this way, we have already missed the real kv we are interested in.


The solution for the above problem is to disable this optimization if we are 
trying to GET/SCAN a row with empty column.


Evaluation from TestSeekOptimization:
Previously:
For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
optimization: 1714 (68.40%), savings: 31.60%
For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
optimization: 1714 (68.40%), savings: 31.60%
For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
optimization: 1458 (58.18%), savings: 41.82%

For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
optimization: 1714 (68.40%), savings: 31.60%
For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
optimization: 1714 (68.40%), savings: 31.60%
For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
optimization: 1458 (58.18%), savings: 41.82%

So we can get about 10% more seek savings ONLY if the ROWCOL bloom filter is 
enabled.[HBASE-4469]



After this change:
For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
optimization: 1458 (58.18%), savings: 41.82%
For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
optimization: 1458 (58.18%), savings: 41.82%
For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
optimization: 1458 (58.18%), savings: 41.82%

For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
optimization: 1458 (58.18%), savings: 41.82%
For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
optimization: 1458 (58.18%), savings: 41.82%
For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
optimization: 1458 (58.18%), savings: 41.82%

So we can get about 10% more seek savings for ALL kinds of bloom filter.


This addresses bug HBASE-4532.
https://issues.apache.org/jira/browse/HBASE-4532


Diffs (updated)
-

  src/main/java/org/apache/hadoop/hbase/KeyValue.java 93538bb 
  src/main/java/org/apache/hadoop/hbase/io/hfile/BlockType.java 9a79a74 
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 5d9b518 
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java 
6cf7cce 
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 1f78dd4 
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 3c34f86 
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java 2e1d23a 
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java c4b60e9 
  src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 
92070b3 
  src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java e4dfc2e 
  src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java 
ebb360c 
  src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 8814812 
  src/main/java/org/apache/hadoop/hbase/util/BloomFilterFactory.java fb4f2df 
  src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 
b8bcc65 
  
src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java 
48e9163 
  src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 0eca9b8 

Diff: https://reviews.apache.org/r/2393/diff


Testing
---

Passed all the unit tests


Thanks,

Liyin



 Avoid top row seek by dedicated bloom filter for delete family bloom filter
 ---

 Key:

[jira] [Updated] (HBASE-4532) Avoid top row seek by dedicated bloom filter for delete family bloom filter


 [ 
https://issues.apache.org/jira/browse/HBASE-4532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liyin Tang updated HBASE-4532:
--

Attachment: hbase-4532-89-fb.patch

 Avoid top row seek by dedicated bloom filter for delete family bloom filter
 ---

 Key: HBASE-4532
 URL: https://issues.apache.org/jira/browse/HBASE-4532
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang
 Attachments: D27.1.patch, D27.1.patch, hbase-4532-89-fb.patch


 The previous jira, HBASE-4469, is to avoid the top row seek operation if 
 row-col bloom filter is enabled. 
 This jira tries to avoid top row seek for all the cases by creating a 
 dedicated bloom filter only for delete family
 The only subtle use case is when we are interested in the top row with empty 
 column.
 For example, 
 we are interested in row1/cf1:/1/put.
 So we seek to the top row: row1/cf1:/MAX_TS/MAXIMUM. And the delete family 
 bloom filter will say there is NO delete family.
 Then it will avoid the top row seek and return a fake kv, which is the last 
 kv for this row (createLastOnRowCol).
 In this way, we have already missed the real kv we are interested in.
 The solution for the above problem is to disable this optimization if we are 
 trying to GET/SCAN a row with empty column.
 Evaluation from TestSeekOptimization:
 Previously:
 For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 So we can get about 10% more seek savings ONLY if the ROWCOL bloom filter is 
 enabled.[HBASE-4469]
 
 After this change:
 For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 So we can get about 10% more seek savings for ALL kinds of bloom filter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4634) test.build.data property overused leading to write data at the wrong place

2011-10-21 Thread nkeywal (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13133045#comment-13133045
 ] 

nkeywal commented on HBASE-4634:


I am going to attach a second version of the patch. I checked indent, it seems 
ok to me.

It took more time than expected because I haven't managed to escape some 
modifications to the get files functions. Not much impacts for the user. 
Basically, there is now a clear distinction between:
- base directory: configurable directory where all tests put their data
- data directory: sub directory used by the test
- cluster directory: subdirectory for the cluster stuff (dfs or hbase)

This will help to have multiple HBaseTestingUtility in the same JVM (// or not).

I have also removed a duplicate getter (something as getHbaseCluster while 
getHBase cluster exists). 

It includes the fixes for HBASE-4639 and most if not all the bad usage 
mentioned in HBASE-4326 (but I did not fix the root issue of HBASE-4326).

It seems to work on my env, currently n-checking again.

I have a systematic errors in the test, but it's in 
#org.apache.hadoop.hbase.io.hfile.TestHFileBlock#testBlockHeapSize , I believe 
it was there at the beginning.

I have random errors in: TestDistributedLogSplitting, 
regionserver.TestServerCustomProtocol, client.TestAdmin 
master.TestHMasterRPCException, 
coprocessor.TestRegionServerCoprocessorExceptionWithAbort, wal.TestLogRolling, 
CatalogTrackerOnCluster and a few else, but it works when I replay the test 
set, so I tend to think they belong to the flaky test sets.

At the end of the day, the patch is not that medium... Sorry for that.

Will add a new comment when the tests will be over and I will have rechecked 
the flaky one, but you can already have a look.



 test.build.data property overused leading to write data at the wrong place
 

 Key: HBASE-4634
 URL: https://issues.apache.org/jira/browse/HBASE-4634
 Project: HBase
  Issue Type: Bug
  Components: test
Affects Versions: 0.92.0
 Environment: all
Reporter: nkeywal
Assignee: nkeywal
 Attachments: 2011020_4634_all.patch, 20111020_4639_TestStoreFile.patch


 test.build.data is overloaded in HBase.At the beginning, it's the Default 
 parent directory for test output., but then it's rewritten to be the 
 directory itself in functions like HBaseTestingUtility#startMiniDFSCluster
 It seems that this value is already used by MiniDFS (i.e. outside of HBase): 
 Name is as it is because mini dfs has hard-codings to put test data here.
 As it is today, there is at least a bug in HBaseTestingUtility:
 {noformat}
   public void initTestDir() {
 if (System.getProperty(TEST_DIRECTORY_KEY) == null) {
   clusterTestBuildDir = setupClusterTestBuildDir();
   System.setProperty(TEST_DIRECTORY_KEY, clusterTestBuildDir.getPath());
 }
   }
 {noformat}
 if you set a value for test.build.data, the test dir will be the parent 
 directory and not a temp subdir, leading to issues as multiple tests will 
 end-ups in the same (bad) directory. This function is barely used today, 
 hence it's not visible, but I would like to use it in some new code.
 A possible fix is to remove the check for null and continue with the 
 overloading, but I don't think it would be a big issue to create a new 
 key(like test.build.data.rootdirectory) specific to the root directory and 
 to use test.build.data only to communicate with MiniDFS. Feedback welcome.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4634) test.build.data property overused leading to write data at the wrong place

2011-10-21 Thread nkeywal (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nkeywal updated HBASE-4634:
---

Attachment: 20111021_4634_all.v2.patch

 test.build.data property overused leading to write data at the wrong place
 

 Key: HBASE-4634
 URL: https://issues.apache.org/jira/browse/HBASE-4634
 Project: HBase
  Issue Type: Bug
  Components: test
Affects Versions: 0.92.0
 Environment: all
Reporter: nkeywal
Assignee: nkeywal
 Attachments: 2011020_4634_all.patch, 
 20111020_4639_TestStoreFile.patch, 20111021_4634_all.v2.patch


 test.build.data is overloaded in HBase.At the beginning, it's the Default 
 parent directory for test output., but then it's rewritten to be the 
 directory itself in functions like HBaseTestingUtility#startMiniDFSCluster
 It seems that this value is already used by MiniDFS (i.e. outside of HBase): 
 Name is as it is because mini dfs has hard-codings to put test data here.
 As it is today, there is at least a bug in HBaseTestingUtility:
 {noformat}
   public void initTestDir() {
 if (System.getProperty(TEST_DIRECTORY_KEY) == null) {
   clusterTestBuildDir = setupClusterTestBuildDir();
   System.setProperty(TEST_DIRECTORY_KEY, clusterTestBuildDir.getPath());
 }
   }
 {noformat}
 if you set a value for test.build.data, the test dir will be the parent 
 directory and not a temp subdir, leading to issues as multiple tests will 
 end-ups in the same (bad) directory. This function is barely used today, 
 hence it's not visible, but I would like to use it in some new code.
 A possible fix is to remove the check for null and continue with the 
 overloading, but I don't think it would be a big issue to create a new 
 key(like test.build.data.rootdirectory) specific to the root directory and 
 to use test.build.data only to communicate with MiniDFS. Feedback welcome.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4634) test.build.data property overused leading to write data at the wrong place


[ 
https://issues.apache.org/jira/browse/HBASE-4634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13133048#comment-13133048
 ] 

Ted Yu commented on HBASE-4634:
---

TestHFileBlock#testBlockHeapSize is fixed by addendum to HBASE-4219
Please refresh your workspace.

 test.build.data property overused leading to write data at the wrong place
 

 Key: HBASE-4634
 URL: https://issues.apache.org/jira/browse/HBASE-4634
 Project: HBase
  Issue Type: Bug
  Components: test
Affects Versions: 0.92.0
 Environment: all
Reporter: nkeywal
Assignee: nkeywal
 Attachments: 2011020_4634_all.patch, 
 20111020_4639_TestStoreFile.patch, 20111021_4634_all.v2.patch


 test.build.data is overloaded in HBase.At the beginning, it's the Default 
 parent directory for test output., but then it's rewritten to be the 
 directory itself in functions like HBaseTestingUtility#startMiniDFSCluster
 It seems that this value is already used by MiniDFS (i.e. outside of HBase): 
 Name is as it is because mini dfs has hard-codings to put test data here.
 As it is today, there is at least a bug in HBaseTestingUtility:
 {noformat}
   public void initTestDir() {
 if (System.getProperty(TEST_DIRECTORY_KEY) == null) {
   clusterTestBuildDir = setupClusterTestBuildDir();
   System.setProperty(TEST_DIRECTORY_KEY, clusterTestBuildDir.getPath());
 }
   }
 {noformat}
 if you set a value for test.build.data, the test dir will be the parent 
 directory and not a temp subdir, leading to issues as multiple tests will 
 end-ups in the same (bad) directory. This function is barely used today, 
 hence it's not visible, but I would like to use it in some new code.
 A possible fix is to remove the check for null and continue with the 
 overloading, but I don't think it would be a big issue to create a new 
 key(like test.build.data.rootdirectory) specific to the root directory and 
 to use test.build.data only to communicate with MiniDFS. Feedback welcome.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3929) Add option to HFile tool to produce basic stats

2011-10-21 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13133050#comment-13133050
 ] 

Hudson commented on HBASE-3929:
---

Integrated in HBase-TRUNK #2352 (See 
[https://builds.apache.org/job/HBase-TRUNK/2352/])
HBASE-3929  Add option to HFile tool to produce basic stats

todd : 
Files : 
* /hbase/trunk/CHANGES.txt
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java


 Add option to HFile tool to produce basic stats
 ---

 Key: HBASE-3929
 URL: https://issues.apache.org/jira/browse/HBASE-3929
 Project: HBase
  Issue Type: New Feature
  Components: io
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Assignee: Matteo Bertozzi
 Fix For: 0.92.0

 Attachments: HBASE-3929-v2.patch, HBASE-3929-v3.patch, 
 hbase-3929-draft.patch, hbase-3929-draft.txt


 In looking at HBASE-3421 I wrote a small tool to scan an HFile and produce 
 some basic statistics about it:
 - min/mean/max key size, value size (uncompressed)
 - min/mean/max number of columns per row (uncompressed)
 - min/mean/max number of bytes per row (uncompressed)
 - the key of the largest row

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4588) The floating point arithmetic to validate memory allocation configurations need to be done as integers

2011-10-21 Thread dhruba borthakur (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13133068#comment-13133068
 ] 

dhruba borthakur commented on HBASE-4588:
-

All the unit tests pass, can somebody pl commit this?

 The floating point arithmetic to validate memory allocation configurations 
 need to be done as integers
 --

 Key: HBASE-4588
 URL: https://issues.apache.org/jira/browse/HBASE-4588
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.0, 0.94.0
Reporter: Jonathan Gray
Assignee: dhruba borthakur
Priority: Minor
 Fix For: 0.92.0

 Attachments: configVerify1.txt, configVerify2.txt


 The floating point arithmetic to validate memory allocation configurations 
 need to be done as integers.
 On our cluster, we had block cache = 0.6 and memstore = 0.2.  It was saying 
 this was  0.8 when it is actually equal.
 Minor bug but annoying nonetheless.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4588) The floating point arithmetic to validate memory allocation configurations need to be done as integers

2011-10-21 Thread Jonathan Gray (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Gray updated HBASE-4588:
-

  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Committed to 92 branch and trunk.  Thanks Dhruba.

 The floating point arithmetic to validate memory allocation configurations 
 need to be done as integers
 --

 Key: HBASE-4588
 URL: https://issues.apache.org/jira/browse/HBASE-4588
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.0, 0.94.0
Reporter: Jonathan Gray
Assignee: dhruba borthakur
Priority: Minor
 Fix For: 0.92.0

 Attachments: configVerify1.txt, configVerify2.txt


 The floating point arithmetic to validate memory allocation configurations 
 need to be done as integers.
 On our cluster, we had block cache = 0.6 and memstore = 0.2.  It was saying 
 this was  0.8 when it is actually equal.
 Minor bug but annoying nonetheless.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4645) Edits Log recovery losing data across column families

[
https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13133086#comment-13133086
]

jirapos...@reviews.apache.org commented on HBASE-4645:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2524/
---

(Updated 2011-10-21 21:43:17.815733)

Review request for Ted Yu, Michael Stack, Jonathan Gray, Lars Hofhansl,
Amitanand Aiyer, Kannan Muthukkaruppan, Karthik Ranganathan, and Nicolas
Spiegelberg.

Summary
---

There is a data loss happening (for some of the column families) when we do the
replay logs.

The correct logic for replay should begin the replay from the minimum across the
maximum in each store.

This addresses bug hbase-4645.
https://issues.apache.org/jira/browse/hbase-4645

Diffs (updated)
-

src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8c32839
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
966262b

Diff: https://reviews.apache.org/r/2524/diff

Testing
---

Initial patch. v1.

mvn test (running).

TBD: add a test case to repro the issue and make sure it fixes.

Thanks,

Amitanand

Edits Log recovery losing data across column families
-

Key: HBASE-4645
URL: https://issues.apache.org/jira/browse/HBASE-4645
Project: HBase
Issue Type: Bug
Affects Versions: 0.89.20100924, 0.92.0
Reporter: Amitanand Aiyer
Assignee: Amitanand Aiyer

[jira] [Commented] (HBASE-4645) Edits Log recovery losing data across column families

2011-10-21 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13133091#comment-13133091
]

jirapos...@reviews.apache.org commented on HBASE-4645:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2524/
---

(Updated 2011-10-21 21:45:49.683712)

Review request for Ted Yu, Michael Stack, Jonathan Gray, Lars Hofhansl,
Amitanand Aiyer, Kannan Muthukkaruppan, Karthik Ranganathan, and Nicolas
Spiegelberg.

Changes
---

fix whitespace errors.

Summary
---

There is a data loss happening (for some of the column families) when we do the
replay logs.

The correct logic for replay should begin the replay from the minimum across the
maximum in each store.

This addresses bug hbase-4645.
https://issues.apache.org/jira/browse/hbase-4645

Diffs (updated)
-

src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8c32839
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
966262b

Diff: https://reviews.apache.org/r/2524/diff

Testing
---

Initial patch. v1.

mvn test (running).

TBD: add a test case to repro the issue and make sure it fixes.

Thanks,

Amitanand

Edits Log recovery losing data across column families
-

Key: HBASE-4645
URL: https://issues.apache.org/jira/browse/HBASE-4645
Project: HBase
Issue Type: Bug
Affects Versions: 0.89.20100924, 0.92.0
Reporter: Amitanand Aiyer
Assignee: Amitanand Aiyer

[jira] [Commented] (HBASE-4645) Edits Log recovery losing data across column families

2011-10-21 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13133095#comment-13133095
]

jirapos...@reviews.apache.org commented on HBASE-4645:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2524/
---

(Updated 2011-10-21 21:46:51.829806)

Review request for Ted Yu, Michael Stack, Jonathan Gray, Lars Hofhansl,
Amitanand Aiyer, Kannan Muthukkaruppan, Karthik Ranganathan, and Nicolas
Spiegelberg.

Changes
---

get rid of some unintended comments.

Summary
---

There is a data loss happening (for some of the column families) when we do the
replay logs.

The correct logic for replay should begin the replay from the minimum across the
maximum in each store.

This addresses bug hbase-4645.
https://issues.apache.org/jira/browse/hbase-4645

Diffs (updated)
-

src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8c32839
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
966262b

Diff: https://reviews.apache.org/r/2524/diff

Testing
---

Initial patch. v1.

mvn test (running).

TBD: add a test case to repro the issue and make sure it fixes.

Thanks,

Amitanand

Edits Log recovery losing data across column families
-

Key: HBASE-4645
URL: https://issues.apache.org/jira/browse/HBASE-4645
Project: HBase
Issue Type: Bug
Affects Versions: 0.89.20100924, 0.92.0
Reporter: Amitanand Aiyer
Assignee: Amitanand Aiyer

[jira] [Commented] (HBASE-4645) Edits Log recovery losing data across column families

2011-10-21 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13133105#comment-13133105
]

jirapos...@reviews.apache.org commented on HBASE-4645:
--

bq. On 2011-10-21 18:31:16, Lars Hofhansl wrote:
bq. Wow... Losing data?! And that went unnoticed for so long.
bq. I guess I don't understand when the store's maxid go out of sync and why
this does not happen all the time.
bq.
bq. Nice find!!

currently, we seem to flush all the column families together.

For this failure scenario to kick in, there has to be a failure after *some*
stores have flushed. But, not
all of them.

Unclean shutdowns are rare. That too in the middle of the flushing wasn't too
common.

But, important to fix, given that we now know about it.

bq. On 2011-10-21 18:31:16, Lars Hofhansl wrote:
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java, line
289
bq. https://reviews.apache.org/r/2524/diff/4/?file=52155#file52155line289
bq.
bq. Is the because before your change it would not need to replay any
logs, but now it does?

yes. earlier we would not have replayed any logs. The test was ensuring that we
don't.

Now that we do our decision based on the minimum across different Stores, we do
end
up replaying some edits.

- Amitanand

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2524/#review2755
---

On 2011-10-21 21:46:51, Amitanand Aiyer wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2524/
bq. ---
bq.
bq. (Updated 2011-10-21 21:46:51)
bq.
bq.
bq. Review request for Ted Yu, Michael Stack, Jonathan Gray, Lars Hofhansl,
Amitanand Aiyer, Kannan Muthukkaruppan, Karthik Ranganathan, and Nicolas
Spiegelberg.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. There is a data loss happening (for some of the column families) when we
do the replay logs.
bq.
bq. The bug seems to be from the fact that during replay-logs we only choose
to replay
bq. the logs from the maximumSequenceID across ALL the stores. This is wrong.
If a
bq. column family is ahead of others (because the crash happened before all
the column
bq. families were flushed), then we lose data for the column families that
have not yet
bq. caught up.
bq.
bq. The correct logic for replay should begin the replay from the minimum
across the
bq. maximum in each store.
bq.
bq.
bq. This addresses bug hbase-4645.
bq. https://issues.apache.org/jira/browse/hbase-4645
bq.
bq.
bq. Diffs
bq. -
bq.
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8c32839
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
966262b
bq.
bq. Diff: https://reviews.apache.org/r/2524/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Initial patch. v1.
bq.
bq. mvn test (running).
bq.
bq. TBD: add a test case to repro the issue and make sure it fixes.
bq.
bq.
bq. Thanks,
bq.
bq. Amitanand
bq.
bq.

Edits Log recovery losing data across column families
-

Key: HBASE-4645
URL: https://issues.apache.org/jira/browse/HBASE-4645
Project: HBase
Issue Type: Bug
Affects Versions: 0.89.20100924, 0.92.0
Reporter: Amitanand Aiyer
Assignee: Amitanand Aiyer

[jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog

2011-10-21 Thread dhruba borthakur (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13133107#comment-13133107
 ] 

dhruba borthakur commented on HBASE-4528:
-

Hi Ramkrisham I agree that the sync can occur from two places: from the rpc 
handler thread or the background LogSyncer thread. That should not impact 
correctness, isn't it true?

I will upload a new patch that addresses your comments about this patch in 
reviewboard.

 The put operation can release the rowlock before sync-ing the Hlog
 --

 Key: HBASE-4528
 URL: https://issues.apache.org/jira/browse/HBASE-4528
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Fix For: 0.94.0

 Attachments: appendNoSync5.txt, appendNoSyncPut1.txt, 
 appendNoSyncPut2.txt, appendNoSyncPut3.txt, appendNoSyncPut4.txt, 
 appendNoSyncPut5.txt, appendNoSyncPut6.txt


 This allows for better throughput when there are hot rows. A single row 
 update improves from 100 puts/sec/server to 5000 puts/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4645) Edits Log recovery losing data across column families

2011-10-21 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13133122#comment-13133122
]

jirapos...@reviews.apache.org commented on HBASE-4645:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2524/
---

(Updated 2011-10-21 22:14:22.830521)

Review request for Ted Yu, Michael Stack, Jonathan Gray, Lars Hofhansl,
Amitanand Aiyer, Kannan Muthukkaruppan, Karthik Ranganathan, and Nicolas
Spiegelberg.

Changes
---

delete the middle cf.

Summary
---

There is a data loss happening (for some of the column families) when we do the
replay logs.

The correct logic for replay should begin the replay from the minimum across the
maximum in each store.

This addresses bug hbase-4645.
https://issues.apache.org/jira/browse/hbase-4645

Diffs (updated)
-

src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8c32839
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
966262b

Diff: https://reviews.apache.org/r/2524/diff

Testing
---

Initial patch. v1.

mvn test (running).

TBD: add a test case to repro the issue and make sure it fixes.

Thanks,

Amitanand

Edits Log recovery losing data across column families
-

Key: HBASE-4645
URL: https://issues.apache.org/jira/browse/HBASE-4645
Project: HBase
Issue Type: Bug
Affects Versions: 0.89.20100924, 0.92.0
Reporter: Amitanand Aiyer
Assignee: Amitanand Aiyer

[jira] [Commented] (HBASE-4645) Edits Log recovery losing data across column families

2011-10-21 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13133133#comment-13133133
]

jirapos...@reviews.apache.org commented on HBASE-4645:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2524/
---

(Updated 2011-10-21 22:21:58.484290)

Review request for Ted Yu, Michael Stack, Jonathan Gray, Lars Hofhansl,
Amitanand Aiyer, Kannan Muthukkaruppan, Karthik Ranganathan, and Nicolas
Spiegelberg.

Summary
---

There is a data loss happening (for some of the column families) when we do the
replay logs.

The correct logic for replay should begin the replay from the minimum across the
maximum in each store.

This addresses bug hbase-4645.
https://issues.apache.org/jira/browse/hbase-4645

Diffs
-

src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8c32839
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
966262b

Diff: https://reviews.apache.org/r/2524/diff

Testing (updated)
---

Initial patch. v1.

mvn test (running).

Thanks,

Amitanand

Edits Log recovery losing data across column families
-

Key: HBASE-4645
URL: https://issues.apache.org/jira/browse/HBASE-4645
Project: HBase
Issue Type: Bug
Affects Versions: 0.89.20100924, 0.92.0
Reporter: Amitanand Aiyer
Assignee: Amitanand Aiyer

[jira] [Commented] (HBASE-4645) Edits Log recovery losing data across column families

2011-10-21 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13133138#comment-13133138
]

jirapos...@reviews.apache.org commented on HBASE-4645:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2524/#review2765
---

Ship it!

update test looks good!

- Kannan

On 2011-10-21 22:21:58, Amitanand Aiyer wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2524/
bq. ---
bq.
bq. (Updated 2011-10-21 22:21:58)
bq.
bq.
bq. Review request for Ted Yu, Michael Stack, Jonathan Gray, Lars Hofhansl,
Amitanand Aiyer, Kannan Muthukkaruppan, Karthik Ranganathan, and Nicolas
Spiegelberg.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. There is a data loss happening (for some of the column families) when we
do the replay logs.
bq.
bq. The bug seems to be from the fact that during replay-logs we only choose
to replay
bq. the logs from the maximumSequenceID across ALL the stores. This is wrong.
If a
bq. column family is ahead of others (because the crash happened before all
the column
bq. families were flushed), then we lose data for the column families that
have not yet
bq. caught up.
bq.
bq. The correct logic for replay should begin the replay from the minimum
across the
bq. maximum in each store.
bq.
bq.
bq. This addresses bug hbase-4645.
bq. https://issues.apache.org/jira/browse/hbase-4645
bq.
bq.
bq. Diffs
bq. -
bq.
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8c32839
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
966262b
bq.
bq. Diff: https://reviews.apache.org/r/2524/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Initial patch. v1.
bq.
bq. mvn test (running).
bq.
bq.
bq. Thanks,
bq.
bq. Amitanand
bq.
bq.

Edits Log recovery losing data across column families
-

Key: HBASE-4645
URL: https://issues.apache.org/jira/browse/HBASE-4645
Project: HBase
Issue Type: Bug
Affects Versions: 0.89.20100924, 0.92.0
Reporter: Amitanand Aiyer
Assignee: Amitanand Aiyer

[jira] [Commented] (HBASE-4536) Allow CF to retain deleted rows

2011-10-21 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13133141#comment-13133141
 ] 

Hudson commented on HBASE-4536:
---

Integrated in HBase-TRUNK #2353 (See 
[https://builds.apache.org/job/HBase-TRUNK/2353/])
HBASE-4536  Allow CF to retain deleted rows

larsh : 
Files : 
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/docbkx/book.xml
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/KeyValue.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Attributes.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Scan.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ColumnTracker.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ExplicitColumnTracker.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanWildcardColumnTracker.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
* /hbase/trunk/src/main/ruby/hbase/admin.rb
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestExplicitColumnTracker.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestKeepDeletes.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMinVersions.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWildcardColumnTracker.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java


 Allow CF to retain deleted rows
 ---

 Key: HBASE-4536
 URL: https://issues.apache.org/jira/browse/HBASE-4536
 Project: HBase
  Issue Type: New Feature
  Components: regionserver
Affects Versions: 0.92.0
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.94.0

 Attachments: 4536-v15.txt, 4536-v16.txt


 Parent allows for a cluster to retain rows for a TTL or keep a minimum number 
 of versions.
 However, if a client deletes a row all version older than the delete tomb 
 stone will be remove at the next major compaction (and even at memstore flush 
 - see HBASE-4241).
 There should be a way to retain those version to guard against software error.
 I see two options here:
 1. Add a new flag HColumnDescriptor. Something like RETAIN_DELETED.
 2. Folds this into the parent change. I.e. keep minimum-number-of-versions of 
 versions even past the delete marker.
 #1 would allow for more flexibility. #2 comes somewhat naturally with parent 
 (from a user viewpoint)
 Comments? Any other options?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4645) Edits Log recovery losing data across column families

[
https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13133151#comment-13133151
]

jirapos...@reviews.apache.org commented on HBASE-4645:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2524/#review2767
---

Ship it!

Thanks for the explanation Amit.

- Lars

Edits Log recovery losing data across column families
-

Key: HBASE-4645
URL: https://issues.apache.org/jira/browse/HBASE-4645
Project: HBase
Issue Type: Bug
Affects Versions: 0.89.20100924, 0.92.0
Reporter: Amitanand Aiyer
Assignee: Amitanand Aiyer

[jira] [Commented] (HBASE-4415) Add configuration script for setup HBase (hbase-setup-conf.sh)

2011-10-21 Thread Eric Yang (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13133174#comment-13133174
 ] 

Eric Yang commented on HBASE-4415:
--

Failed core tests, and javadoc warnings don't seem to be related to this patch.

 Add configuration script for setup HBase (hbase-setup-conf.sh)
 --

 Key: HBASE-4415
 URL: https://issues.apache.org/jira/browse/HBASE-4415
 Project: HBase
  Issue Type: New Feature
  Components: scripts
Affects Versions: 0.90.4, 0.92.0
 Environment: Java 6, Linux
Reporter: Eric Yang
Assignee: Eric Yang
 Fix For: 0.90.4, 0.92.0

 Attachments: HBASE-4415-1.patch, HBASE-4415-2.patch, 
 HBASE-4415-3.patch, HBASE-4415-4.patch, HBASE-4415-5.patch, 
 HBASE-4415-6.patch, HBASE-4415.patch


 The goal of this jura is to provide a installation script for configuring 
 HBase environment and configuration.  By using the same pattern of 
 *-setup-conf.sh for all Hadoop related projects.  For HBase, the usage of the 
 script looks like this:
 {noformat}
 usage: ./hbase-setup-conf.sh parameters
   Optional parameters:
 --hadoop-conf=/etc/hadoopSet Hadoop configuration directory 
 location
 --hadoop-home=/usr   Set Hadoop directory location
 --hadoop-namenode=localhost  Set Hadoop namenode hostname
 --hadoop-replication=3   Set HDFS replication
 --hbase-home=/usrSet HBase directory location
 --hbase-conf=/etc/hbase  Set HBase configuration 
 directory location
 --hbase-log=/var/log/hbase   Set HBase log directory location
 --hbase-pid=/var/run/hbase   Set HBase pid directory location
 --hbase-user=hbase   Set HBase user
 --java-home=/usr/java/defaultSet JAVA_HOME directory location
 --kerberos-realm=KERBEROS.EXAMPLE.COMSet Kerberos realm
 --kerberos-principal-id=_HOSTSet Kerberos principal ID 
 --keytab-dir=/etc/security/keytabs   Set keytab directory
 --regionservers=localhostSet regionservers hostnames
 --zookeeper-home=/usrSet ZooKeeper directory location
 --zookeeper-quorum=localhost Set ZooKeeper Quorum
 --zookeeper-snapshot=/var/lib/zookeeper  Set ZooKeeper snapshot location
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4532) Avoid top row seek by dedicated bloom filter for delete family bloom filter


 [ 
https://issues.apache.org/jira/browse/HBASE-4532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liyin Tang updated HBASE-4532:
--

Attachment: HBASE-4532-apache-trunk.patch

 Avoid top row seek by dedicated bloom filter for delete family bloom filter
 ---

 Key: HBASE-4532
 URL: https://issues.apache.org/jira/browse/HBASE-4532
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang
 Attachments: D27.1.patch, D27.1.patch, HBASE-4532-apache-trunk.patch, 
 hbase-4532-89-fb.patch


 The previous jira, HBASE-4469, is to avoid the top row seek operation if 
 row-col bloom filter is enabled. 
 This jira tries to avoid top row seek for all the cases by creating a 
 dedicated bloom filter only for delete family
 The only subtle use case is when we are interested in the top row with empty 
 column.
 For example, 
 we are interested in row1/cf1:/1/put.
 So we seek to the top row: row1/cf1:/MAX_TS/MAXIMUM. And the delete family 
 bloom filter will say there is NO delete family.
 Then it will avoid the top row seek and return a fake kv, which is the last 
 kv for this row (createLastOnRowCol).
 In this way, we have already missed the real kv we are interested in.
 The solution for the above problem is to disable this optimization if we are 
 trying to GET/SCAN a row with empty column.
 Evaluation from TestSeekOptimization:
 Previously:
 For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 So we can get about 10% more seek savings ONLY if the ROWCOL bloom filter is 
 enabled.[HBASE-4469]
 
 After this change:
 For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 So we can get about 10% more seek savings for ALL kinds of bloom filter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4552) multi-CF bulk load is not atomic across column families

2011-10-21 Thread Jonathan Hsieh (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13133190#comment-13133190
 ] 

Jonathan Hsieh commented on HBASE-4552:
---

Plan

1) Test to show there is an atomicity problem.  Likely just does not use 
LoadIncrementalHFiles
2) Fix for the region server side.
3) Rewrite of LoadIncrementalHFiles so that it groups the proper HFiles into 
the new bulkLoadHFile calls.  This will likely have two parallel steps - the 
first gather enough info to group HFiles and then the second that attempts to 
bulk load.

 multi-CF bulk load is not atomic across column families
 ---

 Key: HBASE-4552
 URL: https://issues.apache.org/jira/browse/HBASE-4552
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Assignee: Jonathan Hsieh
 Fix For: 0.92.0


 Currently the bulk load API simply imports one HFile at a time. With 
 multi-column-family support, this is inappropriate, since different CFs show 
 up separately. Instead, the IPC endpoint should take a of CF - HFiles, so we 
 can online them all under a single region-wide lock.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4588) The floating point arithmetic to validate memory allocation configurations need to be done as integers


[ 
https://issues.apache.org/jira/browse/HBASE-4588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13133201#comment-13133201
 ] 

Hudson commented on HBASE-4588:
---

Integrated in HBase-TRUNK #2354 (See 
[https://builds.apache.org/job/HBase-TRUNK/2354/])
HBASE-4588 The floating point arithmetic to validate memory allocation 
configurations need to be done as integers (dhruba)

jgray : 
Files : 
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java


 The floating point arithmetic to validate memory allocation configurations 
 need to be done as integers
 --

 Key: HBASE-4588
 URL: https://issues.apache.org/jira/browse/HBASE-4588
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.0, 0.94.0
Reporter: Jonathan Gray
Assignee: dhruba borthakur
Priority: Minor
 Fix For: 0.92.0

 Attachments: configVerify1.txt, configVerify2.txt


 The floating point arithmetic to validate memory allocation configurations 
 need to be done as integers.
 On our cluster, we had block cache = 0.6 and memstore = 0.2.  It was saying 
 this was  0.8 when it is actually equal.
 Minor bug but annoying nonetheless.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4627) Ability to specify a custom start/end to RegionSplitter

2011-10-21 Thread Phabricator (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HBASE-4627:
---

Attachment: D39.1.patch

nspiegelberg requested code review of [jira] [HBASE-4627] Ability to specify a 
custom start/end to RegionSplitter.
Reviewers: DUMMY_REVIEWER

  [HBASE-4627]

  added a custom start/end row to RegionSplitter.  Also solved
  an off-by-one error because the end row is prefix-inclusive and not
  exclusive.

  a href=https://issues.apache.org/jira/browse/HBASE-4489; title=Better key 
splitting in RegionSplitterdelHBASE-4489/del/a changed the default 
endKey on HexStringSplit from 7FFF... to ...  While this is correct, 
existing users of 0.90 RegionSplitter have 7FFF as the end key in their schema 
and the last region will not split properly under this new code.  We need to 
let the user specify a custom start/end key range for when situations like this 
arise.  Optimally, we should also write the start/end key in META so we could 
figure this out implicitly instead of requiring the user to explicitly specify 
it.

TEST PLAN
   - mvn test -Dtest=TestRegionSplitter

REVISION DETAIL
  http://reviews.facebook.net/D39

AFFECTED FILES
  src/main/java/org/apache/hadoop/hbase/util/Bytes.java
  src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
  src/test/java/org/apache/hadoop/hbase/util/TestRegionSplitter.java

MANAGE HERALD DIFFERENTIAL RULES
  http://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  http://reviews.facebook.net/herald/transcript/75/

Tip: use the X-Herald-Rules header to filter Herald messages in your client.


 Ability to specify a custom start/end to RegionSplitter
 ---

 Key: HBASE-4627
 URL: https://issues.apache.org/jira/browse/HBASE-4627
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.94.0
Reporter: Nicolas Spiegelberg
Assignee: Nicolas Spiegelberg
 Attachments: D39.1.patch, D39.1.patch


 HBASE-4489 changed the default endKey on HexStringSplit from 7FFF... to 
 ...  While this is correct, existing users of 0.90 RegionSplitter have 
 7FFF as the end key in their schema and the last region will not split 
 properly under this new code.  We need to let the user specify a custom 
 start/end key range for when situations like this arise.  Optimally, we 
 should also write the start/end key in META so we could figure this out 
 implicitly instead of requiring the user to explicitly specify it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4627) Ability to specify a custom start/end to RegionSplitter

2011-10-21 Thread Phabricator (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HBASE-4627:
---

Attachment: D39.1.patch

nspiegelberg requested code review of [jira] [HBASE-4627] Ability to specify a 
custom start/end to RegionSplitter.
Reviewers: DUMMY_REVIEWER

  [HBASE-4627]

  added a custom start/end row to RegionSplitter.  Also solved
  an off-by-one error because the end row is prefix-inclusive and not
  exclusive.

  a href=https://issues.apache.org/jira/browse/HBASE-4489; title=Better key 
splitting in RegionSplitterdelHBASE-4489/del/a changed the default 
endKey on HexStringSplit from 7FFF... to ...  While this is correct, 
existing users of 0.90 RegionSplitter have 7FFF as the end key in their schema 
and the last region will not split properly under this new code.  We need to 
let the user specify a custom start/end key range for when situations like this 
arise.  Optimally, we should also write the start/end key in META so we could 
figure this out implicitly instead of requiring the user to explicitly specify 
it.

TEST PLAN
   - mvn test -Dtest=TestRegionSplitter

REVISION DETAIL
  http://reviews.facebook.net/D39

AFFECTED FILES
  src/main/java/org/apache/hadoop/hbase/util/Bytes.java
  src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
  src/test/java/org/apache/hadoop/hbase/util/TestRegionSplitter.java

MANAGE HERALD DIFFERENTIAL RULES
  http://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  http://reviews.facebook.net/herald/transcript/75/

Tip: use the X-Herald-Rules header to filter Herald messages in your client.


 Ability to specify a custom start/end to RegionSplitter
 ---

 Key: HBASE-4627
 URL: https://issues.apache.org/jira/browse/HBASE-4627
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.94.0
Reporter: Nicolas Spiegelberg
Assignee: Nicolas Spiegelberg
 Attachments: D39.1.patch, D39.1.patch


 HBASE-4489 changed the default endKey on HexStringSplit from 7FFF... to 
 ...  While this is correct, existing users of 0.90 RegionSplitter have 
 7FFF as the end key in their schema and the last region will not split 
 properly under this new code.  We need to let the user specify a custom 
 start/end key range for when situations like this arise.  Optimally, we 
 should also write the start/end key in META so we could figure this out 
 implicitly instead of requiring the user to explicitly specify it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4578) NPE when altering a table that has moving regions

2011-10-21 Thread gaojinchao (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13133238#comment-13133238
 ] 

gaojinchao commented on HBASE-4578:
---

I try to make a patch. But I amn't sure. Can you review this patch before 
testing in a real cluster ?

 NPE when altering a table that has moving regions
 -

 Key: HBASE-4578
 URL: https://issues.apache.org/jira/browse/HBASE-4578
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Priority: Blocker
 Fix For: 0.92.0


 I'm still not a 100% sure on the source of this error, but here's what I was 
 able to get twice while altering a table that was doing a bunch of splits:
 {quote}
 2011-10-11 23:48:59,344 INFO 
 org.apache.hadoop.hbase.master.handler.SplitRegionHandler: Handled SPLIT 
 report); 
 parent=TestTable,0002608338,1318376880454.a75d6815fdfc513fb1c8aabe086c6763. 
 daughter 
 a=TestTable,0002608338,1318376938764.ef170ff6cd8695dc8aec92e542dc9ac1.daughter
  b=TestTable,0003301408,1318376938764.36eb2530341bd46888ede312c5559b5d.
 2011-10-11 23:49:09,579 DEBUG 
 org.apache.hadoop.hbase.master.handler.TableEventHandler: Ignoring table not 
 disabled exception for supporting online schema changes.
 2011-10-11 23:49:09,580 INFO 
 org.apache.hadoop.hbase.master.handler.TableEventHandler: Handling table 
 operation C_M_MODIFY_TABLE on table TestTable
 2011-10-11 23:49:09,612 INFO org.apache.hadoop.hbase.util.FSUtils: 
 TableInfoPath = hdfs://sv4r11s38:9100/hbase/TestTable/.tableinfo tmpPath = 
 hdfs://sv4r11s38:9100/hbase/TestTable/.tmp/.tableinfo.1318376949612
 2011-10-11 23:49:09,692 INFO org.apache.hadoop.hbase.util.FSUtils: 
 TableDescriptor stored. TableInfoPath = 
 hdfs://sv4r11s38:9100/hbase/TestTable/.tableinfo
 2011-10-11 23:49:09,693 INFO org.apache.hadoop.hbase.util.FSUtils: Updated 
 tableinfo=hdfs://sv4r11s38:9100/hbase/TestTable/.tableinfo to blah
 2011-10-11 23:49:09,695 INFO 
 org.apache.hadoop.hbase.master.handler.TableEventHandler: Bucketing regions 
 by region server...
 2011-10-11 23:49:09,695 DEBUG org.apache.hadoop.hbase.client.MetaScanner: 
 Scanning .META. starting at row=TestTable,,00 for max=2147483647 
 rows
 2011-10-11 23:49:09,709 DEBUG 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
 The connection to hconnection-0x132f043bbde02e9 has been closed.
 2011-10-11 23:49:09,709 ERROR org.apache.hadoop.hbase.executor.EventHandler: 
 Caught throwable while processing event C_M_MODIFY_TABLE
 java.lang.NullPointerException
   at java.util.TreeMap.getEntry(TreeMap.java:324)
   at java.util.TreeMap.containsKey(TreeMap.java:209)
   at 
 org.apache.hadoop.hbase.master.handler.TableEventHandler.reOpenAllRegions(TableEventHandler.java:114)
   at 
 org.apache.hadoop.hbase.master.handler.TableEventHandler.process(TableEventHandler.java:90)
   at 
 org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:168)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
 {quote}
 The first time the shell reported that all the regions were updated 
 correctly, the second time it got stuck for a while:
 {quote}
 6/14 regions updated.
 0/14 regions updated.
 ...
 0/14 regions updated.
 2/16 regions updated.
 ...
 2/16 regions updated.
 8/9 regions updated.
 ...
 8/9 regions updated.
 {quote}
 After which I killed it, redid the alter and it worked.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4578) NPE when altering a table that has moving regions

2011-10-21 Thread gaojinchao (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

gaojinchao updated HBASE-4578:
--

Attachment: HBASE-4578_trial_Trunk.patch

 NPE when altering a table that has moving regions
 -

 Key: HBASE-4578
 URL: https://issues.apache.org/jira/browse/HBASE-4578
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Priority: Blocker
 Fix For: 0.92.0

 Attachments: HBASE-4578_trial_Trunk.patch


 I'm still not a 100% sure on the source of this error, but here's what I was 
 able to get twice while altering a table that was doing a bunch of splits:
 {quote}
 2011-10-11 23:48:59,344 INFO 
 org.apache.hadoop.hbase.master.handler.SplitRegionHandler: Handled SPLIT 
 report); 
 parent=TestTable,0002608338,1318376880454.a75d6815fdfc513fb1c8aabe086c6763. 
 daughter 
 a=TestTable,0002608338,1318376938764.ef170ff6cd8695dc8aec92e542dc9ac1.daughter
  b=TestTable,0003301408,1318376938764.36eb2530341bd46888ede312c5559b5d.
 2011-10-11 23:49:09,579 DEBUG 
 org.apache.hadoop.hbase.master.handler.TableEventHandler: Ignoring table not 
 disabled exception for supporting online schema changes.
 2011-10-11 23:49:09,580 INFO 
 org.apache.hadoop.hbase.master.handler.TableEventHandler: Handling table 
 operation C_M_MODIFY_TABLE on table TestTable
 2011-10-11 23:49:09,612 INFO org.apache.hadoop.hbase.util.FSUtils: 
 TableInfoPath = hdfs://sv4r11s38:9100/hbase/TestTable/.tableinfo tmpPath = 
 hdfs://sv4r11s38:9100/hbase/TestTable/.tmp/.tableinfo.1318376949612
 2011-10-11 23:49:09,692 INFO org.apache.hadoop.hbase.util.FSUtils: 
 TableDescriptor stored. TableInfoPath = 
 hdfs://sv4r11s38:9100/hbase/TestTable/.tableinfo
 2011-10-11 23:49:09,693 INFO org.apache.hadoop.hbase.util.FSUtils: Updated 
 tableinfo=hdfs://sv4r11s38:9100/hbase/TestTable/.tableinfo to blah
 2011-10-11 23:49:09,695 INFO 
 org.apache.hadoop.hbase.master.handler.TableEventHandler: Bucketing regions 
 by region server...
 2011-10-11 23:49:09,695 DEBUG org.apache.hadoop.hbase.client.MetaScanner: 
 Scanning .META. starting at row=TestTable,,00 for max=2147483647 
 rows
 2011-10-11 23:49:09,709 DEBUG 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
 The connection to hconnection-0x132f043bbde02e9 has been closed.
 2011-10-11 23:49:09,709 ERROR org.apache.hadoop.hbase.executor.EventHandler: 
 Caught throwable while processing event C_M_MODIFY_TABLE
 java.lang.NullPointerException
   at java.util.TreeMap.getEntry(TreeMap.java:324)
   at java.util.TreeMap.containsKey(TreeMap.java:209)
   at 
 org.apache.hadoop.hbase.master.handler.TableEventHandler.reOpenAllRegions(TableEventHandler.java:114)
   at 
 org.apache.hadoop.hbase.master.handler.TableEventHandler.process(TableEventHandler.java:90)
   at 
 org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:168)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
 {quote}
 The first time the shell reported that all the regions were updated 
 correctly, the second time it got stuck for a while:
 {quote}
 6/14 regions updated.
 0/14 regions updated.
 ...
 0/14 regions updated.
 2/16 regions updated.
 ...
 2/16 regions updated.
 8/9 regions updated.
 ...
 8/9 regions updated.
 {quote}
 After which I killed it, redid the alter and it worked.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4578) NPE when altering a table that has moving regions