date:20111013


 [ 
https://issues.apache.org/jira/browse/HBASE-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Gray updated HBASE-3417:
-

Attachment: HBASE-3417-redux-v1.patch

Adds a new unit test in TestFromClientSide for testing cache on write and evict 
on close by looking inside the block cache.

This test fails on 92 and trunk without the rest of the changes in this patch 
(CacheOnWrite is currently completely broken with hfile v2).

 CacheOnWrite is using the temporary output path for block names, need to use 
 a more consistent block naming scheme
 --

 Key: HBASE-3417
 URL: https://issues.apache.org/jira/browse/HBASE-3417
 Project: HBase
  Issue Type: Bug
  Components: io, regionserver
Affects Versions: 0.92.0
Reporter: Jonathan Gray
Assignee: Jonathan Gray
Priority: Critical
 Fix For: 0.92.0

 Attachments: HBASE-3417-redux-v1.patch, HBASE-3417-v1.patch, 
 HBASE-3417-v2.patch, HBASE-3417-v5.patch


 Currently the block names used in the block cache are built using the 
 filesystem path.  However, for cache on write, the path is a temporary output 
 file.
 The original COW patch actually made some modifications to block naming stuff 
 to make it more consistent but did not do enough.  Should add a separate 
 method somewhere for generating block names using some more easily mocked 
 scheme (rather than just raw path as we generate a random unique file name 
 twice, once for tmp and then again when moved into place).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4558) Refactor TestOpenedRegionHandler and TestOpenRegionHandler.


[ 
https://issues.apache.org/jira/browse/HBASE-4558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126372#comment-13126372
 ] 

Jonathan Gray commented on HBASE-4558:
--

Did this break the build?  TestMasterFailover is not compiling for me.

 Refactor TestOpenedRegionHandler and TestOpenRegionHandler.
 ---

 Key: HBASE-4558
 URL: https://issues.apache.org/jira/browse/HBASE-4558
 Project: HBase
  Issue Type: Improvement
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Minor
 Fix For: 0.92.0

 Attachments: HBASE-4558_1.patch, HBASE-4558_2.patch, 
 HBASE-4558_3.patch


 This is an improvement task taken up to refactor TestOpenedRegionandler and 
 TestOpenRegionHandler so that MockServer and MockRegionServerServices can be 
 accessed from a common utility package.
 If we do this then one of the testcases in TestOpenedRegionHandler need not 
 start up a cluster and also moving it into a common package will help in 
 mocking the server for future testcases.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4558) Refactor TestOpenedRegionHandler and TestOpenRegionHandler.

2011-10-13 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126373#comment-13126373
 ] 

Jonathan Gray commented on HBASE-4558:
--

-  metaRegion, regionServer);
+  metaRegion, regionServer.getServerName());

?

 Refactor TestOpenedRegionHandler and TestOpenRegionHandler.
 ---

 Key: HBASE-4558
 URL: https://issues.apache.org/jira/browse/HBASE-4558
 Project: HBase
  Issue Type: Improvement
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Minor
 Fix For: 0.92.0

 Attachments: HBASE-4558_1.patch, HBASE-4558_2.patch, 
 HBASE-4558_3.patch


 This is an improvement task taken up to refactor TestOpenedRegionandler and 
 TestOpenRegionHandler so that MockServer and MockRegionServerServices can be 
 accessed from a common utility package.
 If we do this then one of the testcases in TestOpenedRegionHandler need not 
 start up a cluster and also moving it into a common package will help in 
 mocking the server for future testcases.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3417) CacheOnWrite is using the temporary output path for block names, need to use a more consistent block naming scheme


[ 
https://issues.apache.org/jira/browse/HBASE-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126378#comment-13126378
 ] 

jirapos...@reviews.apache.org commented on HBASE-3417:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2379/
---

(Updated 2011-10-13 06:31:13.710051)


Review request for hbase, Dhruba Borthakur, Michael Stack, and Mikhail Bautin.


Changes
---

Applies cleanly.


Summary
---

Adds a new test that fails w/o this fix and changes the naming scheme for 
storefiles to use UUID instead of random longs as ascii.

The big change is that the name of the tmp file used when flushing and 
compacting is the same name (but different dir) when moved in place.  This 
makes it so block names are consistent for COW.


This addresses bug HBASE-3417.
https://issues.apache.org/jira/browse/HBASE-3417


Diffs (updated)
-

  /src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java 
1182679 
  /src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java 1182679 
  /src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1182679 
  /src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 1182679 
  /src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java 1182679 

Diff: https://reviews.apache.org/r/2379/diff


Testing
---

TestFromClientSide and TestCacheOnWrite both working


Thanks,

Jonathan



 CacheOnWrite is using the temporary output path for block names, need to use 
 a more consistent block naming scheme
 --

 Key: HBASE-3417
 URL: https://issues.apache.org/jira/browse/HBASE-3417
 Project: HBase
  Issue Type: Bug
  Components: io, regionserver
Affects Versions: 0.92.0
Reporter: Jonathan Gray
Assignee: Jonathan Gray
Priority: Critical
 Fix For: 0.92.0

 Attachments: HBASE-3417-redux-v1.patch, HBASE-3417-v1.patch, 
 HBASE-3417-v2.patch, HBASE-3417-v5.patch


 Currently the block names used in the block cache are built using the 
 filesystem path.  However, for cache on write, the path is a temporary output 
 file.
 The original COW patch actually made some modifications to block naming stuff 
 to make it more consistent but did not do enough.  Should add a separate 
 method somewhere for generating block names using some more easily mocked 
 scheme (rather than just raw path as we generate a random unique file name 
 twice, once for tmp and then again when moved into place).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4558) Refactor TestOpenedRegionHandler and TestOpenRegionHandler.

2011-10-13 Thread Hudson (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126461#comment-13126461
 ] 

Hudson commented on HBASE-4558:
---

Integrated in HBase-0.92 #60 (See 
[https://builds.apache.org/job/HBase-0.92/60/])
HBASE-4558 - Addendum for TestMasterFailover (Ram)
HBASE-4558 Refactor TestOpenedRegionHandler and TestOpenRegionHandler. (Ram)

ramkrishna : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java

ramkrishna : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* 
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestOpenedRegionHandler.java
* 
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/handler/MockRegionServerServices.java
* 
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/handler/MockServer.java
* 
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestCloseRegionHandler.java
* 
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestOpenRegionHandler.java
* 
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java
* 
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/MockServer.java


 Refactor TestOpenedRegionHandler and TestOpenRegionHandler.
 ---

 Key: HBASE-4558
 URL: https://issues.apache.org/jira/browse/HBASE-4558
 Project: HBase
  Issue Type: Improvement
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Minor
 Fix For: 0.92.0

 Attachments: HBASE-4558_1.patch, HBASE-4558_2.patch, 
 HBASE-4558_3.patch


 This is an improvement task taken up to refactor TestOpenedRegionandler and 
 TestOpenRegionHandler so that MockServer and MockRegionServerServices can be 
 accessed from a common utility package.
 If we do this then one of the testcases in TestOpenedRegionHandler need not 
 start up a cluster and also moving it into a common package will help in 
 mocking the server for future testcases.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4519) 25s sleep when expiring sessions in tests

2011-10-13 Thread nkeywal (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126483#comment-13126483
 ] 

nkeywal commented on HBASE-4519:


Seems that HBASE-1291 is on the same subject, and it says:

bq. We currently use 10 seconds for zookeeper.session.timeout. I tried bumping 
this down recently to 2 seconds, and it broke the tests. We need to investigate 
why this change breaks our test.

So the timeout have been increased somewhere...

 25s sleep when expiring sessions in tests
 -

 Key: HBASE-4519
 URL: https://issues.apache.org/jira/browse/HBASE-4519
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.90.4
Reporter: Jean-Daniel Cryans
 Fix For: 0.92.0


 There's a hardcoded 25 seconds sleep in HBaseTestingUtility.expireSession: 
 {code}
 int sessionTimeout = 5 * 1000; // 5 seconds
 ...
 final long sleep = sessionTimeout * 5L;
 LOG.info(ZK Closed Session 0x + Long.toHexString(sessionID) +
   ; sleeping= + sleep);
 Thread.sleep(sleep);
 {code}
 I'm pretty sure this can be lowered at lot, and it would speed up a couple of 
 tests. The only thing I'm afraid of is if this was made to accomodate flaky 
 tests.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-1291) Lowering zookeeper.session.timeout breaks tests

2011-10-13 Thread stack (Resolved) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-1291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-1291.
--

Resolution: Duplicate

Resolving as duplicate of HBASE-4519

 Lowering zookeeper.session.timeout breaks tests
 ---

 Key: HBASE-1291
 URL: https://issues.apache.org/jira/browse/HBASE-1291
 Project: HBase
  Issue Type: Bug
  Components: test
Reporter: Nitay Joffe
Priority: Minor

 We currently use 10 seconds for zookeeper.session.timeout. I tried bumping 
 this down recently to 2 seconds, and it broke the tests. We need to 
 investigate why this change breaks our test.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4519) 25s sleep when expiring sessions in tests

2011-10-13 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126631#comment-13126631
 ] 

stack commented on HBASE-4519:
--

I resolved hbase-1291 as dup of this.  Good one nkeywal.

 25s sleep when expiring sessions in tests
 -

 Key: HBASE-4519
 URL: https://issues.apache.org/jira/browse/HBASE-4519
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.90.4
Reporter: Jean-Daniel Cryans
 Fix For: 0.92.0


 There's a hardcoded 25 seconds sleep in HBaseTestingUtility.expireSession: 
 {code}
 int sessionTimeout = 5 * 1000; // 5 seconds
 ...
 final long sleep = sessionTimeout * 5L;
 LOG.info(ZK Closed Session 0x + Long.toHexString(sessionID) +
   ; sleeping= + sleep);
 Thread.sleep(sleep);
 {code}
 I'm pretty sure this can be lowered at lot, and it would speed up a couple of 
 tests. The only thing I'm afraid of is if this was made to accomodate flaky 
 tests.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3417) CacheOnWrite is using the temporary output path for block names, need to use a more consistent block naming scheme

[
https://issues.apache.org/jira/browse/HBASE-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126639#comment-13126639
]

jirapos...@reviews.apache.org commented on HBASE-3417:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2379/#review2548
---

Ship it!

Just a few questions in the below. I'm good w/ commit. Would suggest open new
issue rather than reopen an old one in future -- makes it easier on the fellas
coming up behind us trying to make some sense of what we did. Good stuff.

/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
https://reviews.apache.org/r/2379/#comment5719

This is the tmp name? Thats what you want? I don't see where it gets set
to renamed filename.

/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
https://reviews.apache.org/r/2379/#comment5720

Why? To be more specific?

Oh, I see how about not removing the '-' from uuid? Is it illegal in
filesystem?

/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
https://reviews.apache.org/r/2379/#comment5721

This is better.

- Michael

On 2011-10-13 06:31:13, Jonathan Gray wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2379/
bq. ---
bq.
bq. (Updated 2011-10-13 06:31:13)
bq.
bq.
bq. Review request for hbase, Dhruba Borthakur, Michael Stack, and Mikhail
Bautin.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Adds a new test that fails w/o this fix and changes the naming scheme for
storefiles to use UUID instead of random longs as ascii.
bq.
bq. The big change is that the name of the tmp file used when flushing and
compacting is the same name (but different dir) when moved in place. This
makes it so block names are consistent for COW.
bq.
bq.
bq. This addresses bug HBASE-3417.
bq. https://issues.apache.org/jira/browse/HBASE-3417
bq.
bq.
bq. Diffs
bq. -
bq.
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java
1182679
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java 1182679
bq./src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1182679
bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
1182679
bq./src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
1182679
bq.
bq. Diff: https://reviews.apache.org/r/2379/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. TestFromClientSide and TestCacheOnWrite both working
bq.
bq.
bq. Thanks,
bq.
bq. Jonathan
bq.
bq.

CacheOnWrite is using the temporary output path for block names, need to use
a more consistent block naming scheme
--

Key: HBASE-3417
URL: https://issues.apache.org/jira/browse/HBASE-3417
Project: HBase
Issue Type: Bug
Components: io, regionserver
Affects Versions: 0.92.0
Reporter: Jonathan Gray
Assignee: Jonathan Gray
Priority: Critical
Fix For: 0.92.0

Attachments: HBASE-3417-redux-v1.patch, HBASE-3417-v1.patch,
HBASE-3417-v2.patch, HBASE-3417-v5.patch

Currently the block names used in the block cache are built using the
filesystem path. However, for cache on write, the path is a temporary output
file.
The original COW patch actually made some modifications to block naming stuff
to make it more consistent but did not do enough. Should add a separate
method somewhere for generating block names using some more easily mocked
scheme (rather than just raw path as we generate a random unique file name
twice, once for tmp and then again when moved into place).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4570) Scan ACID problem with concurrent puts.


[ 
https://issues.apache.org/jira/browse/HBASE-4570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126644#comment-13126644
 ] 

stack commented on HBASE-4570:
--

Good on you Jon.  Keep digging (smile).

 Scan ACID problem with concurrent puts.
 ---

 Key: HBASE-4570
 URL: https://issues.apache.org/jira/browse/HBASE-4570
 Project: HBase
  Issue Type: Bug
  Components: client, regionserver
Affects Versions: 0.90.1, 0.90.3
Reporter: Jonathan Hsieh
 Attachments: hbase-4570.tgz


 When scanning a table sometimes rows that have multiple column families get 
 split into two rows if there are concurrent writes.  In this particular case 
 we are overwriting the contents of a Get directly back onto itself as a Put.
 For example, this is a two cf row (with f1, f2, .. f9 cfs).  It is 
 actually returned as two rows (#55 and #56). Interestingly if the two were 
 merged we would have a single proper row.
 Row row024461 had time stamps: [55: 
 keyvalues={row024461/f0:data/1318200440867/Put/vlen=1000, 
 row024461/f0:qual/1318200440867/Put/vlen=10, 
 row024461/f1:data/1318200440867/Put/vlen=1000, 
 row024461/f1:qual/1318200440867/Put/vlen=10, 
 row024461/f2:data/1318200440867/Put/vlen=1000, 
 row024461/f2:qual/1318200440867/Put/vlen=10, 
 row024461/f3:data/1318200440867/Put/vlen=1000, 
 row024461/f3:qual/1318200440867/Put/vlen=10, 
 row024461/f4:data/1318200440867/Put/vlen=1000, 
 row024461/f4:qual/1318200440867/Put/vlen=10}, 
 56: keyvalues={row024461/f5:data/1318200440867/Put/vlen=1000, 
 row024461/f5:qual/1318200440867/Put/vlen=10, 
 row024461/f6:data/1318200440867/Put/vlen=1000, 
 row024461/f6:qual/1318200440867/Put/vlen=10, 
 row024461/f7:data/1318200440867/Put/vlen=1000, 
 row024461/f7:qual/1318200440867/Put/vlen=10, 
 row024461/f8:data/1318200440867/Put/vlen=1000, 
 row024461/f8:qual/1318200440867/Put/vlen=10, 
 row024461/f9:data/1318200440867/Put/vlen=1000, 
 row024461/f9:qual/1318200440867/Put/vlen=10}]
 I've only tested this on 0.90.1+patches and 0.90.3+patches, but it is 
 consistent and duplicatable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4540) OpenedRegionHandler is not enforcing atomicity of the operation it is performing


[ 
https://issues.apache.org/jira/browse/HBASE-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126646#comment-13126646
 ] 

stack commented on HBASE-4540:
--

Can we resolve this issue now Ram?

 OpenedRegionHandler is not enforcing atomicity of the operation it is 
 performing
 

 Key: HBASE-4540
 URL: https://issues.apache.org/jira/browse/HBASE-4540
 Project: HBase
  Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Fix For: 0.92.0

 Attachments: HBASE-4540_1.patch, HBASE-4540_90.patch, 
 HBASE-4540_90_1.patch


 - OpenedRegionHandler has not yet deleted the znode of the region R1 opened 
 by RS1.
 - RS1 goes down.
 - Servershutdownhandler assigns the region R1 to RS2.
 - The znode of R1 is moved to OFFLINE state by master or OPENING state by 
 RS2 if RS2 has started opening the region.
 - Now the first OpenedRegionHandler tries to delete the znode thinking its 
 in OPENED state but fails.
 - Though it fails it removes the node from RIT and adds RS1 as the owner of 
 R1 in master's memory.
 - Now when RS2 completes opening the region the master is not able to open 
 the region as already the reigon has been deleted from RIT.
 {code}
 Master
 ==
 2011-10-05 20:49:45,301 INFO 
 org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Finished 
 processing of shutdown of linux146,60020,1317827727647
 2011-10-05 20:49:54,177 DEBUG org.apache.hadoop.hbase.master.HMaster: Not 
 running balancer because 1 region(s) in transition: 
 {3e69d628a8bd8e9b7c5e7a2a6e03aad9=t1,,1317827883842.3e69d628a8bd8e9b7c5e7a2a6e03aad9.
  state=PENDING_OPEN, ts=1317827985272, server=linux76,60020,1317827746847}
 2011-10-05 20:49:57,720 DEBUG 
 org.apache.hadoop.hbase.master.AssignmentManager: Handling 
 transition=M_ZK_REGION_OFFLINE, server=linux76,6,1317827742012, 
 region=3e69d628a8bd8e9b7c5e7a2a6e03aad9
 2011-10-05 20:50:14,501 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
 master:6-0x132d3dc13090023 Deleting existing unassigned node for 
 3e69d628a8bd8e9b7c5e7a2a6e03aad9 that is in expected state RS_ZK_REGION_OPENED
 2011-10-05 20:50:14,505 WARN org.apache.hadoop.hbase.zookeeper.ZKAssign: 
 master:6-0x132d3dc13090023 Attempting to delete unassigned node 
 3e69d628a8bd8e9b7c5e7a2a6e03aad9 in RS_ZK_REGION_OPENED state but node is in 
 RS_ZK_REGION_OPENING state
 After the region is opened in RS2
 =
 2011-10-05 20:50:48,066 DEBUG 
 org.apache.hadoop.hbase.master.AssignmentManager: Handling 
 transition=RS_ZK_REGION_OPENING, server=linux76,60020,1317827746847, 
 region=3e69d628a8bd8e9b7c5e7a2a6e03aad9, which is more than 15 seconds late
 2011-10-05 20:50:48,290 WARN 
 org.apache.hadoop.hbase.master.AssignmentManager: Received OPENING for region 
 3e69d628a8bd8e9b7c5e7a2a6e03aad9 from server linux76,60020,1317827746847 but 
 region was in  the state null and not in expected PENDING_OPEN or OPENING 
 states
 2011-10-05 20:50:53,743 DEBUG 
 org.apache.hadoop.hbase.master.AssignmentManager: Handling 
 transition=RS_ZK_REGION_OPENING, server=linux76,60020,1317827746847, 
 region=3e69d628a8bd8e9b7c5e7a2a6e03aad9
 2011-10-05 20:50:54,182 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: 
 Scanned 1 catalog row(s) and gc'd 0 unreferenced parent region(s)
 2011-10-05 20:50:54,397 WARN 
 org.apache.hadoop.hbase.master.AssignmentManager: Received OPENING for region 
 3e69d628a8bd8e9b7c5e7a2a6e03aad9 from server linux76,60020,1317827746847 but 
 region was in  the state null and not in expected PENDING_OPEN or OPENING 
 states
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4556) Fix all incorrect uses of InternalScanner.next(...)


[ 
https://issues.apache.org/jira/browse/HBASE-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126648#comment-13126648
 ] 

stack commented on HBASE-4556:
--

+1 on commit of v1

 Fix all incorrect uses of InternalScanner.next(...)
 ---

 Key: HBASE-4556
 URL: https://issues.apache.org/jira/browse/HBASE-4556
 Project: HBase
  Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Attachments: 4556-v1.txt, 4556.txt


 There are cases all over the code where InternalScanner.next(...) is not used 
 correctly.
 I see this a lot:
 {code}
 while(scanner.next(...)) {
 }
 {code}
 The correct pattern is:
 {code}
 boolean more = false;
 do {
more = scanner.next(...);
 } while (more);
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4532) Avoid top row seek by dedicated bloom filter for delete family


[ 
https://issues.apache.org/jira/browse/HBASE-4532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126649#comment-13126649
 ] 

stack commented on HBASE-4532:
--

OK. I was confused.  I'm +0 on this patch (since I am not familiar with what is 
going on here -- it looks innocuous enough).  Jon you going to commit?

 Avoid top row seek by dedicated bloom filter for delete family
 --

 Key: HBASE-4532
 URL: https://issues.apache.org/jira/browse/HBASE-4532
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang

 HBASE-4469 avoids the top row seek operation if row-col bloom filter is 
 enabled. 
 This jira tries to avoid top row seek for all the cases by creating a 
 dedicated bloom filter only for delete family.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4532) Avoid top row seek by dedicated bloom filter for delete family

2011-10-13 Thread stack (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4532:
-

Comment: was deleted

(was: OK. I was confused.  I'm +0 on this patch (since I am not familiar with 
what is going on here -- it looks innocuous enough).  Jon you going to commit?)

 Avoid top row seek by dedicated bloom filter for delete family
 --

 Key: HBASE-4532
 URL: https://issues.apache.org/jira/browse/HBASE-4532
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang

 HBASE-4469 avoids the top row seek operation if row-col bloom filter is 
 enabled. 
 This jira tries to avoid top row seek for all the cases by creating a 
 dedicated bloom filter only for delete family.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4469) Avoid top row seek by looking up bloomfilter

2011-10-13 Thread ramkrishna.s.vasudevan (Updated) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126651#comment-13126651
 ] 

stack commented on HBASE-4469:
--

OK. I was confused. I'm +0 on this patch (since I am not familiar with what is 
going on here – it looks innocuous enough on review). Jon you going to commit?

 Avoid top row seek by looking up bloomfilter
 

 Key: HBASE-4469
 URL: https://issues.apache.org/jira/browse/HBASE-4469
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang

 The problem is that when seeking for the row/col in the hfile, we will go to 
 top of the row in order to check for row delete marker (delete family). 
 However, if the bloomfilter is enabled for the column family, then if a 
 delete family operation is done on a row, the row is already being added to 
 bloomfilter. We can take advantage of this factor to avoid seeking to the top 
 of row.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4586) ThriftServer doesn't expose autoflush/writebuffer settings needed for write-heavy loads

2011-10-13 Thread Norbert Burger (Created) (JIRA)

ThriftServer doesn't expose autoflush/writebuffer settings needed for 
write-heavy loads
---

 Key: HBASE-4586
 URL: https://issues.apache.org/jira/browse/HBASE-4586
 Project: HBase
  Issue Type: Bug
  Components: thrift
Affects Versions: 0.90.4
Reporter: Norbert Burger


For write-heavy/bulk-load environments, two common recommendations are to 
disable autoFlush and increase the size of the write buffer:

HTable.setAutoFlush(false)
HTable.setWriteBufferSize(long writeBufferSize)

Neither of exposed via the Thrift API, which seems to preclude using the Thrift 
interface for write-heavy environments.

As a workaround, we could specify hbase.client.write.buffer in hbase-site.xml, 
but there is unfortunately not an equivalent configuration setting for 
autoflush.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4462) Properly treating SocketTimeoutException

2011-10-13 Thread Ted Yu (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4462:
--

Fix Version/s: (was: 0.92.0)
   0.90.5

 Properly treating SocketTimeoutException
 

 Key: HBASE-4462
 URL: https://issues.apache.org/jira/browse/HBASE-4462
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.90.4
Reporter: Jean-Daniel Cryans
 Fix For: 0.90.5


 SocketTimeoutException is currently treated like any IOE inside of 
 HCM.getRegionServerWithRetries and I think this is a problem. This method 
 should only do retries in cases where we are pretty sure the operation will 
 complete, but with STE we already waited for (by default) 60 seconds and 
 nothing happened.
 I found this while debugging Douglas Campbell's problem on the mailing list 
 where it seemed like he was using the same scanner from multiple threads, but 
 actually it was just the same client doing retries while the first run didn't 
 even finish yet (that's another problem). You could see the first scanner, 
 then up to two other handlers waiting for it to finish in order to run 
 (because of the synchronization on RegionScanner).
 So what should we do? We could treat STE as a DoNotRetryException and let the 
 client deal with it, or we could retry only once.
 There's also the option of having a different behavior for get/put/icv/scan, 
 the issue with operations that modify a cell is that you don't know if the 
 operation completed or not (same when a RS dies hard after completing let's 
 say a Put but just before returning to the client).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4459) HbaseObjectWritable code is a byte, we will eventually run out of codes


 [ 
https://issues.apache.org/jira/browse/HBASE-4459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-4459:
--

Attachment: 4459.txt

Pls review the patch.

 HbaseObjectWritable code is a byte, we will eventually run out of codes
 ---

 Key: HBASE-4459
 URL: https://issues.apache.org/jira/browse/HBASE-4459
 Project: HBase
  Issue Type: Bug
  Components: io
Reporter: Jonathan Gray
Assignee: ramkrishna.s.vasudevan
Priority: Critical
 Fix For: 0.92.0

 Attachments: 4459.txt


 There are about 90 classes/codes in HbaseObjectWritable currently and 
 Byte.MAX_VALUE is 127.  In addition, anyone wanting to add custom classes but 
 not break compatibility might want to leave a gap before using codes and 
 that's difficult in such limited space.
 Eventually we should get rid of this pattern that makes compatibility 
 difficult (better client/server protocol handshake) but we should probably at 
 least bump this to a short for 0.94.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4459) HbaseObjectWritable code is a byte, we will eventually run out of codes

2011-10-13 Thread ramkrishna.s.vasudevan (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-4459:
--

Status: Patch Available  (was: Open)

 HbaseObjectWritable code is a byte, we will eventually run out of codes
 ---

 Key: HBASE-4459
 URL: https://issues.apache.org/jira/browse/HBASE-4459
 Project: HBase
  Issue Type: Bug
  Components: io
Reporter: Jonathan Gray
Assignee: ramkrishna.s.vasudevan
Priority: Critical
 Fix For: 0.92.0

 Attachments: 4459.txt


 There are about 90 classes/codes in HbaseObjectWritable currently and 
 Byte.MAX_VALUE is 127.  In addition, anyone wanting to add custom classes but 
 not break compatibility might want to leave a gap before using codes and 
 that's difficult in such limited space.
 Eventually we should get rid of this pattern that makes compatibility 
 difficult (better client/server protocol handshake) but we should probably at 
 least bump this to a short for 0.94.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4560) Script to find hanging test cases in build

2011-10-13 Thread ramkrishna.s.vasudevan (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-4560:
--

Attachment: findHangingTest.sh

Here is updated script which can deal with the following (seen in 0.92 builds):

Forking command line: /bin/sh -c cd 
/home/jenkins/jenkins-slave/workspace/HBase-0.92/trunk  
/home/jenkins/tools/java/jdk1.6.0_20-32/jre/bin/java -enableassertions 
-Xmx1400m -jar 
/home/jenkins/jenkins-slave/workspace/HBase-0.92/trunk/target/surefire/surefirebooter4703782102378142256.jar
 
/home/jenkins/jenkins-slave/workspace/HBase-0.92/trunk/target/surefire/surefire3949665940469355164tmp
 
/home/jenkins/jenkins-slave/workspace/HBase-0.92/trunk/target/surefire/surefire6167598571578719491tmp

 Script to find hanging test cases in build
 --

 Key: HBASE-4560
 URL: https://issues.apache.org/jira/browse/HBASE-4560
 Project: HBase
  Issue Type: Improvement
Reporter: ramkrishna.s.vasudevan
Priority: Minor
 Attachments: findHangingTest.sh, findHangingTest.sh


 A script that parses the console output to get the hanging test names. This 
 will be very useful to know the hanging test case names when we see in some 
 builds all the test cases run but still the build shows failed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-4565) Maven HBase build broken on cygwin with copynativelib.sh call.

2011-10-13 Thread Suraj Varma (Assigned) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suraj Varma reassigned HBASE-4565:
--

Assignee: Suraj Varma

 Maven HBase build broken on cygwin with copynativelib.sh call.
 --

 Key: HBASE-4565
 URL: https://issues.apache.org/jira/browse/HBASE-4565
 Project: HBase
  Issue Type: Bug
  Components: build
Affects Versions: 0.92.0
 Environment: cygwin (on xp and win7)
Reporter: Suraj Varma
Assignee: Suraj Varma
  Labels: build, maven
 Fix For: 0.92.0


 This is broken in both 0.92 as well as trunk pom.xml
 Here's a sample maven log snippet from trunk (from Mayuresh on user mailing 
 list)
 [INFO] [antrun:run {execution: package}]
 [INFO] Executing tasks
 main:
[mkdir] Created dir: 
 D:\workspace\mkshirsa\hbase-trunk\target\hbase-0.93-SNAPSHOT\hbase-0.93-SNAPSHOT\lib\native\${build.platform}
 [exec] ls: cannot access D:workspacemkshirsahbase-trunktarget/nativelib: 
 No such file or directory
 [exec] tar (child): Cannot connect to D: resolve failed
 [INFO] 
 
 [ERROR] BUILD ERROR
 [INFO] 
 
 [INFO] An Ant BuildException has occured: exec returned: 3328
 There are two issues: 
 1) The ant run task below doesn't resolve the windows file separator returned 
 by the project.build.directory - this causes the above resolve failed.
 !-- Using Unix cp to preserve symlinks, using script to handle wildcards --
 echo file=${project.build.directory}/copynativelibs.sh
 if [ `ls ${project.build.directory}/nativelib | wc -l` -ne 0]; then
 2) The tar argument value below also has a similar issue in that the path arg 
 doesn't resolve right.
 !-- Using Unix tar to preserve symlinks --
 exec executable=tar failonerror=yes 
 dir=${project.build.directory}/${project.artifactId}-${project.version}
 arg value=czf/
 arg 
 value=/cygdrive/c/workspaces/hbase-0.92-svn/target/${project.artifactId}-${project.version}.tar.gz/
 arg value=./
 /exec
 In both cases, the fix would probably be to use a cross-platform way to 
 handle the directory locations. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4565) Maven HBase build broken on cygwin with copynativelib.sh call.

2011-10-13 Thread Suraj Varma (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126722#comment-13126722
 ] 

Suraj Varma commented on HBASE-4565:


Working on one ... assigning this to myself.

 Maven HBase build broken on cygwin with copynativelib.sh call.
 --

 Key: HBASE-4565
 URL: https://issues.apache.org/jira/browse/HBASE-4565
 Project: HBase
  Issue Type: Bug
  Components: build
Affects Versions: 0.92.0
 Environment: cygwin (on xp and win7)
Reporter: Suraj Varma
Assignee: Suraj Varma
  Labels: build, maven
 Fix For: 0.92.0


 This is broken in both 0.92 as well as trunk pom.xml
 Here's a sample maven log snippet from trunk (from Mayuresh on user mailing 
 list)
 [INFO] [antrun:run {execution: package}]
 [INFO] Executing tasks
 main:
[mkdir] Created dir: 
 D:\workspace\mkshirsa\hbase-trunk\target\hbase-0.93-SNAPSHOT\hbase-0.93-SNAPSHOT\lib\native\${build.platform}
 [exec] ls: cannot access D:workspacemkshirsahbase-trunktarget/nativelib: 
 No such file or directory
 [exec] tar (child): Cannot connect to D: resolve failed
 [INFO] 
 
 [ERROR] BUILD ERROR
 [INFO] 
 
 [INFO] An Ant BuildException has occured: exec returned: 3328
 There are two issues: 
 1) The ant run task below doesn't resolve the windows file separator returned 
 by the project.build.directory - this causes the above resolve failed.
 !-- Using Unix cp to preserve symlinks, using script to handle wildcards --
 echo file=${project.build.directory}/copynativelibs.sh
 if [ `ls ${project.build.directory}/nativelib | wc -l` -ne 0]; then
 2) The tar argument value below also has a similar issue in that the path arg 
 doesn't resolve right.
 !-- Using Unix tar to preserve symlinks --
 exec executable=tar failonerror=yes 
 dir=${project.build.directory}/${project.artifactId}-${project.version}
 arg value=czf/
 arg 
 value=/cygdrive/c/workspaces/hbase-0.92-svn/target/${project.artifactId}-${project.version}.tar.gz/
 arg value=./
 /exec
 In both cases, the fix would probably be to use a cross-platform way to 
 handle the directory locations. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4586) ThriftServer doesn't expose autoflush/writebuffer settings needed for write-heavy loads

2011-10-13 Thread Ted Yu (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126742#comment-13126742
 ] 

Ted Yu commented on HBASE-4586:
---

According to Tim Sell:
This isn't trivial because we use HTablePool and it doesn't expose setting 
autoFlush on.

 ThriftServer doesn't expose autoflush/writebuffer settings needed for 
 write-heavy loads
 ---

 Key: HBASE-4586
 URL: https://issues.apache.org/jira/browse/HBASE-4586
 Project: HBase
  Issue Type: Bug
  Components: thrift
Affects Versions: 0.90.4
Reporter: Norbert Burger

 For write-heavy/bulk-load environments, two common recommendations are to 
 disable autoFlush and increase the size of the write buffer:
 HTable.setAutoFlush(false)
 HTable.setWriteBufferSize(long writeBufferSize)
 Neither of exposed via the Thrift API, which seems to preclude using the 
 Thrift interface for write-heavy environments.
 As a workaround, we could specify hbase.client.write.buffer in 
 hbase-site.xml, but there is unfortunately not an equivalent configuration 
 setting for autoflush.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4469) Avoid top row seek by looking up bloomfilter

2011-10-13 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126760#comment-13126760
 ] 

Liyin Tang commented on HBASE-4469:
---

@stack. HBASE-4469 optimizes the top row seek if the ROWCOL Bloom filter is 
enabled.
And HBASE-4532  will optimize the top row seek if ROW or NONE Bloom filter is 
enabled.
So HBASE-4469 + HBASE-4532 will optimize all the cases.
 
And it is necessary to commit this first:)


 Avoid top row seek by looking up bloomfilter
 

 Key: HBASE-4469
 URL: https://issues.apache.org/jira/browse/HBASE-4469
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang

 The problem is that when seeking for the row/col in the hfile, we will go to 
 top of the row in order to check for row delete marker (delete family). 
 However, if the bloomfilter is enabled for the column family, then if a 
 delete family operation is done on a row, the row is already being added to 
 bloomfilter. We can take advantage of this factor to avoid seeking to the top 
 of row.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3417) CacheOnWrite is using the temporary output path for block names, need to use a more consistent block naming scheme


[ 
https://issues.apache.org/jira/browse/HBASE-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126764#comment-13126764
 ] 

jirapos...@reviews.apache.org commented on HBASE-3417:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2379/#review2555
---



/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java
https://reviews.apache.org/r/2379/#comment5731

Can we keep these final and CacheConfig immutable, and create a separate 
instance of CacheConfig during testing instead?



/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java
https://reviews.apache.org/r/2379/#comment5732

The same as above.



/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java
https://reviews.apache.org/r/2379/#comment5736

Using the approach above we can avoid creating new setters.

If it's not feasible, then appending ForTesting to these method names 
will probably be enough to discourage use of them in production.



/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java
https://reviews.apache.org/r/2379/#comment5733

Maybe we can avoid these setters using the approach I described above? If 
not, it should either be package-private, or the name should include something 
like ForTesting.



/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java
https://reviews.apache.org/r/2379/#comment5734

The same as above.



/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
https://reviews.apache.org/r/2379/#comment5737

This does not return null anymore in the new version? This will probably 
result in an IOException down the line, is that handled well? We could throw an 
IOException right here as an alternative.



/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
https://reviews.apache.org/r/2379/#comment5738

@Michael: This pattern does not allow dashes in the UUID, it only allows 
0-9 and a-f.



/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
https://reviews.apache.org/r/2379/#comment5739

suffix.length() = 0 is unnecessary.



/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
https://reviews.apache.org/r/2379/#comment5740

Nice test! This goes all the way from client side to HFile, unlike 
TestCacheOnWrite.


- Mikhail


On 2011-10-13 06:31:13, Jonathan Gray wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2379/
bq.  ---
bq.  
bq.  (Updated 2011-10-13 06:31:13)
bq.  
bq.  
bq.  Review request for hbase, Dhruba Borthakur, Michael Stack, and Mikhail 
Bautin.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Adds a new test that fails w/o this fix and changes the naming scheme for 
storefiles to use UUID instead of random longs as ascii.
bq.  
bq.  The big change is that the name of the tmp file used when flushing and 
compacting is the same name (but different dir) when moved in place.  This 
makes it so block names are consistent for COW.
bq.  
bq.  
bq.  This addresses bug HBASE-3417.
bq.  https://issues.apache.org/jira/browse/HBASE-3417
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java 
1182679 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java 1182679 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1182679 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 
1182679 
bq./src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java 
1182679 
bq.  
bq.  Diff: https://reviews.apache.org/r/2379/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  TestFromClientSide and TestCacheOnWrite both working
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Jonathan
bq.  
bq.



 CacheOnWrite is using the temporary output path for block names, need to use 
 a more consistent block naming scheme
 --

 Key: HBASE-3417
 URL: https://issues.apache.org/jira/browse/HBASE-3417
 Project: HBase
  Issue Type: Bug
  Components: io, regionserver
Affects Versions: 0.92.0
Reporter: Jonathan Gray
Assignee: Jonathan Gray
Priority: Critical
 Fix For: 0.92.0

 Attachments: HBASE-3417-redux-v1.patch, HBASE-3417-v1.patch, 
 HBASE-3417-v2.patch, HBASE-3417-v5.patch


 Currently the block names used in the block cache are built using the 
 filesystem path.  However, for cache on write,

[jira] [Commented] (HBASE-3417) CacheOnWrite is using the temporary output path for block names, need to use a more consistent block naming scheme

2011-10-13 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126766#comment-13126766
]

jirapos...@reviews.apache.org commented on HBASE-3417:
--

bq. On 2011-10-13 17:58:45, Mikhail Bautin wrote:
bq.

Nice fix, Jonathan! For some reason I assumed that HBASE-3417 was part of the
HFile v2 patch when I submitted the patch, but apparently it was not.

- Mikhail

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2379/#review2555
---

CacheOnWrite is using the temporary output path for block names, need to use
a more consistent block naming scheme
--

Attachments: HBASE-3417-redux-v1.patch, HBASE-3417-v1.patch,
HBASE-3417-v2.patch, HBASE-3417-v5.patch

[jira] [Commented] (HBASE-4489) Better key splitting in RegionSplitter

2011-10-13 Thread Dave Revell (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126767#comment-13126767
]

Dave Revell commented on HBASE-4489:

nspiegelberg: going to change the upper bound for MD5StringSplit from 7FFF...
to ... OK with you?

Better key splitting in RegionSplitter
--

Key: HBASE-4489
URL: https://issues.apache.org/jira/browse/HBASE-4489
Project: HBase
Issue Type: Improvement
Affects Versions: 0.90.4
Reporter: Dave Revell
Assignee: Dave Revell
Attachments: HBASE-4489-branch0.90-v1.patch,
HBASE-4489-branch0.90-v2.patch, HBASE-4489-branch0.90-v3.patch,
HBASE-4489-trunk-v1.patch, HBASE-4489-trunk-v2.patch,
HBASE-4489-trunk-v3.patch

The RegionSplitter utility allows users to create a pre-split table from the
command line or do a rolling split on an existing table. It supports
pluggable split algorithms that implement the SplitAlgorithm interface. The
only/default SplitAlgorithm is one that assumes keys fall in the range from
ASCII string to ASCII string 7FFF. This is not a sane
default, and seems useless to most users. Users are likely to be surprised by
the fact that all the region splits occur in in the byte range of ASCII
characters.
A better default split algorithm would be one that evenly divides the space
of all bytes, which is what this patch does. Making a table with five regions
would split at \x33\x33..., \x66\x66, \x99\x99..., \xCC\xCC..., and
\xFF\xFF.

[jira] [Commented] (HBASE-4558) Refactor TestOpenedRegionHandler and TestOpenRegionHandler.

2011-10-13 Thread Lars Hofhansl (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126768#comment-13126768
 ] 

Lars Hofhansl commented on HBASE-4558:
--

I don't think the addendum needs to be mentioned in CHANGES.txt. A user looking 
at the set of changes won't care :)

 Refactor TestOpenedRegionHandler and TestOpenRegionHandler.
 ---

 Key: HBASE-4558
 URL: https://issues.apache.org/jira/browse/HBASE-4558
 Project: HBase
  Issue Type: Improvement
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Minor
 Fix For: 0.92.0

 Attachments: HBASE-4558_1.patch, HBASE-4558_2.patch, 
 HBASE-4558_3.patch


 This is an improvement task taken up to refactor TestOpenedRegionandler and 
 TestOpenRegionHandler so that MockServer and MockRegionServerServices can be 
 accessed from a common utility package.
 If we do this then one of the testcases in TestOpenedRegionHandler need not 
 start up a cluster and also moving it into a common package will help in 
 mocking the server for future testcases.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-4556) Fix all incorrect uses of InternalScanner.next(...)

2011-10-13 Thread Lars Hofhansl (Resolved) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl resolved HBASE-4556.
--

Resolution: Fixed

Committed to 0.92 and trunk.

 Fix all incorrect uses of InternalScanner.next(...)
 ---

 Key: HBASE-4556
 URL: https://issues.apache.org/jira/browse/HBASE-4556
 Project: HBase
  Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Attachments: 4556-v1.txt, 4556.txt


 There are cases all over the code where InternalScanner.next(...) is not used 
 correctly.
 I see this a lot:
 {code}
 while(scanner.next(...)) {
 }
 {code}
 The correct pattern is:
 {code}
 boolean more = false;
 do {
more = scanner.next(...);
 } while (more);
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4489) Better key splitting in RegionSplitter

2011-10-13 Thread Nicolas Spiegelberg (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126806#comment-13126806
 ] 

Nicolas Spiegelberg commented on HBASE-4489:


@Dave: go for it

 Better key splitting in RegionSplitter
 --

 Key: HBASE-4489
 URL: https://issues.apache.org/jira/browse/HBASE-4489
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.90.4
Reporter: Dave Revell
Assignee: Dave Revell
 Attachments: HBASE-4489-branch0.90-v1.patch, 
 HBASE-4489-branch0.90-v2.patch, HBASE-4489-branch0.90-v3.patch, 
 HBASE-4489-trunk-v1.patch, HBASE-4489-trunk-v2.patch, 
 HBASE-4489-trunk-v3.patch


 The RegionSplitter utility allows users to create a pre-split table from the 
 command line or do a rolling split on an existing table. It supports 
 pluggable split algorithms that implement the SplitAlgorithm interface. The 
 only/default SplitAlgorithm is one that assumes keys fall in the range from 
 ASCII string  to ASCII string 7FFF. This is not a sane 
 default, and seems useless to most users. Users are likely to be surprised by 
 the fact that all the region splits occur in in the byte range of ASCII 
 characters.
 A better default split algorithm would be one that evenly divides the space 
 of all bytes, which is what this patch does. Making a table with five regions 
 would split at \x33\x33..., \x66\x66, \x99\x99..., \xCC\xCC..., and 
 \xFF\xFF.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4065) TableOutputFormat ignores failure to create table instance

2011-10-13 Thread Brock Noland (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brock Noland updated HBASE-4065:


Resolution: Fixed
Status: Resolved  (was: Patch Available)

Marking issue resolve.

 TableOutputFormat ignores failure to create table instance
 --

 Key: HBASE-4065
 URL: https://issues.apache.org/jira/browse/HBASE-4065
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.3
Reporter: Todd Lipcon
Assignee: Brock Noland
 Fix For: 0.94.0

 Attachments: HBASE-4065.1.patch, HBASE-4065.2.patch


 If TableOutputFormat in the new API fails to create a table, it simply logs 
 this at ERROR level and then continues on its way. Then, the first write() to 
 the table will throw a NPE since table hasn't been set.
 Instead, it should probably rethrow the exception as a RuntimeException in 
 setConf, or do what the old-API TOF does and not create the HTable instance 
 until getRecordWriter, where it can throw an IOE.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3417) CacheOnWrite is using the temporary output path for block names, need to use a more consistent block naming scheme

2011-10-13 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126832#comment-13126832
]

jirapos...@reviews.apache.org commented on HBASE-3417:
--

bq. On 2011-10-13 14:49:35, Michael Stack wrote:
bq. /src/main/java/org/apache/hadoop/hbase/regionserver/Store.java, line 496
bq. https://reviews.apache.org/r/2379/diff/2/?file=49944#file49944line496
bq.
bq. This is the tmp name? Thats what you want? I don't see where it
gets set to renamed filename.

The tmp name and perm name are the same. The tmp is just in .tmp

bq. On 2011-10-13 14:49:35, Michael Stack wrote:
bq. /src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java, line
166
bq. https://reviews.apache.org/r/2379/diff/2/?file=49945#file49945line166
bq.
bq. Why? To be more specific?
bq.
bq. Oh, I see how about not removing the '-' from uuid? Is it
illegal in filesystem?

I am removing the '-'. This checks for alphanumerics only.

- Jonathan

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2379/#review2548
---

CacheOnWrite is using the temporary output path for block names, need to use
a more consistent block naming scheme
--

Attachments: HBASE-3417-redux-v1.patch, HBASE-3417-v1.patch,
HBASE-3417-v2.patch, HBASE-3417-v5.patch

[jira] [Commented] (HBASE-3417) CacheOnWrite is using the temporary output path for block names, need to use a more consistent block naming scheme

2011-10-13 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126834#comment-13126834
]

jirapos...@reviews.apache.org commented on HBASE-3417:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2379/#review2561
---

Thanks for the reviews stack and mikhail!

/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
https://reviews.apache.org/r/2379/#comment5744

I think i'll restore the old behavior of returning null, but if it gets an
IOE, that will be bubbled up (that should happen for a harder failure than just
a simple rename failure, in which case, we probably want to bubble it up?)

- Jonathan

CacheOnWrite is using the temporary output path for block names, need to use
a more consistent block naming scheme
--

Attachments: HBASE-3417-redux-v1.patch, HBASE-3417-v1.patch,
HBASE-3417-v2.patch, HBASE-3417-v5.patch

[jira] [Commented] (HBASE-3417) CacheOnWrite is using the temporary output path for block names, need to use a more consistent block naming scheme

2011-10-13 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126838#comment-13126838
]

jirapos...@reviews.apache.org commented on HBASE-3417:
--

bq. On 2011-10-13 19:24:46, Jonathan Gray wrote:
bq. /src/main/java/org/apache/hadoop/hbase/regionserver/Store.java, lines
1228-1229
bq. https://reviews.apache.org/r/2379/diff/2/?file=49944#file49944line1228
bq.
bq. I think i'll restore the old behavior of returning null, but if it
gets an IOE, that will be bubbled up (that should happen for a harder failure
than just a simple rename failure, in which case, we probably want to bubble it
up?)

Sounds good. By the way, is a null return value handled correctly upstream? The
whole compaction probably needs to be considered failed in that case.

- Mikhail

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2379/#review2561
---

CacheOnWrite is using the temporary output path for block names, need to use
a more consistent block naming scheme
--

Attachments: HBASE-3417-redux-v1.patch, HBASE-3417-v1.patch,
HBASE-3417-v2.patch, HBASE-3417-v5.patch

[jira] [Commented] (HBASE-3417) CacheOnWrite is using the temporary output path for block names, need to use a more consistent block naming scheme

2011-10-13 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126839#comment-13126839
]

jirapos...@reviews.apache.org commented on HBASE-3417:
--

Guess I should have looked at that. Seems that it is not handled.

I'm going to change the behavior to throw an IOException rather than return
null.

I will kick off the test suite before committing.

- Jonathan

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2379/#review2561
---

CacheOnWrite is using the temporary output path for block names, need to use
a more consistent block naming scheme
--

Attachments: HBASE-3417-redux-v1.patch, HBASE-3417-v1.patch,
HBASE-3417-v2.patch, HBASE-3417-v5.patch

[jira] [Commented] (HBASE-3417) CacheOnWrite is using the temporary output path for block names, need to use a more consistent block naming scheme

2011-10-13 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126842#comment-13126842
]

jirapos...@reviews.apache.org commented on HBASE-3417:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2379/#review2564
---

/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java
https://reviews.apache.org/r/2379/#comment5748

I like this suggestion... keeping config immutable and go make a new one if
you need to change something... w/ all final.

- Michael

CacheOnWrite is using the temporary output path for block names, need to use
a more consistent block naming scheme
--

Attachments: HBASE-3417-redux-v1.patch, HBASE-3417-v1.patch,
HBASE-3417-v2.patch, HBASE-3417-v5.patch

[jira] [Commented] (HBASE-3417) CacheOnWrite is using the temporary output path for block names, need to use a more consistent block naming scheme

2011-10-13 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126841#comment-13126841
]

jirapos...@reviews.apache.org commented on HBASE-3417:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2379/
---

(Updated 2011-10-13 19:33:02.273799)

Review request for hbase, Dhruba Borthakur, Michael Stack, and Mikhail Bautin.

Changes
---

Changes from Mikhail review. Changes it so if we fail rename we throw an
exception, otherwise it looks like we would eventually get an NPE down the
line? There's no special casing for a null return.

Will run test suite before committing.

Summary
---

Adds a new test that fails w/o this fix and changes the naming scheme for
storefiles to use UUID instead of random longs as ascii.

The big change is that the name of the tmp file used when flushing and
compacting is the same name (but different dir) when moved in place. This
makes it so block names are consistent for COW.

This addresses bug HBASE-3417.
https://issues.apache.org/jira/browse/HBASE-3417

Diffs (updated)
-

/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java
1183046
/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java 1183046
/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1183046
/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 1183046
/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java 1183046

Diff: https://reviews.apache.org/r/2379/diff

Testing
---

TestFromClientSide and TestCacheOnWrite both working

Thanks,

Jonathan

CacheOnWrite is using the temporary output path for block names, need to use
a more consistent block naming scheme
--

Attachments: HBASE-3417-redux-v1.patch, HBASE-3417-v1.patch,
HBASE-3417-v2.patch, HBASE-3417-v5.patch

[jira] [Commented] (HBASE-3417) CacheOnWrite is using the temporary output path for block names, need to use a more consistent block naming scheme

2011-10-13 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126847#comment-13126847
]

jirapos...@reviews.apache.org commented on HBASE-3417:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2379/#review2565
---

/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
https://reviews.apache.org/r/2379/#comment5749

@Mikhail yeah. I figured it out. I was thinking we should allow
dashes... then it looks like a uuid and makes compare by human easier.

- Michael

On 2011-10-13 19:33:02, Jonathan Gray wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2379/
bq. ---
bq.
bq. (Updated 2011-10-13 19:33:02)
bq.
bq.
bq. Review request for hbase, Dhruba Borthakur, Michael Stack, and Mikhail
Bautin.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Adds a new test that fails w/o this fix and changes the naming scheme for
storefiles to use UUID instead of random longs as ascii.
bq.
bq. The big change is that the name of the tmp file used when flushing and
compacting is the same name (but different dir) when moved in place. This
makes it so block names are consistent for COW.
bq.
bq.
bq. This addresses bug HBASE-3417.
bq. https://issues.apache.org/jira/browse/HBASE-3417
bq.
bq.
bq. Diffs
bq. -
bq.
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java
1183046
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java 1183046
bq./src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1183046
bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
1183046
bq./src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
1183046
bq.
bq. Diff: https://reviews.apache.org/r/2379/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. TestFromClientSide and TestCacheOnWrite both working
bq.
bq.
bq. Thanks,
bq.
bq. Jonathan
bq.
bq.

CacheOnWrite is using the temporary output path for block names, need to use
a more consistent block naming scheme
--

Attachments: HBASE-3417-redux-v1.patch, HBASE-3417-v1.patch,
HBASE-3417-v2.patch, HBASE-3417-v5.patch

[jira] [Commented] (HBASE-3417) CacheOnWrite is using the temporary output path for block names, need to use a more consistent block naming scheme

2011-10-13 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126848#comment-13126848
]

jirapos...@reviews.apache.org commented on HBASE-3417:
--

bq. On 2011-10-13 19:31:38, Michael Stack wrote:
bq. /src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java, line
100
bq. https://reviews.apache.org/r/2379/diff/2/?file=49943#file49943line100
bq.
bq. I like this suggestion... keeping config immutable and go make a new
one if you need to change something... w/ all final.

I like the idea too, but it's more clunky. Then CacheConfig can't be final in
things like Store and it becomes not very useful in other tests. I'm going to
make the test method names more explicit.

- Jonathan

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2379/#review2564
---

CacheOnWrite is using the temporary output path for block names, need to use
a more consistent block naming scheme
--

Attachments: HBASE-3417-redux-v1.patch, HBASE-3417-v1.patch,
HBASE-3417-v2.patch, HBASE-3417-v5.patch

[jira] [Commented] (HBASE-3417) CacheOnWrite is using the temporary output path for block names, need to use a more consistent block naming scheme

2011-10-13 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126851#comment-13126851
 ] 

jirapos...@reviews.apache.org commented on HBASE-3417:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2379/
---

(Updated 2011-10-13 19:36:33.845621)


Review request for hbase, Dhruba Borthakur, Michael Stack, and Mikhail Bautin.


Changes
---

Renamed setters to be prefixed with:  forTestsOnly_

(I'm still opting to keep it this way because I know that the next set of tests 
I write for the compressed cache stuff is going to want to do the same thing)


Summary
---

Adds a new test that fails w/o this fix and changes the naming scheme for 
storefiles to use UUID instead of random longs as ascii.

The big change is that the name of the tmp file used when flushing and 
compacting is the same name (but different dir) when moved in place.  This 
makes it so block names are consistent for COW.


This addresses bug HBASE-3417.
https://issues.apache.org/jira/browse/HBASE-3417


Diffs (updated)
-

  /src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java 
1183049 
  /src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java 1183049 
  /src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1183049 
  /src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 1183049 
  /src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java 1183049 

Diff: https://reviews.apache.org/r/2379/diff


Testing
---

TestFromClientSide and TestCacheOnWrite both working


Thanks,

Jonathan



 CacheOnWrite is using the temporary output path for block names, need to use 
 a more consistent block naming scheme
 --

 Key: HBASE-3417
 URL: https://issues.apache.org/jira/browse/HBASE-3417
 Project: HBase
  Issue Type: Bug
  Components: io, regionserver
Affects Versions: 0.92.0
Reporter: Jonathan Gray
Assignee: Jonathan Gray
Priority: Critical
 Fix For: 0.92.0

 Attachments: HBASE-3417-redux-v1.patch, HBASE-3417-v1.patch, 
 HBASE-3417-v2.patch, HBASE-3417-v5.patch


 Currently the block names used in the block cache are built using the 
 filesystem path.  However, for cache on write, the path is a temporary output 
 file.
 The original COW patch actually made some modifications to block naming stuff 
 to make it more consistent but did not do enough.  Should add a separate 
 method somewhere for generating block names using some more easily mocked 
 scheme (rather than just raw path as we generate a random unique file name 
 twice, once for tmp and then again when moved into place).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3417) CacheOnWrite is using the temporary output path for block names, need to use a more consistent block naming scheme

2011-10-13 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126853#comment-13126853
]

jirapos...@reviews.apache.org commented on HBASE-3417:
--

we had this argument months ago. there's a good consistency in the UIs w/ the
current region encoded names (used as dir names) w/ these uuid (minus -) names.
The dashes makes mismatched (as I recall my thoughts on the matter way back
when it originally had dashes).

i stand by the no dashes! :)

- Jonathan

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2379/#review2565
---

On 2011-10-13 19:36:33, Jonathan Gray wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2379/
bq. ---
bq.
bq. (Updated 2011-10-13 19:36:33)
bq.
bq.
bq. Review request for hbase, Dhruba Borthakur, Michael Stack, and Mikhail
Bautin.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Adds a new test that fails w/o this fix and changes the naming scheme for
storefiles to use UUID instead of random longs as ascii.
bq.
bq. The big change is that the name of the tmp file used when flushing and
compacting is the same name (but different dir) when moved in place. This
makes it so block names are consistent for COW.
bq.
bq.
bq. This addresses bug HBASE-3417.
bq. https://issues.apache.org/jira/browse/HBASE-3417
bq.
bq.
bq. Diffs
bq. -
bq.
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java
1183049
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java 1183049
bq./src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1183049
bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
1183049
bq./src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
1183049
bq.
bq. Diff: https://reviews.apache.org/r/2379/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. TestFromClientSide and TestCacheOnWrite both working
bq.
bq.
bq. Thanks,
bq.
bq. Jonathan
bq.
bq.

CacheOnWrite is using the temporary output path for block names, need to use
a more consistent block naming scheme
--

Attachments: HBASE-3417-redux-v1.patch, HBASE-3417-v1.patch,
HBASE-3417-v2.patch, HBASE-3417-v5.patch

[jira] [Commented] (HBASE-4459) HbaseObjectWritable code is a byte, we will eventually run out of codes

2011-10-13 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126854#comment-13126854
 ] 

Jonathan Gray commented on HBASE-4459:
--

- Why is Queue added within the scope of this JIRA?  Seems unrelated.

- Can you remove the unnecessary import re-org at the top?

- Can we have a unit test which shows the backwards compatibility of this?

Thanks for working on this Ram.

 HbaseObjectWritable code is a byte, we will eventually run out of codes
 ---

 Key: HBASE-4459
 URL: https://issues.apache.org/jira/browse/HBASE-4459
 Project: HBase
  Issue Type: Bug
  Components: io
Reporter: Jonathan Gray
Assignee: ramkrishna.s.vasudevan
Priority: Critical
 Fix For: 0.92.0

 Attachments: 4459.txt


 There are about 90 classes/codes in HbaseObjectWritable currently and 
 Byte.MAX_VALUE is 127.  In addition, anyone wanting to add custom classes but 
 not break compatibility might want to leave a gap before using codes and 
 that's difficult in such limited space.
 Eventually we should get rid of this pattern that makes compatibility 
 difficult (better client/server protocol handshake) but we should probably at 
 least bump this to a short for 0.94.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3417) CacheOnWrite is using the temporary output path for block names, need to use a more consistent block naming scheme

[
https://issues.apache.org/jira/browse/HBASE-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126856#comment-13126856
]

jirapos...@reviews.apache.org commented on HBASE-3417:
--

bq. On 2011-10-13 19:33:27, Michael Stack wrote:
bq. /src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java, line
166
bq. https://reviews.apache.org/r/2379/diff/2/?file=49945#file49945line166
bq.
bq. @Mikhail yeah. I figured it out. I was thinking we should
allow dashes... then it looks like a uuid and makes compare by human easier.
bq.
bq. Jonathan Gray wrote:
bq. we had this argument months ago. there's a good consistency in the
UIs w/ the current region encoded names (used as dir names) w/ these uuid
(minus -) names. The dashes makes mismatched (as I recall my thoughts on the
matter way back when it originally had dashes).
bq.
bq. i stand by the no dashes! :)

I took a look. The UUID ends up being the key in a LRU cache. I'm fine w/ it
(I argued for keeping them back then too... sorry for rehearsing old arg)

- Michael

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2379/#review2565
---

CacheOnWrite is using the temporary output path for block names, need to use
a more consistent block naming scheme
--

Attachments: HBASE-3417-redux-v1.patch, HBASE-3417-v1.patch,
HBASE-3417-v2.patch, HBASE-3417-v5.patch

[jira] [Commented] (HBASE-4459) HbaseObjectWritable code is a byte, we will eventually run out of codes

2011-10-13 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126864#comment-13126864
 ] 

stack commented on HBASE-4459:
--

What is this about?

{code}
+} else if (Queue.class.isAssignableFrom(declClass)) {
+  Queue queue = (Queue)instanceObj;
+  int length = queue.size();
+  out.writeInt(length);
+  Object obj;
+  for (int i = 0; i  length; i++) {
+obj = queue.remove();
+writeObject(out, obj, obj.getClass(), conf);
+  }
{code}

Is it necessary to this patch?

And this:

{code}
+} else if (Queue.class.isAssignableFrom(declaredClass)) {
+  int length = in.readInt();
+  instance = new ConcurrentLinkedQueue();
+  for (int i = 0; i  length; i++) {
+((Queue)instance).add(readObject(in, conf));
+  }
{code}

Looks good Ram.  Any chance of a test to prove it works the way it used to?

 HbaseObjectWritable code is a byte, we will eventually run out of codes
 ---

 Key: HBASE-4459
 URL: https://issues.apache.org/jira/browse/HBASE-4459
 Project: HBase
  Issue Type: Bug
  Components: io
Reporter: Jonathan Gray
Assignee: ramkrishna.s.vasudevan
Priority: Critical
 Fix For: 0.92.0

 Attachments: 4459.txt


 There are about 90 classes/codes in HbaseObjectWritable currently and 
 Byte.MAX_VALUE is 127.  In addition, anyone wanting to add custom classes but 
 not break compatibility might want to leave a gap before using codes and 
 that's difficult in such limited space.
 Eventually we should get rid of this pattern that makes compatibility 
 difficult (better client/server protocol handshake) but we should probably at 
 least bump this to a short for 0.94.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4556) Fix all incorrect uses of InternalScanner.next(...)

2011-10-13 Thread Hudson (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126873#comment-13126873
 ] 

Hudson commented on HBASE-4556:
---

Integrated in HBase-0.92 #61 (See 
[https://builds.apache.org/job/HBase-0.92/61/])
HBASE-4556 Fix all incorrect uses of InternalScanner.next(...)

larsh : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/HMerge.java


 Fix all incorrect uses of InternalScanner.next(...)
 ---

 Key: HBASE-4556
 URL: https://issues.apache.org/jira/browse/HBASE-4556
 Project: HBase
  Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Attachments: 4556-v1.txt, 4556.txt


 There are cases all over the code where InternalScanner.next(...) is not used 
 correctly.
 I see this a lot:
 {code}
 while(scanner.next(...)) {
 }
 {code}
 The correct pattern is:
 {code}
 boolean more = false;
 do {
more = scanner.next(...);
 } while (more);
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4587) support providing multiple tables to an TableMapper derivatives

2011-10-13 Thread Rajeev Rao (Created) (JIRA)

support providing multiple tables to an TableMapper derivatives
---

 Key: HBASE-4587
 URL: https://issues.apache.org/jira/browse/HBASE-4587
 Project: HBase
  Issue Type: Bug
  Components: mapreduce
Affects Versions: 0.90.3
Reporter: Rajeev Rao


We need a way for hbase to support providing multiple tables to the Mapper (in 
the same way we can provide globs of hdfs files). 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4587) HBase MR needs to support multiple tables as input

2011-10-13 Thread Alex Newman (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Newman updated HBASE-4587:
---

Description: 
Some requirements:
 - Being able to have multiple tables as your input path
 - Being able to filter on specific columns/column families
 - Providing the source location (table/row/column) to the results


  was:We need a way for hbase to support providing multiple tables to the 
Mapper (in the same way we can provide globs of hdfs files). 

   Assignee: Alex Newman
Summary: HBase MR needs to support multiple tables as input  (was: 
support providing multiple tables to an TableMapper derivatives)

 HBase MR needs to support multiple tables as input
 --

 Key: HBASE-4587
 URL: https://issues.apache.org/jira/browse/HBASE-4587
 Project: HBase
  Issue Type: Bug
  Components: mapreduce
Affects Versions: 0.90.3
Reporter: Rajeev Rao
Assignee: Alex Newman

 Some requirements:
  - Being able to have multiple tables as your input path
  - Being able to filter on specific columns/column families
  - Providing the source location (table/row/column) to the results

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-4078) Silent Data Offlining During HDFS Flakiness

2011-10-13 Thread Nicolas Spiegelberg (Resolved) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nicolas Spiegelberg resolved HBASE-4078.


   Resolution: Fixed
Fix Version/s: 0.94.0
   0.92.0

added to 89, 92,  94

 Silent Data Offlining During HDFS Flakiness
 ---

 Key: HBASE-4078
 URL: https://issues.apache.org/jira/browse/HBASE-4078
 Project: HBase
  Issue Type: Bug
  Components: io, regionserver
Affects Versions: 0.89.20100924, 0.90.3, 0.92.0
Reporter: Nicolas Spiegelberg
Assignee: Pritam Damania
Priority: Blocker
 Fix For: 0.92.0, 0.94.0

 Attachments: 
 0001-Validate-store-files-after-compactions-flushes.patch, 
 0001-Validate-store-files.patch


 See HBASE-1436 .  The bug fix for this JIRA is a temporary workaround for 
 improperly moving partially-written files from TMP into the region directory 
 when a FS error occurs.  Unfortunately, the fix is to ignore all IO 
 exceptions, which masks off-lining due to FS flakiness.  We need to 
 permanently fix the problem that created HBASE-1436  then at least have the 
 option to not open a region during times of flakey FS.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4585) Avoid seek operation when current kv is deleted


[ 
https://issues.apache.org/jira/browse/HBASE-4585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126901#comment-13126901
 ] 

jirapos...@reviews.apache.org commented on HBASE-4585:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2384/
---

Review request for hbase, Michael Stack, Jonathan Gray, Mikhail Bautin, Kannan 
Muthukkaruppan, and Nicolas Spiegelberg.


Summary
---

When the current kv is deleted during the matching in the ScanQueryMatcher, 
currently the matcher will return skip and continue to seek.
Actually, if the current kv is deleted because of family deleted or column 
deleted, the matcher should seek to next col.
If the current kv is deleted because of version deleted, the matcher should 
just return skip.

I will submit another patch against apache-trunk later.


This addresses bug HBASE-4585.
https://issues.apache.org/jira/browse/HBASE-4585


Diffs
-

  src/main/java/org/apache/hadoop/hbase/regionserver/DeleteTracker.java b425bf2 
  src/main/java/org/apache/hadoop/hbase/regionserver/ScanDeleteTracker.java 
1a891a6 
  src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 
a1d7de5 
  src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 
e8ac027 
  src/test/java/org/apache/hadoop/hbase/regionserver/TestScanDeleteTracker.java 
afb3fcc 

Diff: https://reviews.apache.org/r/2384/diff


Testing
---

Running all the unit tests.


Thanks,

Liyin



 Avoid seek operation when current kv is deleted
 ---

 Key: HBASE-4585
 URL: https://issues.apache.org/jira/browse/HBASE-4585
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang

 When the current kv is deleted during the matching in the ScanQueryMatcher, 
 currently the matcher will return skip and continue to seek.
 Actually, if the current kv is deleted because of family deleted or column 
 deleted, the matcher should seek to next col.
 If the current kv is deleted because of version deleted, the matcher should 
 just return skip.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4585) Avoid seek operation when current kv is deleted

2011-10-13 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126911#comment-13126911
 ] 

jirapos...@reviews.apache.org commented on HBASE-4585:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2384/
---

(Updated 2011-10-13 20:53:40.516803)


Review request for hbase, Michael Stack, Jonathan Gray, Mikhail Bautin, Kannan 
Muthukkaruppan, and Nicolas Spiegelberg.


Changes
---

update the summary, so we can easily filter out in the email.


Summary (updated)
---

When the current kv is deleted during the matching in the ScanQueryMatcher, 
currently the matcher will return skip and continue to seek.
Actually, if the current kv is deleted because of family deleted or column 
deleted, the matcher should seek to next col.
If the current kv is deleted because of version deleted, the matcher should 
just return skip.

I will submit another patch against apache-trunk later.


This addresses bug HBASE-4585.
https://issues.apache.org/jira/browse/HBASE-4585


Diffs
-

  src/main/java/org/apache/hadoop/hbase/regionserver/DeleteTracker.java b425bf2 
  src/main/java/org/apache/hadoop/hbase/regionserver/ScanDeleteTracker.java 
1a891a6 
  src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 
a1d7de5 
  src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 
e8ac027 
  src/test/java/org/apache/hadoop/hbase/regionserver/TestScanDeleteTracker.java 
afb3fcc 

Diff: https://reviews.apache.org/r/2384/diff


Testing
---

Running all the unit tests.


Thanks,

Liyin



 Avoid seek operation when current kv is deleted
 ---

 Key: HBASE-4585
 URL: https://issues.apache.org/jira/browse/HBASE-4585
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang

 When the current kv is deleted during the matching in the ScanQueryMatcher, 
 currently the matcher will return skip and continue to seek.
 Actually, if the current kv is deleted because of family deleted or column 
 deleted, the matcher should seek to next col.
 If the current kv is deleted because of version deleted, the matcher should 
 just return skip.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4585) Avoid seek operation when current kv is deleted

2011-10-13 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126912#comment-13126912
 ] 

jirapos...@reviews.apache.org commented on HBASE-4585:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2384/#review2569
---


nice work.  lovin that test blocks read :)


src/main/java/org/apache/hadoop/hbase/regionserver/DeleteTracker.java
https://reviews.apache.org/r/2384/#comment5753

whitespace.  and maybe add a small comment about what this enum is being 
used for (and where it is used)



src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
https://reviews.apache.org/r/2384/#comment5754

whitespace all through here, but this looks really nice!


- Jonathan


On 2011-10-13 20:53:40, Liyin wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2384/
bq.  ---
bq.  
bq.  (Updated 2011-10-13 20:53:40)
bq.  
bq.  
bq.  Review request for hbase, Michael Stack, Jonathan Gray, Mikhail Bautin, 
Kannan Muthukkaruppan, and Nicolas Spiegelberg.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  When the current kv is deleted during the matching in the 
ScanQueryMatcher, currently the matcher will return skip and continue to seek.
bq.  Actually, if the current kv is deleted because of family deleted or column 
deleted, the matcher should seek to next col.
bq.  If the current kv is deleted because of version deleted, the matcher 
should just return skip.
bq.  
bq.  I will submit another patch against apache-trunk later.
bq.  
bq.  
bq.  This addresses bug HBASE-4585.
bq.  https://issues.apache.org/jira/browse/HBASE-4585
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/regionserver/DeleteTracker.java 
b425bf2 
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/ScanDeleteTracker.java 
1a891a6 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 
a1d7de5 
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 
e8ac027 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanDeleteTracker.java 
afb3fcc 
bq.  
bq.  Diff: https://reviews.apache.org/r/2384/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Running all the unit tests.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Liyin
bq.  
bq.



 Avoid seek operation when current kv is deleted
 ---

 Key: HBASE-4585
 URL: https://issues.apache.org/jira/browse/HBASE-4585
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang

 When the current kv is deleted during the matching in the ScanQueryMatcher, 
 currently the matcher will return skip and continue to seek.
 Actually, if the current kv is deleted because of family deleted or column 
 deleted, the matcher should seek to next col.
 If the current kv is deleted because of version deleted, the matcher should 
 just return skip.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4489) Better key splitting in RegionSplitter

2011-10-13 Thread Dave Revell (Updated) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Dave Revell updated HBASE-4489:
---

Attachment: HBASE-4489-trunk-v4.patch

New patch for trunk with last night's feedback.
- The SplitAlgorithm to use is now a required parameter
- MD5StringSplit is now called HexStringSplit
- The ceiling of 7FFF is now , and tests were changed to
accomodate this.

No changes are necessary on the 0.90 branch, so the -v3 patch for 0.90 is still
current.

Better key splitting in RegionSplitter
--

[jira] [Commented] (HBASE-4382) Region encoded name is hash of tablename + start key + regionid (timestamp); should include end key when hashing.

2011-10-13 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126921#comment-13126921
 ] 

Alex Newman commented on HBASE-4382:


Looks done to me 

 HRegionIinfo.java:
private void setHashCode() {
int result = Arrays.hashCode(this.regionName);
result ^= this.regionId;
result ^= Arrays.hashCode(this.startKey);
result ^= Arrays.hashCode(this.endKey);
result ^= Boolean.valueOf(this.offLine).hashCode();
result ^= Arrays.hashCode(this.tableName);
this.hashCode = result;
  }

 Region encoded name is hash of tablename + start key + regionid (timestamp); 
 should include end key when hashing.
 -

 Key: HBASE-4382
 URL: https://issues.apache.org/jira/browse/HBASE-4382
 Project: HBase
  Issue Type: Improvement
Reporter: stack
  Labels: noob

 Seems odd that region encoded name is same for regions if made in same second 
 with same start key tough their end keys are different.  It can happen in 
 unit test.  Should mix in the end key when coming up w/ the region name 
 encoded name.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4585) Avoid seek operation when current kv is deleted


[ 
https://issues.apache.org/jira/browse/HBASE-4585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126924#comment-13126924
 ] 

jirapos...@reviews.apache.org commented on HBASE-4585:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2384/
---

(Updated 2011-10-13 21:09:08.566988)


Review request for hbase, Michael Stack, Jonathan Gray, Mikhail Bautin, Kannan 
Muthukkaruppan, and Nicolas Spiegelberg.


Changes
---

Address Jonathan's comments.


Summary
---

When the current kv is deleted during the matching in the ScanQueryMatcher, 
currently the matcher will return skip and continue to seek.
Actually, if the current kv is deleted because of family deleted or column 
deleted, the matcher should seek to next col.
If the current kv is deleted because of version deleted, the matcher should 
just return skip.

I will submit another patch against apache-trunk later.


This addresses bug HBASE-4585.
https://issues.apache.org/jira/browse/HBASE-4585


Diffs (updated)
-

  src/main/java/org/apache/hadoop/hbase/regionserver/DeleteTracker.java b425bf2 
  src/main/java/org/apache/hadoop/hbase/regionserver/ScanDeleteTracker.java 
1a891a6 
  src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 
a1d7de5 
  src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 
e8ac027 
  src/test/java/org/apache/hadoop/hbase/regionserver/TestScanDeleteTracker.java 
afb3fcc 

Diff: https://reviews.apache.org/r/2384/diff


Testing
---

Running all the unit tests.


Thanks,

Liyin



 Avoid seek operation when current kv is deleted
 ---

 Key: HBASE-4585
 URL: https://issues.apache.org/jira/browse/HBASE-4585
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang

 When the current kv is deleted during the matching in the ScanQueryMatcher, 
 currently the matcher will return skip and continue to seek.
 Actually, if the current kv is deleted because of family deleted or column 
 deleted, the matcher should seek to next col.
 If the current kv is deleted because of version deleted, the matcher should 
 just return skip.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4382) Region encoded name is hash of tablename + start key + regionid (timestamp); should include end key when hashing.

2011-10-13 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126923#comment-13126923
 ] 

Alex Newman commented on HBASE-4382:


Is there anything else we need to do here?

 Region encoded name is hash of tablename + start key + regionid (timestamp); 
 should include end key when hashing.
 -

 Key: HBASE-4382
 URL: https://issues.apache.org/jira/browse/HBASE-4382
 Project: HBase
  Issue Type: Improvement
Reporter: stack
Assignee: Alex Newman
  Labels: noob

 Seems odd that region encoded name is same for regions if made in same second 
 with same start key tough their end keys are different.  It can happen in 
 unit test.  Should mix in the end key when coming up w/ the region name 
 encoded name.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-4382) Region encoded name is hash of tablename + start key + regionid (timestamp); should include end key when hashing.

2011-10-13 Thread Alex Newman (Assigned) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Newman reassigned HBASE-4382:
--

Assignee: Alex Newman

 Region encoded name is hash of tablename + start key + regionid (timestamp); 
 should include end key when hashing.
 -

 Key: HBASE-4382
 URL: https://issues.apache.org/jira/browse/HBASE-4382
 Project: HBase
  Issue Type: Improvement
Reporter: stack
Assignee: Alex Newman
  Labels: noob

 Seems odd that region encoded name is same for regions if made in same second 
 with same start key tough their end keys are different.  It can happen in 
 unit test.  Should mix in the end key when coming up w/ the region name 
 encoded name.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4585) Avoid seek operation when current kv is deleted


[ 
https://issues.apache.org/jira/browse/HBASE-4585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126931#comment-13126931
 ] 

jirapos...@reviews.apache.org commented on HBASE-4585:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2384/#review2570
---



src/main/java/org/apache/hadoop/hbase/regionserver/DeleteTracker.java
https://reviews.apache.org/r/2384/#comment5755

deleteResult sounds like a variable name and not a description of what is 
returned.



src/main/java/org/apache/hadoop/hbase/regionserver/DeleteTracker.java
https://reviews.apache.org/r/2384/#comment5758

Add some javadoc for this enum.



src/main/java/org/apache/hadoop/hbase/regionserver/DeleteTracker.java
https://reviews.apache.org/r/2384/#comment5757

Does this mean that a particular timestamp has been deleted? What about 
cases when everything up to a specific timestamp has been deleted?



src/main/java/org/apache/hadoop/hbase/regionserver/ScanDeleteTracker.java
https://reviews.apache.org/r/2384/#comment5763

This should probably be updated to reflect the new more granular return 
value.



src/main/java/org/apache/hadoop/hbase/regionserver/ScanDeleteTracker.java
https://reviews.apache.org/r/2384/#comment5756

The same as above about the @return javadoc.



src/main/java/org/apache/hadoop/hbase/regionserver/ScanDeleteTracker.java
https://reviews.apache.org/r/2384/#comment5764

The function should probably be renamed to reflect the more granular return 
value.



src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
https://reviews.apache.org/r/2384/#comment5760

nit [coding style]: add a space between switch and (



src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
https://reviews.apache.org/r/2384/#comment5759

A better error message would include the received result.



src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java
https://reviews.apache.org/r/2384/#comment5761

Remove the period and put HBASE-4585 in parentheses for clarity.



src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java
https://reviews.apache.org/r/2384/#comment5762

Remove the period and put HBASE-4585 in parentheses for clarity.


- Mikhail


On 2011-10-13 21:09:08, Liyin wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2384/
bq.  ---
bq.  
bq.  (Updated 2011-10-13 21:09:08)
bq.  
bq.  
bq.  Review request for hbase, Michael Stack, Jonathan Gray, Mikhail Bautin, 
Kannan Muthukkaruppan, and Nicolas Spiegelberg.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  When the current kv is deleted during the matching in the 
ScanQueryMatcher, currently the matcher will return skip and continue to seek.
bq.  Actually, if the current kv is deleted because of family deleted or column 
deleted, the matcher should seek to next col.
bq.  If the current kv is deleted because of version deleted, the matcher 
should just return skip.
bq.  
bq.  I will submit another patch against apache-trunk later.
bq.  
bq.  
bq.  This addresses bug HBASE-4585.
bq.  https://issues.apache.org/jira/browse/HBASE-4585
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/regionserver/DeleteTracker.java 
b425bf2 
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/ScanDeleteTracker.java 
1a891a6 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 
a1d7de5 
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 
e8ac027 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanDeleteTracker.java 
afb3fcc 
bq.  
bq.  Diff: https://reviews.apache.org/r/2384/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Running all the unit tests.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Liyin
bq.  
bq.



 Avoid seek operation when current kv is deleted
 ---

 Key: HBASE-4585
 URL: https://issues.apache.org/jira/browse/HBASE-4585
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang

 When the current kv is deleted during the matching in the ScanQueryMatcher, 
 currently the matcher will return skip and continue to seek.
 Actually, if the current kv is deleted because of family deleted or column 
 deleted, the matcher should seek to next col.
 If the current kv is deleted because of version deleted, the matcher should 
 just return skip.

--
This message is automatically generated by JIRA.
If you

[jira] [Updated] (HBASE-4587) HBase MR support for multiple tables as input

2011-10-13 Thread Alex Newman (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Newman updated HBASE-4587:
---

Description: 
Some requirements:
 - Being able to have multiple tables as your input path
 - Being able to filter on specific columns/column families
 - Providing the source location (table/row/column) to the results
 - Multiple clusters
 - Different schemas.

Overall this seems difficult for now so I am going to punt on it. On the other 
hand it would be easy enough to write all of the MR values into an intermediate 
table and then work from there.

  was:
Some requirements:
 - Being able to have multiple tables as your input path
 - Being able to filter on specific columns/column families
 - Providing the source location (table/row/column) to the results


Summary: HBase MR support for multiple tables as input  (was: HBase MR 
needs to support multiple tables as input)

 HBase MR support for multiple tables as input
 -

 Key: HBASE-4587
 URL: https://issues.apache.org/jira/browse/HBASE-4587
 Project: HBase
  Issue Type: Bug
  Components: mapreduce
Affects Versions: 0.90.3
Reporter: Rajeev Rao
Assignee: Alex Newman

 Some requirements:
  - Being able to have multiple tables as your input path
  - Being able to filter on specific columns/column families
  - Providing the source location (table/row/column) to the results
  - Multiple clusters
  - Different schemas.
 Overall this seems difficult for now so I am going to punt on it. On the 
 other hand it would be easy enough to write all of the MR values into an 
 intermediate table and then work from there.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-1872) Dirty, fast, kill table script

2011-10-13 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-1872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126935#comment-13126935
 ] 

Alex Newman commented on HBASE-1872:


I am pretty sure that wont work
if not rowidStr.startsWith(tableNameStr)

will delete all tables which start with the same prefix no?

 Dirty, fast, kill table script
 --

 Key: HBASE-1872
 URL: https://issues.apache.org/jira/browse/HBASE-1872
 Project: HBase
  Issue Type: Bug
Reporter: stack
  Labels: noob
 Attachments: kill_table.rb


 Some fellas embedding hbase want to be able to kill tables quickly between 
 tests; they don't want to have to wait on enable/disable stuff.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4585) Avoid seek operation when current kv is deleted


[ 
https://issues.apache.org/jira/browse/HBASE-4585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126961#comment-13126961
 ] 

jirapos...@reviews.apache.org commented on HBASE-4585:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2384/
---

(Updated 2011-10-13 21:33:17.853931)


Review request for hbase, Michael Stack, Jonathan Gray, Mikhail Bautin, Kannan 
Muthukkaruppan, and Nicolas Spiegelberg.


Changes
---

Address Mikhail's comments.


Summary
---

When the current kv is deleted during the matching in the ScanQueryMatcher, 
currently the matcher will return skip and continue to seek.
Actually, if the current kv is deleted because of family deleted or column 
deleted, the matcher should seek to next col.
If the current kv is deleted because of version deleted, the matcher should 
just return skip.

I will submit another patch against apache-trunk later.


This addresses bug HBASE-4585.
https://issues.apache.org/jira/browse/HBASE-4585


Diffs (updated)
-

  src/test/java/org/apache/hadoop/hbase/regionserver/TestScanDeleteTracker.java 
afb3fcc 
  src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 
e8ac027 
  src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 
a1d7de5 
  src/main/java/org/apache/hadoop/hbase/regionserver/DeleteTracker.java b425bf2 
  src/main/java/org/apache/hadoop/hbase/regionserver/ScanDeleteTracker.java 
1a891a6 

Diff: https://reviews.apache.org/r/2384/diff


Testing
---

Running all the unit tests.


Thanks,

Liyin



 Avoid seek operation when current kv is deleted
 ---

 Key: HBASE-4585
 URL: https://issues.apache.org/jira/browse/HBASE-4585
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang

 When the current kv is deleted during the matching in the ScanQueryMatcher, 
 currently the matcher will return skip and continue to seek.
 Actually, if the current kv is deleted because of family deleted or column 
 deleted, the matcher should seek to next col.
 If the current kv is deleted because of version deleted, the matcher should 
 just return skip.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4382) Region encoded name is hash of tablename + start key + regionid (timestamp); should include end key when hashing.


[ 
https://issues.apache.org/jira/browse/HBASE-4382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126967#comment-13126967
 ] 

stack commented on HBASE-4382:
--

I think you are looking hashcode vs encoded name.  See 
http://hbase.apache.org/xref/org/apache/hadoop/hbase/HRegionInfo.html#357

 Region encoded name is hash of tablename + start key + regionid (timestamp); 
 should include end key when hashing.
 -

 Key: HBASE-4382
 URL: https://issues.apache.org/jira/browse/HBASE-4382
 Project: HBase
  Issue Type: Improvement
Reporter: stack
Assignee: Alex Newman
  Labels: noob

 Seems odd that region encoded name is same for regions if made in same second 
 with same start key tough their end keys are different.  It can happen in 
 unit test.  Should mix in the end key when coming up w/ the region name 
 encoded name.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4382) Region encoded name is hash of tablename + start key + regionid (timestamp); should include end key when hashing.


[ 
https://issues.apache.org/jira/browse/HBASE-4382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126968#comment-13126968
 ] 

Alex Newman commented on HBASE-4382:


*nod

 Region encoded name is hash of tablename + start key + regionid (timestamp); 
 should include end key when hashing.
 -

 Key: HBASE-4382
 URL: https://issues.apache.org/jira/browse/HBASE-4382
 Project: HBase
  Issue Type: Improvement
Reporter: stack
Assignee: Alex Newman
  Labels: noob

 Seems odd that region encoded name is same for regions if made in same second 
 with same start key tough their end keys are different.  It can happen in 
 unit test.  Should mix in the end key when coming up w/ the region name 
 encoded name.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4418) Show all the hbase configuration in the web ui


[ 
https://issues.apache.org/jira/browse/HBASE-4418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13126979#comment-13126979
 ] 

Liyin Tang commented on HBASE-4418:
---

@stack, it is pretty safte to commit HBASE-4418_1.patch :)

 Show all the hbase configuration in the web ui
 --

 Key: HBASE-4418
 URL: https://issues.apache.org/jira/browse/HBASE-4418
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang
 Attachments: HBASE-4418_1.patch, HBASE-4418_2.patch


 The motivation is to show ALL the HBase configuration, which takes effect in 
 the run time, in a global place.
 So we can easily know which configuration takes effect and what the value is.
 The configuration shows all the HBase and DFS configuration entry in the 
 configuration file and also includes all the HBase default setting in the 
 code, which is not the config file.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3417) CacheOnWrite is using the temporary output path for block names, need to use a more consistent block naming scheme

2011-10-13 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127018#comment-13127018
]

Jonathan Gray commented on HBASE-3417:
--

Stack, I was going to open a new JIRA, but it is the exact same issue and a
nearly identical patch (primary difference is pre/post hfile v2). It was just
incorrect to close this following commit of hfile v2 which was unrelated to
this bug. Nothing was ever committed under this JIRA so just reopened with an
updated patch.

I think things get confusing when there is more than one commit per branch per
jira. We should probably ban that practice. Or at least institute some kind
of standardized commit message (HBASE-3417, HBASE-3417-B, HBASE-3417-C, etc) or
some such thing.

CacheOnWrite is using the temporary output path for block names, need to use
a more consistent block naming scheme
--

Attachments: HBASE-3417-redux-v1.patch, HBASE-3417-v1.patch,
HBASE-3417-v2.patch, HBASE-3417-v5.patch

[jira] [Commented] (HBASE-4585) Avoid seek operation when current kv is deleted


[ 
https://issues.apache.org/jira/browse/HBASE-4585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127032#comment-13127032
 ] 

jirapos...@reviews.apache.org commented on HBASE-4585:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2384/
---

(Updated 2011-10-13 22:18:51.653430)


Review request for hbase, Dhruba Borthakur, Michael Stack, Jonathan Gray, 
Mikhail Bautin, Kannan Muthukkaruppan, Jerry Chen, Karthik Ranganathan, and 
Nicolas Spiegelberg.


Summary
---

When the current kv is deleted during the matching in the ScanQueryMatcher, 
currently the matcher will return skip and continue to seek.
Actually, if the current kv is deleted because of family deleted or column 
deleted, the matcher should seek to next col.
If the current kv is deleted because of version deleted, the matcher should 
just return skip.

I will submit another patch against apache-trunk later.


This addresses bug HBASE-4585.
https://issues.apache.org/jira/browse/HBASE-4585


Diffs
-

  src/test/java/org/apache/hadoop/hbase/regionserver/TestScanDeleteTracker.java 
afb3fcc 
  src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 
e8ac027 
  src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 
a1d7de5 
  src/main/java/org/apache/hadoop/hbase/regionserver/DeleteTracker.java b425bf2 
  src/main/java/org/apache/hadoop/hbase/regionserver/ScanDeleteTracker.java 
1a891a6 

Diff: https://reviews.apache.org/r/2384/diff


Testing
---

Running all the unit tests.


Thanks,

Liyin



 Avoid seek operation when current kv is deleted
 ---

 Key: HBASE-4585
 URL: https://issues.apache.org/jira/browse/HBASE-4585
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang

 When the current kv is deleted during the matching in the ScanQueryMatcher, 
 currently the matcher will return skip and continue to seek.
 Actually, if the current kv is deleted because of family deleted or column 
 deleted, the matcher should seek to next col.
 If the current kv is deleted because of version deleted, the matcher should 
 just return skip.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4568) Make zk dump jsp response more quickly

2011-10-13 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127048#comment-13127048
 ] 

jirapos...@reviews.apache.org commented on HBASE-4568:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2385/
---

Review request for hbase, Dhruba Borthakur, Michael Stack, Jonathan Gray, 
Mikhail Bautin, Pritam Damania, Prakash Khemani, Amitanand Aiyer, Kannan 
Muthukkaruppan, Jerry Chen, Karthik Ranganathan, and Nicolas Spiegelberg.


Summary
---

1) For each zk dump, currently hbase will create a zk client instance every 
time. 
This is quite slow when any machines in the quorum is dead. Because it will 
connect to each machine in the zk quorum again.

code
HMaster master = (HMaster)getServletContext().getAttribute(HMaster.MASTER);
Configuration conf = master.getConfiguration();
HBaseAdmin hbadmin = new HBaseAdmin(conf);
HConnection connection = hbadmin.getConnection();
ZooKeeperWatcher watcher = connection.getZooKeeperWatcher();
/code

So we can simplify this:
code
HMaster master = (HMaster)getServletContext().getAttribute(HMaster.MASTER);
ZooKeeperWatcher watcher = master.getZooKeeperWatcher();
/code

2) Also when hbase call getServerStats() for each machine in the zk quorum, it 
hard coded the default time out as 1 min. 
It would be nice to make this configurable and set it to a low time out.

When hbase tries to connect to each machine in the zk quorum, it will create 
the socket, and then set the socket time out, and read it with this time out.
It means hbase will create a socket and connect to the zk server with 0 time 
out at first, which will take a long time. 
Because a timeout of zero is interpreted as an infinite timeout. The connection 
will then block until established or an error occurs.

3) The recoverable zookeeper should be real exponentially backoff when there is 
connection loss exception, which will give hbase much longer time window to 
recover from zk machine failures.


This addresses bug HBASE-4568.
https://issues.apache.org/jira/browse/HBASE-4568


Diffs
-

  src/main/java/org/apache/hadoop/hbase/util/RetryCounter.java 61ea552 
  src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java 
b8c4f61 
  src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 699a5f5 
  src/main/resources/hbase-webapps/master/zk.jsp b31d94c 

Diff: https://reviews.apache.org/r/2385/diff


Testing
---

Running all the unit tests


Thanks,

Liyin



 Make zk dump jsp response more quickly
 --

 Key: HBASE-4568
 URL: https://issues.apache.org/jira/browse/HBASE-4568
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang

 For each zk dump, currently hbase will create a zk client instance every 
 time. 
 This is quite slow when any machines in the quorum is dead. Because it will 
 connect to each machine in the zk quorum again.
   HMaster master = (HMaster)getServletContext().getAttribute(HMaster.MASTER);
   Configuration conf = master.getConfiguration();
   HBaseAdmin hbadmin = new HBaseAdmin(conf);
   HConnection connection = hbadmin.getConnection();
   ZooKeeperWatcher watcher = connection.getZooKeeperWatcher();
 So we can simplify this:
   HMaster master = (HMaster)getServletContext().getAttribute(HMaster.MASTER);
   ZooKeeperWatcher watcher = master.getZooKeeperWatcher();
 Also when hbase call getServerStats() for each machine in the zk quorum, it 
 hard coded the default time out as 1 min. 
 It would be nice to make this configurable and set it to a low time out.
 When hbase tries to connect to each machine in the zk quorum, it will create 
 the socket, and then set the socket time out, and read it with this time out.
 It means hbase will create a socket and connect to the zk server with 0 time 
 out at first, which will take a long time. 
 Because a timeout of zero is interpreted as an infinite timeout. The 
 connection will then block until established or an error occurs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-3417) CacheOnWrite is using the temporary output path for block names, need to use a more consistent block naming scheme

2011-10-13 Thread Jonathan Gray (Resolved) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jonathan Gray resolved HBASE-3417.
--

Resolution: Fixed
Release Note: StoreFile naming changes from random ascii longs to [0-9a-f]
uuids. The same name is used in the tmp and perm location. This name is what
gets used for the block name in the cache.

Committed to 92 and trunk. Thanks Stack and Mikhail for the speedy reviews.

CacheOnWrite is using the temporary output path for block names, need to use
a more consistent block naming scheme
--

Attachments: HBASE-3417-redux-v1.patch, HBASE-3417-v1.patch,
HBASE-3417-v2.patch, HBASE-3417-v5.patch

[jira] [Updated] (HBASE-3417) CacheOnWrite is using the temporary output path for block names, need to use a more consistent block naming scheme

[
https://issues.apache.org/jira/browse/HBASE-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jonathan Gray updated HBASE-3417:
-

Release Note:
StoreFile naming changes from random ascii longs to [0-9a-f] uuids. The same
name is used in the tmp and perm location. This name is what gets used for the
block name in the cache.

This change is a backwards compatible slow migration, all new files will be
named with the new scheme, old files will still be readable. However, once you
have new files created in the new scheme, previous installations will fail.

was:StoreFile naming changes from random ascii longs to [0-9a-f] uuids. The
same name is used in the tmp and perm location. This name is what gets used
for the block name in the cache.

CacheOnWrite is using the temporary output path for block names, need to use
a more consistent block naming scheme
--

Attachments: HBASE-3417-redux-v1.patch, HBASE-3417-v1.patch,
HBASE-3417-v2.patch, HBASE-3417-v5.patch

[jira] [Commented] (HBASE-3417) CacheOnWrite is using the temporary output path for block names, need to use a more consistent block naming scheme


[ 
https://issues.apache.org/jira/browse/HBASE-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127051#comment-13127051
 ] 

Jonathan Gray commented on HBASE-3417:
--

I didn't mark as incompatible but it is only one-way compatible.

There is actually a very trivial change that can be made in the 0.90 branch (or 
any other branches) to make this change compatible in all directions.  Just 
need to update the REF_NAME_PARSER regex to be what it is in this change 
(tolerant of [a-f] in addition to digits).  That's it.

 CacheOnWrite is using the temporary output path for block names, need to use 
 a more consistent block naming scheme
 --

 Key: HBASE-3417
 URL: https://issues.apache.org/jira/browse/HBASE-3417
 Project: HBase
  Issue Type: Bug
  Components: io, regionserver
Affects Versions: 0.92.0
Reporter: Jonathan Gray
Assignee: Jonathan Gray
Priority: Critical
 Fix For: 0.92.0

 Attachments: HBASE-3417-redux-v1.patch, HBASE-3417-v1.patch, 
 HBASE-3417-v2.patch, HBASE-3417-v5.patch


 Currently the block names used in the block cache are built using the 
 filesystem path.  However, for cache on write, the path is a temporary output 
 file.
 The original COW patch actually made some modifications to block naming stuff 
 to make it more consistent but did not do enough.  Should add a separate 
 method somewhere for generating block names using some more easily mocked 
 scheme (rather than just raw path as we generate a random unique file name 
 twice, once for tmp and then again when moved into place).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3417) CacheOnWrite is using the temporary output path for block names, need to use a more consistent block naming scheme

2011-10-13 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127054#comment-13127054
 ] 

Jonathan Gray commented on HBASE-3417:
--

In StoreFile.java:
{code}
   private static final Pattern REF_NAME_PARSER =
-Pattern.compile(^(\\d+)(?:\\.(.+))?$);
+Pattern.compile(^([0-9a-f]+)(?:\\.(.+))?$);
{code}

If you ever need to go backwards from 92 to a previous version.

 CacheOnWrite is using the temporary output path for block names, need to use 
 a more consistent block naming scheme
 --

 Key: HBASE-3417
 URL: https://issues.apache.org/jira/browse/HBASE-3417
 Project: HBase
  Issue Type: Bug
  Components: io, regionserver
Affects Versions: 0.92.0
Reporter: Jonathan Gray
Assignee: Jonathan Gray
Priority: Critical
 Fix For: 0.92.0

 Attachments: HBASE-3417-redux-v1.patch, HBASE-3417-v1.patch, 
 HBASE-3417-v2.patch, HBASE-3417-v5.patch


 Currently the block names used in the block cache are built using the 
 filesystem path.  However, for cache on write, the path is a temporary output 
 file.
 The original COW patch actually made some modifications to block naming stuff 
 to make it more consistent but did not do enough.  Should add a separate 
 method somewhere for generating block names using some more easily mocked 
 scheme (rather than just raw path as we generate a random unique file name 
 twice, once for tmp and then again when moved into place).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4568) Make zk dump jsp response more quickly


[ 
https://issues.apache.org/jira/browse/HBASE-4568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127058#comment-13127058
 ] 

jirapos...@reviews.apache.org commented on HBASE-4568:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2385/
---

(Updated 2011-10-13 22:33:54.109200)


Review request for hbase, Dhruba Borthakur, Michael Stack, Jonathan Gray, 
Mikhail Bautin, Pritam Damania, Prakash Khemani, Amitanand Aiyer, Kannan 
Muthukkaruppan, Jerry Chen, Karthik Ranganathan, and Nicolas Spiegelberg.


Changes
---

Remove trailing space.


Summary
---

1) For each zk dump, currently hbase will create a zk client instance every 
time. 
This is quite slow when any machines in the quorum is dead. Because it will 
connect to each machine in the zk quorum again.

code
HMaster master = (HMaster)getServletContext().getAttribute(HMaster.MASTER);
Configuration conf = master.getConfiguration();
HBaseAdmin hbadmin = new HBaseAdmin(conf);
HConnection connection = hbadmin.getConnection();
ZooKeeperWatcher watcher = connection.getZooKeeperWatcher();
/code

So we can simplify this:
code
HMaster master = (HMaster)getServletContext().getAttribute(HMaster.MASTER);
ZooKeeperWatcher watcher = master.getZooKeeperWatcher();
/code

2) Also when hbase call getServerStats() for each machine in the zk quorum, it 
hard coded the default time out as 1 min. 
It would be nice to make this configurable and set it to a low time out.

When hbase tries to connect to each machine in the zk quorum, it will create 
the socket, and then set the socket time out, and read it with this time out.
It means hbase will create a socket and connect to the zk server with 0 time 
out at first, which will take a long time. 
Because a timeout of zero is interpreted as an infinite timeout. The connection 
will then block until established or an error occurs.

3) The recoverable zookeeper should be real exponentially backoff when there is 
connection loss exception, which will give hbase much longer time window to 
recover from zk machine failures.


This addresses bug HBASE-4568.
https://issues.apache.org/jira/browse/HBASE-4568


Diffs (updated)
-

  src/main/java/org/apache/hadoop/hbase/util/RetryCounter.java 61ea552 
  src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java 
b8c4f61 
  src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 699a5f5 
  src/main/resources/hbase-webapps/master/zk.jsp b31d94c 

Diff: https://reviews.apache.org/r/2385/diff


Testing
---

Running all the unit tests


Thanks,

Liyin



 Make zk dump jsp response more quickly
 --

 Key: HBASE-4568
 URL: https://issues.apache.org/jira/browse/HBASE-4568
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang

 For each zk dump, currently hbase will create a zk client instance every 
 time. 
 This is quite slow when any machines in the quorum is dead. Because it will 
 connect to each machine in the zk quorum again.
   HMaster master = (HMaster)getServletContext().getAttribute(HMaster.MASTER);
   Configuration conf = master.getConfiguration();
   HBaseAdmin hbadmin = new HBaseAdmin(conf);
   HConnection connection = hbadmin.getConnection();
   ZooKeeperWatcher watcher = connection.getZooKeeperWatcher();
 So we can simplify this:
   HMaster master = (HMaster)getServletContext().getAttribute(HMaster.MASTER);
   ZooKeeperWatcher watcher = master.getZooKeeperWatcher();
 Also when hbase call getServerStats() for each machine in the zk quorum, it 
 hard coded the default time out as 1 min. 
 It would be nice to make this configurable and set it to a low time out.
 When hbase tries to connect to each machine in the zk quorum, it will create 
 the socket, and then set the socket time out, and read it with this time out.
 It means hbase will create a socket and connect to the zk server with 0 time 
 out at first, which will take a long time. 
 Because a timeout of zero is interpreted as an infinite timeout. The 
 connection will then block until established or an error occurs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4469) Avoid top row seek by looking up bloomfilter

2011-10-13 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127059#comment-13127059
 ] 

Jonathan Gray commented on HBASE-4469:
--

Liyin, can you post the final patch to this JIRA?  I will commit.  Thanks!

 Avoid top row seek by looking up bloomfilter
 

 Key: HBASE-4469
 URL: https://issues.apache.org/jira/browse/HBASE-4469
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang

 The problem is that when seeking for the row/col in the hfile, we will go to 
 top of the row in order to check for row delete marker (delete family). 
 However, if the bloomfilter is enabled for the column family, then if a 
 delete family operation is done on a row, the row is already being added to 
 bloomfilter. We can take advantage of this factor to avoid seeking to the top 
 of row.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4568) Make zk dump jsp response more quickly


[ 
https://issues.apache.org/jira/browse/HBASE-4568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127060#comment-13127060
 ] 

jirapos...@reviews.apache.org commented on HBASE-4568:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2385/
---

(Updated 2011-10-13 22:36:46.694728)


Review request for hbase, Dhruba Borthakur, Michael Stack, Jonathan Gray, 
Mikhail Bautin, Pritam Damania, Prakash Khemani, Amitanand Aiyer, Kannan 
Muthukkaruppan, Jerry Chen, Karthik Ranganathan, and Nicolas Spiegelberg.


Summary
---

1) For each zk dump, currently hbase will create a zk client instance every 
time. 
This is quite slow when any machines in the quorum is dead. Because it will 
connect to each machine in the zk quorum again.

code
HMaster master = (HMaster)getServletContext().getAttribute(HMaster.MASTER);
Configuration conf = master.getConfiguration();
HBaseAdmin hbadmin = new HBaseAdmin(conf);
HConnection connection = hbadmin.getConnection();
ZooKeeperWatcher watcher = connection.getZooKeeperWatcher();
/code

So we can simplify this:
code
HMaster master = (HMaster)getServletContext().getAttribute(HMaster.MASTER);
ZooKeeperWatcher watcher = master.getZooKeeperWatcher();
/code

2) Also when hbase call getServerStats() for each machine in the zk quorum, it 
hard coded the default time out as 1 min. 
It would be nice to make this configurable and set it to a low time out.

When hbase tries to connect to each machine in the zk quorum, it will create 
the socket, and then set the socket time out, and read it with this time out.
It means hbase will create a socket and connect to the zk server with 0 time 
out at first, which will take a long time. 
Because a timeout of zero is interpreted as an infinite timeout. The connection 
will then block until established or an error occurs.

3) The recoverable zookeeper should be real exponentially backoff when there is 
connection loss exception, which will give hbase much longer time window to 
recover from zk machine failures.


This addresses bug HBASE-4568.
https://issues.apache.org/jira/browse/HBASE-4568


Diffs (updated)
-

  src/main/java/org/apache/hadoop/hbase/util/RetryCounter.java 61ea552 
  src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java 
b8c4f61 
  src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 699a5f5 
  src/main/resources/hbase-webapps/master/zk.jsp b31d94c 

Diff: https://reviews.apache.org/r/2385/diff


Testing
---

Running all the unit tests


Thanks,

Liyin



 Make zk dump jsp response more quickly
 --

 Key: HBASE-4568
 URL: https://issues.apache.org/jira/browse/HBASE-4568
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang

 For each zk dump, currently hbase will create a zk client instance every 
 time. 
 This is quite slow when any machines in the quorum is dead. Because it will 
 connect to each machine in the zk quorum again.
   HMaster master = (HMaster)getServletContext().getAttribute(HMaster.MASTER);
   Configuration conf = master.getConfiguration();
   HBaseAdmin hbadmin = new HBaseAdmin(conf);
   HConnection connection = hbadmin.getConnection();
   ZooKeeperWatcher watcher = connection.getZooKeeperWatcher();
 So we can simplify this:
   HMaster master = (HMaster)getServletContext().getAttribute(HMaster.MASTER);
   ZooKeeperWatcher watcher = master.getZooKeeperWatcher();
 Also when hbase call getServerStats() for each machine in the zk quorum, it 
 hard coded the default time out as 1 min. 
 It would be nice to make this configurable and set it to a low time out.
 When hbase tries to connect to each machine in the zk quorum, it will create 
 the socket, and then set the socket time out, and read it with this time out.
 It means hbase will create a socket and connect to the zk server with 0 time 
 out at first, which will take a long time. 
 Because a timeout of zero is interpreted as an infinite timeout. The 
 connection will then block until established or an error occurs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4469) Avoid top row seek by looking up bloomfilter

2011-10-13 Thread Liyin Tang (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liyin Tang updated HBASE-4469:
--

Attachment: HBASE-4469_1.patch

 Avoid top row seek by looking up bloomfilter
 

 Key: HBASE-4469
 URL: https://issues.apache.org/jira/browse/HBASE-4469
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang
 Attachments: HBASE-4469_1.patch


 The problem is that when seeking for the row/col in the hfile, we will go to 
 top of the row in order to check for row delete marker (delete family). 
 However, if the bloomfilter is enabled for the column family, then if a 
 delete family operation is done on a row, the row is already being added to 
 bloomfilter. We can take advantage of this factor to avoid seeking to the top 
 of row.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4469) Avoid top row seek by looking up bloomfilter

2011-10-13 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127066#comment-13127066
 ] 

Liyin Tang commented on HBASE-4469:
---

Cool, I just downloaded the patch from review board 
(https://reviews.apache.org/r/2235/) and attached here:)
Thanks Jonathan.


 Avoid top row seek by looking up bloomfilter
 

 Key: HBASE-4469
 URL: https://issues.apache.org/jira/browse/HBASE-4469
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang
 Attachments: HBASE-4469_1.patch


 The problem is that when seeking for the row/col in the hfile, we will go to 
 top of the row in order to check for row delete marker (delete family). 
 However, if the bloomfilter is enabled for the column family, then if a 
 delete family operation is done on a row, the row is already being added to 
 bloomfilter. We can take advantage of this factor to avoid seeking to the top 
 of row.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4568) Make zk dump jsp response more quickly


[ 
https://issues.apache.org/jira/browse/HBASE-4568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127067#comment-13127067
 ] 

jirapos...@reviews.apache.org commented on HBASE-4568:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2385/#review2574
---


+1. Changes look good Liyin.

- Kannan


On 2011-10-13 22:36:46, Liyin wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2385/
bq.  ---
bq.  
bq.  (Updated 2011-10-13 22:36:46)
bq.  
bq.  
bq.  Review request for hbase, Dhruba Borthakur, Michael Stack, Jonathan Gray, 
Mikhail Bautin, Pritam Damania, Prakash Khemani, Amitanand Aiyer, Kannan 
Muthukkaruppan, Jerry Chen, Karthik Ranganathan, and Nicolas Spiegelberg.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  1) For each zk dump, currently hbase will create a zk client instance 
every time. 
bq.  This is quite slow when any machines in the quorum is dead. Because it 
will connect to each machine in the zk quorum again.
bq.  
bq.  code
bq.  HMaster master = (HMaster)getServletContext().getAttribute(HMaster.MASTER);
bq.  Configuration conf = master.getConfiguration();
bq.  HBaseAdmin hbadmin = new HBaseAdmin(conf);
bq.  HConnection connection = hbadmin.getConnection();
bq.  ZooKeeperWatcher watcher = connection.getZooKeeperWatcher();
bq.  /code
bq.  
bq.  So we can simplify this:
bq.  code
bq.  HMaster master = (HMaster)getServletContext().getAttribute(HMaster.MASTER);
bq.  ZooKeeperWatcher watcher = master.getZooKeeperWatcher();
bq.  /code
bq.  
bq.  2) Also when hbase call getServerStats() for each machine in the zk 
quorum, it hard coded the default time out as 1 min. 
bq.  It would be nice to make this configurable and set it to a low time out.
bq.  
bq.  When hbase tries to connect to each machine in the zk quorum, it will 
create the socket, and then set the socket time out, and read it with this time 
out.
bq.  It means hbase will create a socket and connect to the zk server with 0 
time out at first, which will take a long time. 
bq.  Because a timeout of zero is interpreted as an infinite timeout. The 
connection will then block until established or an error occurs.
bq.  
bq.  3) The recoverable zookeeper should be real exponentially backoff when 
there is connection loss exception, which will give hbase much longer time 
window to recover from zk machine failures.
bq.  
bq.  
bq.  This addresses bug HBASE-4568.
bq.  https://issues.apache.org/jira/browse/HBASE-4568
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/util/RetryCounter.java 61ea552 
bq.
src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java 
b8c4f61 
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 699a5f5 
bq.src/main/resources/hbase-webapps/master/zk.jsp b31d94c 
bq.  
bq.  Diff: https://reviews.apache.org/r/2385/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Running all the unit tests
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Liyin
bq.  
bq.



 Make zk dump jsp response more quickly
 --

 Key: HBASE-4568
 URL: https://issues.apache.org/jira/browse/HBASE-4568
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang

 For each zk dump, currently hbase will create a zk client instance every 
 time. 
 This is quite slow when any machines in the quorum is dead. Because it will 
 connect to each machine in the zk quorum again.
   HMaster master = (HMaster)getServletContext().getAttribute(HMaster.MASTER);
   Configuration conf = master.getConfiguration();
   HBaseAdmin hbadmin = new HBaseAdmin(conf);
   HConnection connection = hbadmin.getConnection();
   ZooKeeperWatcher watcher = connection.getZooKeeperWatcher();
 So we can simplify this:
   HMaster master = (HMaster)getServletContext().getAttribute(HMaster.MASTER);
   ZooKeeperWatcher watcher = master.getZooKeeperWatcher();
 Also when hbase call getServerStats() for each machine in the zk quorum, it 
 hard coded the default time out as 1 min. 
 It would be nice to make this configurable and set it to a low time out.
 When hbase tries to connect to each machine in the zk quorum, it will create 
 the socket, and then set the socket time out, and read it with this time out.
 It means hbase will create a socket and connect to the zk server with 0 time 
 out at first, which will take a long time. 
 Because a timeout of zero is interpreted as an infinite timeout. The 
 connection will then block until established or an

[jira] [Updated] (HBASE-4579) CST.requestCompaction semantics changed, logs are now spammed when too many store files

2011-10-13 Thread Jean-Daniel Cryans (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Daniel Cryans updated HBASE-4579:
--

Summary: CST.requestCompaction semantics changed, logs are now spammed when 
too many store files  (was: Compactions that can't run are requested forever)

I spent some time debugging this and the debug I added really helped:

{quote}
2011-10-13 22:04:16,633 DEBUG 
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Large Compaction 
requested: 
regionName=TestTable,0028803615,1318381132063.1724203bae0dda87cddfaac9700b9dc6.,
 storeName=info, fileCount=4, fileSize=219.6m (54.8m, 55.0m, 55.0m, 54.8m), 
priority=-1, time=6062002855079899; Because: regionserver62023.cacheFlusher; 
compaction_queue=(1:0), split_queue=0
2011-10-13 22:04:21,999 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
Flush requested on 
TestTable,0028803615,1318381132063.1724203bae0dda87cddfaac9700b9dc6.
2011-10-13 22:04:21,999 WARN 
org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Region 
TestTable,0028803615,1318381132063.1724203bae0dda87cddfaac9700b9dc6. has too 
many store files; delaying flush up to 9ms
2011-10-13 22:04:21,999 DEBUG org.apache.hadoop.hbase.regionserver.Store: info: 
no store files to compact 1724203bae0dda87cddfaac9700b9dc6
2011-10-13 22:04:22,000 ERROR org.apache.hadoop.hbase.regionserver.Store:
java.lang.Exception
at 
org.apache.hadoop.hbase.regionserver.Store.compactSelection(Store.java:1007)
at 
org.apache.hadoop.hbase.regionserver.Store.requestCompaction(Store.java:908)
at 
org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(CompactSplitThread.java:204)
at 
org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(CompactSplitThread.java:177)
at 
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:360)
at 
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:239)
...
2011-10-13 22:05:52,584 INFO 
org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Waited 90585ms on a 
compaction to clean up 'too many store files'; waited long enough... proceeding 
with flush of 
TestTable,0028803615,1318381132063.1724203bae0dda87cddfaac9700b9dc6.
{quote}

The exception is there just to show me where the call was coming from.

So what we see is that compaction is grabbing all the store files and during 
that time the MemStoreFlusher is trying to flush but hits the too many store 
files issue. It used to be that requesting a compaction was just putting 
something on a queue silently, but now it's actually going all the way down 
into Store and that's why it's now spammy.

Fixes I think are required:

 - For debugging purposes, show the encoded name when printing the info: no 
store files to compact message because it's pretty much useless if you don't 
know what it belongs to.
 - Require the compaction only once, the code already knows when it's the first 
time you spin so no need to ask it again and again.
 - Set a higher priority for the compaction

 CST.requestCompaction semantics changed, logs are now spammed when too many 
 store files
 ---

 Key: HBASE-4579
 URL: https://issues.apache.org/jira/browse/HBASE-4579
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Priority: Critical
 Fix For: 0.92.0


 Another bug I'm not so sure what's going on. I see this in my log:
 {quote}
 2011-10-12 00:23:43,435 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
 info: no store files to compact
 2011-10-12 00:23:44,335 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
 info: no store files to compact
 2011-10-12 00:23:45,236 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
 info: no store files to compact
 2011-10-12 00:23:46,136 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
 info: no store files to compact
 2011-10-12 00:23:47,036 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
 info: no store files to compact
 2011-10-12 00:23:47,936 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
 info: no store files to compact
 {quote}
 It spams for a while, and a little later instead I get:
 {quote}
 2011-10-12 00:26:52,139 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
 Skipped compaction of info.  Only 2 file(s) of size 176.4m have met 
 compaction criteria.
 2011-10-12 00:26:53,040 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
 Skipped compaction of info.  Only 2 file(s) of size 176.4m have met 
 compaction criteria.
 2011-10-12 00:26:53,940 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
 Skipped compaction of info.  Only 2 file(s) of size 176.4m have met 
 compaction criteria.
 2011-10-12 00:26:54,840 DEBUG

[jira] [Commented] (HBASE-4469) Avoid top row seek by looking up bloomfilter

2011-10-13 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127071#comment-13127071
 ] 

Jonathan Gray commented on HBASE-4469:
--

Thanks Liyin.  Unfortunately because the RB integration isn't very tight, to 
follow Apache protocol, you need to attach the patch to the JIRA and select the 
radio button that assigns it to apache.

This also helps to ensure that there's no confusion about which version was 
committed and that we don't have a hard dependency on RB in any way.

It'll all be second nature before you know it :)

 Avoid top row seek by looking up bloomfilter
 

 Key: HBASE-4469
 URL: https://issues.apache.org/jira/browse/HBASE-4469
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang
 Attachments: HBASE-4469_1.patch


 The problem is that when seeking for the row/col in the hfile, we will go to 
 top of the row in order to check for row delete marker (delete family). 
 However, if the bloomfilter is enabled for the column family, then if a 
 delete family operation is done on a row, the row is already being added to 
 bloomfilter. We can take advantage of this factor to avoid seeking to the top 
 of row.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4585) Avoid seek operation when current kv is deleted


[ 
https://issues.apache.org/jira/browse/HBASE-4585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127077#comment-13127077
 ] 

jirapos...@reviews.apache.org commented on HBASE-4585:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2384/#review2571
---


+1. Another nice optimization Liyin!!! One cosmetic comment inlined.


src/main/java/org/apache/hadoop/hbase/regionserver/DeleteTracker.java
https://reviews.apache.org/r/2384/#comment5765

returns code - return codes


- Kannan


On 2011-10-13 22:18:51, Liyin wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2384/
bq.  ---
bq.  
bq.  (Updated 2011-10-13 22:18:51)
bq.  
bq.  
bq.  Review request for hbase, Dhruba Borthakur, Michael Stack, Jonathan Gray, 
Mikhail Bautin, Kannan Muthukkaruppan, Jerry Chen, Karthik Ranganathan, and 
Nicolas Spiegelberg.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  When the current kv is deleted during the matching in the 
ScanQueryMatcher, currently the matcher will return skip and continue to seek.
bq.  Actually, if the current kv is deleted because of family deleted or column 
deleted, the matcher should seek to next col.
bq.  If the current kv is deleted because of version deleted, the matcher 
should just return skip.
bq.  
bq.  I will submit another patch against apache-trunk later.
bq.  
bq.  
bq.  This addresses bug HBASE-4585.
bq.  https://issues.apache.org/jira/browse/HBASE-4585
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanDeleteTracker.java 
afb3fcc 
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 
e8ac027 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 
a1d7de5 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/DeleteTracker.java 
b425bf2 
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/ScanDeleteTracker.java 
1a891a6 
bq.  
bq.  Diff: https://reviews.apache.org/r/2384/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Running all the unit tests.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Liyin
bq.  
bq.



 Avoid seek operation when current kv is deleted
 ---

 Key: HBASE-4585
 URL: https://issues.apache.org/jira/browse/HBASE-4585
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang

 When the current kv is deleted during the matching in the ScanQueryMatcher, 
 currently the matcher will return skip and continue to seek.
 Actually, if the current kv is deleted because of family deleted or column 
 deleted, the matcher should seek to next col.
 If the current kv is deleted because of version deleted, the matcher should 
 just return skip.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4218) Delta Encoding of KeyValues (aka prefix compression)

2011-10-13 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127079#comment-13127079
 ] 

jirapos...@reviews.apache.org commented on HBASE-4218:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2308/#review2573
---



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
https://reviews.apache.org/r/2308/#comment5767

Nit [Coding style]: space between (byte) and 9.



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
https://reviews.apache.org/r/2308/#comment5769

Add a comment about what the following string constants are for (presumably 
FileInfo keys).



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
https://reviews.apache.org/r/2308/#comment5768

Remove trailing whitespace here and below.



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
https://reviews.apache.org/r/2308/#comment5770

Create a string constant for NONE.



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
https://reviews.apache.org/r/2308/#comment5771

Use the string constant for NONE.



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/KeyValue.java
https://reviews.apache.org/r/2308/#comment5772

Size of the key length field in bytes



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/KeyValue.java
https://reviews.apache.org/r/2308/#comment5773

Size of the key type field in bytes



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/KeyValue.java
https://reviews.apache.org/r/2308/#comment5774

Size of the row length field in bytes



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/KeyValue.java
https://reviews.apache.org/r/2308/#comment5775

Size of the family length field in bytes



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/KeyValue.java
https://reviews.apache.org/r/2308/#comment5776

Size of the timestamp field in bytes



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/KeyValue.java
https://reviews.apache.org/r/2308/#comment5777

This needs to use the new constants defined for row length, etc.


- Mikhail


On 2011-10-08 00:51:01, Jacek Migdal wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2308/
bq.  ---
bq.  
bq.  (Updated 2011-10-08 00:51:01)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Delta encoding for key values.
bq.  
bq.  
bq.  This addresses bug HBASE-4218.
bq.  https://issues.apache.org/jira/browse/HBASE-4218
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
 1180113 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/KeyValue.java
 1180113 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java
 1180113 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/deltaencoder/BitsetKeyDeltaEncoder.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/deltaencoder/BufferedDeltaEncoder.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/deltaencoder/CompressionState.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/deltaencoder/CopyKeyDeltaEncoder.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/deltaencoder/DeltaEncodedBlock.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/deltaencoder/DeltaEncoder.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/deltaencoder/DeltaEncoderAlgorithms.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/deltaencoder/DeltaEncoderToSmallBufferException.java
 PRE-CREATION 
bq.

[jira] [Updated] (HBASE-4551) Small fixes to compile against 0.23-SNAPSHOT

2011-10-13 Thread Todd Lipcon (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon updated HBASE-4551:
---

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Committed to 92 and trunk

 Small fixes to compile against 0.23-SNAPSHOT
 

 Key: HBASE-4551
 URL: https://issues.apache.org/jira/browse/HBASE-4551
 Project: HBase
  Issue Type: Bug
  Components: build
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
 Fix For: 0.92.0

 Attachments: hbase-4551.txt, hbase-4551.txt


 - fix pom.xml to properly pull the test artifacts
 - fix TestHLog to not use the private cluster.getNameNode() API

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4469) Avoid top row seek by looking up bloomfilter


 [ 
https://issues.apache.org/jira/browse/HBASE-4469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Gray updated HBASE-4469:
-

Fix Version/s: 0.94.0

Committed to trunk.

 Avoid top row seek by looking up bloomfilter
 

 Key: HBASE-4469
 URL: https://issues.apache.org/jira/browse/HBASE-4469
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang
 Fix For: 0.94.0

 Attachments: HBASE-4469_1.patch


 The problem is that when seeking for the row/col in the hfile, we will go to 
 top of the row in order to check for row delete marker (delete family). 
 However, if the bloomfilter is enabled for the column family, then if a 
 delete family operation is done on a row, the row is already being added to 
 bloomfilter. We can take advantage of this factor to avoid seeking to the top 
 of row.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4469) Avoid top row seek by looking up bloomfilter


[ 
https://issues.apache.org/jira/browse/HBASE-4469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127089#comment-13127089
 ] 

Jonathan Gray commented on HBASE-4469:
--

What is the protocol now?  This needs to go into the fb-89 branch, so do I keep 
this JIRA open until that happens, or should we just add some fb-89-pending tag 
or something?

 Avoid top row seek by looking up bloomfilter
 

 Key: HBASE-4469
 URL: https://issues.apache.org/jira/browse/HBASE-4469
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang
 Fix For: 0.94.0

 Attachments: HBASE-4469_1.patch


 The problem is that when seeking for the row/col in the hfile, we will go to 
 top of the row in order to check for row delete marker (delete family). 
 However, if the bloomfilter is enabled for the column family, then if a 
 delete family operation is done on a row, the row is already being added to 
 bloomfilter. We can take advantage of this factor to avoid seeking to the top 
 of row.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4469) Avoid top row seek by looking up bloomfilter


[ 
https://issues.apache.org/jira/browse/HBASE-4469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127090#comment-13127090
 ] 

Jonathan Gray commented on HBASE-4469:
--

(i'm not putting in 92 branch because this is feature)

 Avoid top row seek by looking up bloomfilter
 

 Key: HBASE-4469
 URL: https://issues.apache.org/jira/browse/HBASE-4469
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang
 Fix For: 0.94.0

 Attachments: HBASE-4469_1.patch


 The problem is that when seeking for the row/col in the hfile, we will go to 
 top of the row in order to check for row delete marker (delete family). 
 However, if the bloomfilter is enabled for the column family, then if a 
 delete family operation is done on a row, the row is already being added to 
 bloomfilter. We can take advantage of this factor to avoid seeking to the top 
 of row.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4078) Silent Data Offlining During HDFS Flakiness

2011-10-13 Thread Hudson (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127103#comment-13127103
 ] 

Hudson commented on HBASE-4078:
---

Integrated in HBase-0.92 #62 (See 
[https://builds.apache.org/job/HBase-0.92/62/])
HBASE-4078 Validate store files after flush/compaction

nspiegelberg : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* 
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java


 Silent Data Offlining During HDFS Flakiness
 ---

 Key: HBASE-4078
 URL: https://issues.apache.org/jira/browse/HBASE-4078
 Project: HBase
  Issue Type: Bug
  Components: io, regionserver
Affects Versions: 0.89.20100924, 0.90.3, 0.92.0
Reporter: Nicolas Spiegelberg
Assignee: Pritam Damania
Priority: Blocker
 Fix For: 0.92.0, 0.94.0

 Attachments: 
 0001-Validate-store-files-after-compactions-flushes.patch, 
 0001-Validate-store-files.patch


 See HBASE-1436 .  The bug fix for this JIRA is a temporary workaround for 
 improperly moving partially-written files from TMP into the region directory 
 when a FS error occurs.  Unfortunately, the fix is to ignore all IO 
 exceptions, which masks off-lining due to FS flakiness.  We need to 
 permanently fix the problem that created HBASE-1436  then at least have the 
 option to not open a region during times of flakey FS.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-3446) ProcessServerShutdown fails if META moves, orphaning lots of regions

2011-10-13 Thread stack (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-3446:
-

Attachment: 3446v23.txt

Here is what I just committed to 0.92 and trunk.

 ProcessServerShutdown fails if META moves, orphaning lots of regions
 

 Key: HBASE-3446
 URL: https://issues.apache.org/jira/browse/HBASE-3446
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.0
Reporter: Todd Lipcon
Assignee: stack
Priority: Blocker
 Fix For: 0.92.0

 Attachments: 3446-v11.txt, 3446-v12.txt, 3446-v13.txt, 3446-v14.txt, 
 3446-v2.txt, 3446-v3.txt, 3446-v4.txt, 3446-v7.txt, 3446-v9.txt, 3446.txt, 
 3446v15.txt, 3446v23.txt


 I ran a rolling restart on a 5 node cluster with lots of regions, and 
 afterwards had LOTS of regions left orphaned. The issue appears to be that 
 ProcessServerShutdown failed because the server hosting META was restarted 
 around the same time as another server was being processed

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4469) Avoid top row seek by looking up bloomfilter


[ 
https://issues.apache.org/jira/browse/HBASE-4469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127114#comment-13127114
 ] 

Liyin Tang commented on HBASE-4469:
---

@Jonathan, 
For this jira specifically, it has been committed to 89-fb internal branch 
before cutting the public 89-fb branch.
So it should already in the public 89-fb right now.




 Avoid top row seek by looking up bloomfilter
 

 Key: HBASE-4469
 URL: https://issues.apache.org/jira/browse/HBASE-4469
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang
 Fix For: 0.94.0

 Attachments: HBASE-4469_1.patch


 The problem is that when seeking for the row/col in the hfile, we will go to 
 top of the row in order to check for row delete marker (delete family). 
 However, if the bloomfilter is enabled for the column family, then if a 
 delete family operation is done on a row, the row is already being added to 
 bloomfilter. We can take advantage of this factor to avoid seeking to the top 
 of row.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-4579) CST.requestCompaction semantics changed, logs are now spammed when too many store files

2011-10-13 Thread Jean-Daniel Cryans (Assigned) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Daniel Cryans reassigned HBASE-4579:
-

Assignee: Jean-Daniel Cryans

 CST.requestCompaction semantics changed, logs are now spammed when too many 
 store files
 ---

 Key: HBASE-4579
 URL: https://issues.apache.org/jira/browse/HBASE-4579
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: Jean-Daniel Cryans
Priority: Critical
 Fix For: 0.92.0


 Another bug I'm not so sure what's going on. I see this in my log:
 {quote}
 2011-10-12 00:23:43,435 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
 info: no store files to compact
 2011-10-12 00:23:44,335 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
 info: no store files to compact
 2011-10-12 00:23:45,236 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
 info: no store files to compact
 2011-10-12 00:23:46,136 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
 info: no store files to compact
 2011-10-12 00:23:47,036 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
 info: no store files to compact
 2011-10-12 00:23:47,936 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
 info: no store files to compact
 {quote}
 It spams for a while, and a little later instead I get:
 {quote}
 2011-10-12 00:26:52,139 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
 Skipped compaction of info.  Only 2 file(s) of size 176.4m have met 
 compaction criteria.
 2011-10-12 00:26:53,040 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
 Skipped compaction of info.  Only 2 file(s) of size 176.4m have met 
 compaction criteria.
 2011-10-12 00:26:53,940 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
 Skipped compaction of info.  Only 2 file(s) of size 176.4m have met 
 compaction criteria.
 2011-10-12 00:26:54,840 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
 Skipped compaction of info.  Only 2 file(s) of size 176.4m have met 
 compaction criteria.
 2011-10-12 00:26:55,741 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
 Skipped compaction of info.  Only 2 file(s) of size 176.4m have met 
 compaction criteria.
 2011-10-12 00:26:56,641 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
 Skipped compaction of info.  Only 2 file(s) of size 176.4m have met 
 compaction criteria.
 {quote}
 I believe I also saw something like that for flushes, but the region was 
 closing so at least I know why it was spamming (would be nice if it just 
 unrequested the flush):
 {quote}
 2011-10-12 00:26:40,693 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
 Flush requested on 
 TestTable,0038168581,1318378894213.2beb8a1e29382a8d3e90a88b9662e5f5.
 2011-10-12 00:26:40,694 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
 NOT flushing memstore for region 
 TestTable,0038168581,1318378894213.2beb8a1e29382a8d3e90a88b9662e5f5., 
 flushing=false, writesEnabled=false
 2011-10-12 00:26:40,733 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
 Flush requested on 
 TestTable,0038168581,1318378894213.2beb8a1e29382a8d3e90a88b9662e5f5.
 2011-10-12 00:26:40,733 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
 NOT flushing memstore for region 
 TestTable,0038168581,1318378894213.2beb8a1e29382a8d3e90a88b9662e5f5., 
 flushing=false, writesEnabled=false
 2011-10-12 00:26:40,873 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
 Flush requested on 
 TestTable,0038168581,1318378894213.2beb8a1e29382a8d3e90a88b9662e5f5.
 2011-10-12 00:26:40,873 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
 NOT flushing memstore for region 
 TestTable,0038168581,1318378894213.2beb8a1e29382a8d3e90a88b9662e5f5., 
 flushing=false, writesEnabled=false
 2011-10-12 00:26:40,873 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
 Flush requested on 
 TestTable,0038168581,1318378894213.2beb8a1e29382a8d3e90a88b9662e5f5.
 2011-10-12 00:26:40,873 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
 NOT flushing memstore for region 
 TestTable,0038168581,1318378894213.2beb8a1e29382a8d3e90a88b9662e5f5., 
 flushing=false, writesEnabled=false
 2011-10-12 00:26:40,921 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
 Flush requested on 
 TestTable,0038168581,1318378894213.2beb8a1e29382a8d3e90a88b9662e5f5.
 2011-10-12 00:26:40,922 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
 NOT flushing memstore for region 
 TestTable,0038168581,1318378894213.2beb8a1e29382a8d3e90a88b9662e5f5., 
 flushing=false, writesEnabled=false
 2011-10-12 00:26:40,923 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
 Flush requested on 
 TestTable,0038168581,1318378894213.2beb8a1e29382a8d3e90a88b9662e5f5.
 2011-10-12 00:26:40,923 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
 NOT flushing memstore for

[jira] [Updated] (HBASE-4469) Avoid top row seek by looking up bloomfilter


 [ 
https://issues.apache.org/jira/browse/HBASE-4469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Gray updated HBASE-4469:
-

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Got it, thanks Liyin!  Nice work!

 Avoid top row seek by looking up bloomfilter
 

 Key: HBASE-4469
 URL: https://issues.apache.org/jira/browse/HBASE-4469
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang
 Fix For: 0.94.0

 Attachments: HBASE-4469_1.patch


 The problem is that when seeking for the row/col in the hfile, we will go to 
 top of the row in order to check for row delete marker (delete family). 
 However, if the bloomfilter is enabled for the column family, then if a 
 delete family operation is done on a row, the row is already being added to 
 bloomfilter. We can take advantage of this factor to avoid seeking to the top 
 of row.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4579) CST.requestCompaction semantics changed, logs are now spammed when too many store files

2011-10-13 Thread Jean-Daniel Cryans (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Daniel Cryans updated HBASE-4579:
--

Attachment: HBASE-4579.patch

Does the 3 things I listed above, tested on a cluster and verified it stopped 
spamming.

 CST.requestCompaction semantics changed, logs are now spammed when too many 
 store files
 ---

 Key: HBASE-4579
 URL: https://issues.apache.org/jira/browse/HBASE-4579
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: Jean-Daniel Cryans
Priority: Critical
 Fix For: 0.92.0

 Attachments: HBASE-4579.patch


 Another bug I'm not so sure what's going on. I see this in my log:
 {quote}
 2011-10-12 00:23:43,435 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
 info: no store files to compact
 2011-10-12 00:23:44,335 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
 info: no store files to compact
 2011-10-12 00:23:45,236 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
 info: no store files to compact
 2011-10-12 00:23:46,136 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
 info: no store files to compact
 2011-10-12 00:23:47,036 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
 info: no store files to compact
 2011-10-12 00:23:47,936 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
 info: no store files to compact
 {quote}
 It spams for a while, and a little later instead I get:
 {quote}
 2011-10-12 00:26:52,139 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
 Skipped compaction of info.  Only 2 file(s) of size 176.4m have met 
 compaction criteria.
 2011-10-12 00:26:53,040 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
 Skipped compaction of info.  Only 2 file(s) of size 176.4m have met 
 compaction criteria.
 2011-10-12 00:26:53,940 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
 Skipped compaction of info.  Only 2 file(s) of size 176.4m have met 
 compaction criteria.
 2011-10-12 00:26:54,840 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
 Skipped compaction of info.  Only 2 file(s) of size 176.4m have met 
 compaction criteria.
 2011-10-12 00:26:55,741 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
 Skipped compaction of info.  Only 2 file(s) of size 176.4m have met 
 compaction criteria.
 2011-10-12 00:26:56,641 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
 Skipped compaction of info.  Only 2 file(s) of size 176.4m have met 
 compaction criteria.
 {quote}
 I believe I also saw something like that for flushes, but the region was 
 closing so at least I know why it was spamming (would be nice if it just 
 unrequested the flush):
 {quote}
 2011-10-12 00:26:40,693 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
 Flush requested on 
 TestTable,0038168581,1318378894213.2beb8a1e29382a8d3e90a88b9662e5f5.
 2011-10-12 00:26:40,694 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
 NOT flushing memstore for region 
 TestTable,0038168581,1318378894213.2beb8a1e29382a8d3e90a88b9662e5f5., 
 flushing=false, writesEnabled=false
 2011-10-12 00:26:40,733 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
 Flush requested on 
 TestTable,0038168581,1318378894213.2beb8a1e29382a8d3e90a88b9662e5f5.
 2011-10-12 00:26:40,733 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
 NOT flushing memstore for region 
 TestTable,0038168581,1318378894213.2beb8a1e29382a8d3e90a88b9662e5f5., 
 flushing=false, writesEnabled=false
 2011-10-12 00:26:40,873 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
 Flush requested on 
 TestTable,0038168581,1318378894213.2beb8a1e29382a8d3e90a88b9662e5f5.
 2011-10-12 00:26:40,873 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
 NOT flushing memstore for region 
 TestTable,0038168581,1318378894213.2beb8a1e29382a8d3e90a88b9662e5f5., 
 flushing=false, writesEnabled=false
 2011-10-12 00:26:40,873 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
 Flush requested on 
 TestTable,0038168581,1318378894213.2beb8a1e29382a8d3e90a88b9662e5f5.
 2011-10-12 00:26:40,873 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
 NOT flushing memstore for region 
 TestTable,0038168581,1318378894213.2beb8a1e29382a8d3e90a88b9662e5f5., 
 flushing=false, writesEnabled=false
 2011-10-12 00:26:40,921 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
 Flush requested on 
 TestTable,0038168581,1318378894213.2beb8a1e29382a8d3e90a88b9662e5f5.
 2011-10-12 00:26:40,922 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
 NOT flushing memstore for region 
 TestTable,0038168581,1318378894213.2beb8a1e29382a8d3e90a88b9662e5f5., 
 flushing=false, writesEnabled=false
 2011-10-12 00:26:40,923 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
 Flush requested on

[jira] [Commented] (HBASE-4335) Splits can create temporary holes in .META. that confuse clients and regionservers