date:20120322

2012-03-22 Thread ramkrishna.s.vasudevan (Updated) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235536#comment-13235536
]

Hadoop QA commented on HBASE-5606:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12519319/5606.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1255//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1255//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1255//console

This message is automatically generated.

SplitLogManger async delete node hangs log splitting when ZK connection is
lost

Key: HBASE-5606
URL: https://issues.apache.org/jira/browse/HBASE-5606
Project: HBase
Issue Type: Bug
Components: wal
Affects Versions: 0.92.0
Reporter: Gopinathan A
Priority: Critical
Fix For: 0.92.2

Attachments: 5606.txt

1. One rs died, the servershutdownhandler found it out and started the
distributed log splitting;
2. All tasks are failed due to ZK connection lost, so the all the tasks were
deleted asynchronously;
3. Servershutdownhandler retried the log splitting;
4. The asynchronously deletion in step 2 finally happened for new task
5. This made the SplitLogManger in hanging state.
This leads to .META. region not assigened for long time
{noformat}
hbase-root-master-HOST-192-168-47-204.log.2012-03-14(55413,79):2012-03-14
19:28:47,932 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: put up
splitlog task at znode
/hbase/splitlog/hdfs%3A%2F%2F192.168.47.205%3A9000%2Fhbase%2F.logs%2Flinux-114.site%2C60020%2C1331720381665-splitting%2Flinux-114.site%252C60020%252C1331720381665.1331752316170
hbase-root-master-HOST-192-168-47-204.log.2012-03-14(89303,79):2012-03-14
19:34:32,387 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: put up
splitlog task at znode
/hbase/splitlog/hdfs%3A%2F%2F192.168.47.205%3A9000%2Fhbase%2F.logs%2Flinux-114.site%2C60020%2C1331720381665-splitting%2Flinux-114.site%252C60020%252C1331720381665.1331752316170
{noformat}
{noformat}
hbase-root-master-HOST-192-168-47-204.log.2012-03-14(80417,99):2012-03-14
19:34:31,196 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback: deleted
/hbase/splitlog/hdfs%3A%2F%2F192.168.47.205%3A9000%2Fhbase%2F.logs%2Flinux-114.site%2C60020%2C1331720381665-splitting%2Flinux-114.site%252C60020%252C1331720381665.1331752316170
hbase-root-master-HOST-192-168-47-204.log.2012-03-14(89456,99):2012-03-14
19:34:32,497 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback: deleted
/hbase/splitlog/hdfs%3A%2F%2F192.168.47.205%3A9000%2Fhbase%2F.logs%2Flinux-114.site%2C60020%2C1331720381665-splitting%2Flinux-114.site%252C60020%252C1331720381665.1331752316170
{noformat}

[jira] [Commented] (HBASE-5615) the master never do balance becauseof balance the parent region

2012-03-22 Thread xufeng (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235541#comment-13235541
 ] 

xufeng commented on HBASE-5615:
---

the log of step2 from 158-1-131-48,20020,1331918756600

 the master never do balance becauseof  balance the parent region
 

 Key: HBASE-5615
 URL: https://issues.apache.org/jira/browse/HBASE-5615
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.7
Reporter: xufeng
Assignee: xufeng
Priority: Critical
 Attachments: HBASE-5615.patch


 the master never do balance becauseof when master do rebuildUserRegions()，it 
 will add the parent region into  AssignmentManager#servers,
 if balancer let the parent region to move,the parent will in RIT forever.thus 
 balance will never be executed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3134) [replication] Add the ability to enable/disable streams

2012-03-22 Thread Teruyoshi Zenmyo (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235593#comment-13235593
 ] 

Teruyoshi Zenmyo commented on HBASE-3134:
-

Thanks for feedbacks and testing J-D.
I'll update the patch (


 [replication] Add the ability to enable/disable streams
 ---

 Key: HBASE-3134
 URL: https://issues.apache.org/jira/browse/HBASE-3134
 Project: HBase
  Issue Type: New Feature
  Components: replication
Reporter: Jean-Daniel Cryans
Assignee: Teruyoshi Zenmyo
Priority: Minor
  Labels: replication
 Fix For: 0.94.0

 Attachments: 3134-v2.txt, 3134-v3.txt, 3134.txt, HBASE-3134.patch, 
 HBASE-3134.patch, HBASE-3134.patch, HBASE-3134.patch


 This jira was initially in the scope of HBASE-2201, but was pushed out since 
 it has low value compared to the required effort (and when want to ship 
 0.90.0 rather soonish).
 We need to design a way to enable/disable replication streams in a 
 determinate fashion.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5584) Coprocessor hooks can be called in the respective handlers


 [ 
https://issues.apache.org/jira/browse/HBASE-5584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-5584:
--

Attachment: HBASE-5584-2.patch

@Andrew
Instead of sleep i have added CountdownLatch for the assertion of create table. 
 Pls review.

 Coprocessor hooks can be called in the respective handlers
 --

 Key: HBASE-5584
 URL: https://issues.apache.org/jira/browse/HBASE-5584
 Project: HBase
  Issue Type: Improvement
  Components: coprocessors
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Fix For: 0.96.0

 Attachments: HBASE-5584-1.patch, HBASE-5584-2.patch, HBASE-5584.patch


 Following points can be changed w.r.t to coprocessors
 - Call preCreate, postCreate, preEnable, postEnable, etc. in their 
 respective handlers
 - Currently it is called in the HMaster thus making the postApis async w.r.t 
 the handlers
 - Similar is the case with the balancer.
 with current behaviour once we are in the postEnable(for eg) we any way need 
 to wait for the main enable handler to 
 be completed.
 We should ensure that we dont wait in the main thread so again we need to 
 spawn a thread and wait on that.
 On the other hand if the pre and post api is called on the handlers then only 
 that handler thread will be
 used in the pre/post apis
 If the above said plan is ok i can prepare a patch for all such related 
 changes.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3134) [replication] Add the ability to enable/disable streams

2012-03-22 Thread Teruyoshi Zenmyo (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235597#comment-13235597
 ] 

Teruyoshi Zenmyo commented on HBASE-3134:
-

I'm sorry, the above has sent on the way by mistake. 

Thanks for feedbacks and testing, J-D.
I'll update the patch (maybe tomorrow if I can)

@Lars
Please push this to later version.
I hope that 0.94 is released faster.

Thanks

 [replication] Add the ability to enable/disable streams
 ---

 Key: HBASE-3134
 URL: https://issues.apache.org/jira/browse/HBASE-3134
 Project: HBase
  Issue Type: New Feature
  Components: replication
Reporter: Jean-Daniel Cryans
Assignee: Teruyoshi Zenmyo
Priority: Minor
  Labels: replication
 Fix For: 0.94.0

 Attachments: 3134-v2.txt, 3134-v3.txt, 3134.txt, HBASE-3134.patch, 
 HBASE-3134.patch, HBASE-3134.patch, HBASE-3134.patch


 This jira was initially in the scope of HBASE-2201, but was pushed out since 
 it has low value compared to the required effort (and when want to ship 
 0.90.0 rather soonish).
 We need to design a way to enable/disable replication streams in a 
 determinate fashion.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5335) Dynamic Schema Configurations

2012-03-22 Thread Nicolas Spiegelberg (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235602#comment-13235602
]

Nicolas Spiegelberg commented on HBASE-5335:

Working on a port for trunk. 3 items:

1) coprocessors already use the generic KV api to store their info in
HTableDescriptor. would be nice for someone with coprocessor experience to
view it.
2) Because of #1, may want to consider making a separate config map for this
JIRA. this would require changing the persistent data format though :( We
could use the keyword 'CONFIG' instead of 'ADVANCED'?
3) There is currently a bug in the trunk version of online schema change where
HBaseAdmin.getAlterStatus() is not synchronous. This is causing the new
TestFromClientSide3 to fail. Problem does not exist in 89fb :(

Dynamic Schema Configurations
-

Key: HBASE-5335
URL: https://issues.apache.org/jira/browse/HBASE-5335
Project: HBase
Issue Type: New Feature
Reporter: Nicolas Spiegelberg
Assignee: Nicolas Spiegelberg
Labels: configuration, schema
Attachments: D2247.1.patch, D2247.2.patch, D2247.3.patch,
D2247.4.patch, D2247.5.patch, D2247.6.patch, D2247.7.patch

Currently, the ability for a core developer to add per-table per-CF
configuration settings is very heavyweight. You need to add a reserved
keyword all the way up the stack you have to support this variable
long-term if you're going to expose it explicitly to the user. This has
ended up with using Configuration.get() a lot because it is lightweight and
you can tweak settings while you're trying to understand system behavior
[since there are many config params that may never need to be tuned]. We
need to add the ability to put read arbitrary KV settings in the HBase
schema. Combined with online schema change, this will allow us to safely
iterate on configuration settings.

[jira] [Commented] (HBASE-4955) Use the official versions of surefire junit

2012-03-22 Thread nkeywal (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235605#comment-13235605
]

nkeywal commented on HBASE-4955:

Monthly update:
- The JUnit team merged my fix on the JUnit trunk, so the next JUnit version
(should be 4.11) will be ok for us. I don't have the release date however.
- The surefire regression (SUREFIRE-827) is not yet fixed, still planned for
2.13. No release date.

Use the official versions of surefire junit
-

Key: HBASE-4955
URL: https://issues.apache.org/jira/browse/HBASE-4955
Project: HBase
Issue Type: Improvement
Components: test
Affects Versions: 0.94.0
Environment: all
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor

We currently use private versions for Surefire JUnit since HBASE-4763.
This JIRA traks what we need to move to official versions.
Surefire 2.11 is just out, but, after some tests, it does not contain all
what we need.
JUnit. Could be for JUnit 4.11. Issue to monitor:
https://github.com/KentBeck/junit/issues/359: fixed in our version, no
feedback for an integration on trunk
Surefire: Could be for Surefire 2.12. Issues to monitor are:
329 (category support): fixed, we use the official implementation from the
trunk
786 (@Category with forkMode=always): fixed, we use the official
implementation from the trunk
791 (incorrect elapsed time on test failure): fixed, we use the official
implementation from the trunk
793 (incorrect time in the XML report): Not fixed (reopen) on trunk, fixed on
our version.
760 (does not take into account the test method): fixed in trunk, not fixed
in our version
798 (print immediately the test class name): not fixed in trunk, not fixed in
our version
799 (Allow test parallelization when forkMode=always): not fixed in trunk,
not fixed in our version
800 (redirectTestOutputToFile not taken into account): not yet fix on trunk,
fixed on our version
800 793 are the more important to monitor, it's the only ones that are
fixed in our version but not on trunk.

[jira] [Commented] (HBASE-5589) Add of the offline call to the Master Interface

2012-03-22 Thread Jonathan Hsieh (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235608#comment-13235608
 ] 

Jonathan Hsieh commented on HBASE-5589:
---

@Ram I believe I've fixed this.  At least according to the subversion tab here 
on this jira, CHANGES.txt has been updated.

 Add of the offline call to the Master Interface
 ---

 Key: HBASE-5589
 URL: https://issues.apache.org/jira/browse/HBASE-5589
 Project: HBase
  Issue Type: Improvement
  Components: hbck
Affects Versions: 0.90.6, 0.92.0, 0.94.0, 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0

 Attachments: hbase-5588-0.94.patch, hbase-5589-0.90.patch, 
 hbase-5589-0.92.patch, hbase-5589.patch


 Hbck from HBASE-5128 requires an offline method on the master to properly 
 cleanup state during certain assignment repair operations.  This will this 
 method will be added to recent and older versions of HBase.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5328) Small changes to Master to make it more testable

2012-03-22 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235612#comment-13235612
 ] 

jirapos...@reviews.apache.org commented on HBASE-5328:
--



bq.  On 2012-03-22 00:25:30, jmhsieh wrote:
bq.   src/main/java/org/apache/hadoop/hbase/master/HMaster.java, line 493
bq.   https://reviews.apache.org/r/4436/diff/1/?file=94295#file94295line493
bq.  
bq.   nit: comment is wrong for change.

I fixed this in an addendum.


bq.  On 2012-03-22 00:25:30, jmhsieh wrote:
bq.   src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java, line 
165
bq.   https://reviews.apache.org/r/4436/diff/1/?file=94300#file94300line165
bq.  
bq.   was: remove all instances of TODO Auto-generated method stub?  
bq.   
bq.   Replace with a comment to say this should be overridden (or throw 
some sort of RuntimeException?)
bq.  
bq.  Michael Stack wrote:
bq.  Its a mock.  I want default, basic behaviors.
bq.  
bq.  jmhsieh wrote:
bq.  I'm not going to push too hard here, but leaving the auto gen stub 
generally tells me incomplete code instead of telling basic behavior or 
not default behavior.

I looked at this this morning again since it reviewers balk.   I tried to come 
up w/ text to put in place of what was auto-generated but anything I conjured 
seemed inauthentic compared to what the machine generated so I am going to just 
leave it (It shouldn't be overridden, at least not currently, not until we need 
the mock to do more, and neither do I want to throw runtime exceptions, etc). 


bq.  On 2012-03-22 00:25:30, jmhsieh wrote:
bq.   src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java, 
line 114
bq.   https://reviews.apache.org/r/4436/diff/1/?file=94303#file94303line114
bq.  
bq.   make these joins have a timeout (prevent hanging tests?)

Done in an addendum.


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4436/#review6188
---


On 2012-03-21 23:18:23, Michael Stack wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4436/
bq.  ---
bq.  
bq.  (Updated 2012-03-21 23:18:23)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  M src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java
bq.Make this class public so its waitForRoot(long) can be used by HMaster.
bq.Remove the stalling waitForRoot no arg.
bq.  M src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
bq.Javadoc.  Add check if stopped flag cycling waiting on assignment.
bq.  M src/main/java/org/apache/hadoop/hbase/master/HMaster.java
bq.Moved check if schema change flag out to a method rather than
bq.have it on tail of constructor.
bq.Moved other initialization stuff like get of assignment manager
bq.and server manager out into methods so could be intercepted by
bq.tests and mocking.
bq.Change how we wait on root so we sleep 100ms at a time and always
bq.check stopped flag rather than block for ever.
bq.Added more checking if stopped flag.
bq.Added flag for when rpc server is up, mostly for tests.
bq.  M src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java
bq.Unused import.
bq.  M src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
bq.Comment.
bq.  M src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
bq.Remove unused code.
bq.  M src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java
bq.Change how we wait on root.  DOn't use removed method.
bq.  A src/test/java/org/apache/hadoop/hbase/master/MockRegionS
bq.  
bq.  
bq.  This addresses bug hbase-5328.
bq.  https://issues.apache.org/jira/browse/hbase-5328
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java 
beaff97 
bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 
d47ef10 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java cd1755f 
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java 
a929e31 
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 7f97b01 
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 
79b6604 
bq.src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java 
90fa45a 
bq.src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 
PRE-CREATION 
bq.src/test/java/org/apache/hadoop/hbase/master/Mocking.java PRE-CREATION 
bq.src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java

[jira] [Commented] (HBASE-5589) Add of the offline call to the Master Interface

2012-03-22 Thread ramkrishna.s.vasudevan (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235617#comment-13235617
 ] 

ramkrishna.s.vasudevan commented on HBASE-5589:
---

@Jon
Sorry.. i saw it in bugs section.  Just now noticed its an improvement.

 Add of the offline call to the Master Interface
 ---

 Key: HBASE-5589
 URL: https://issues.apache.org/jira/browse/HBASE-5589
 Project: HBase
  Issue Type: Improvement
  Components: hbck
Affects Versions: 0.90.6, 0.92.0, 0.94.0, 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0

 Attachments: hbase-5588-0.94.patch, hbase-5589-0.90.patch, 
 hbase-5589-0.92.patch, hbase-5589.patch


 Hbck from HBASE-5128 requires an offline method on the master to properly 
 cleanup state during certain assignment repair operations.  This will this 
 method will be added to recent and older versions of HBase.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5584) Coprocessor hooks can be called in the respective handlers

2012-03-22 Thread ramkrishna.s.vasudevan (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235622#comment-13235622
]

Hadoop QA commented on HBASE-5584:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12519440/HBASE-5584-2.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1256//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1256//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1256//console

This message is automatically generated.

Coprocessor hooks can be called in the respective handlers
--

Key: HBASE-5584
URL: https://issues.apache.org/jira/browse/HBASE-5584
Project: HBase
Issue Type: Improvement
Components: coprocessors
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.96.0

Attachments: HBASE-5584-1.patch, HBASE-5584-2.patch, HBASE-5584.patch

Following points can be changed w.r.t to coprocessors
- Call preCreate, postCreate, preEnable, postEnable, etc. in their
respective handlers
- Currently it is called in the HMaster thus making the postApis async w.r.t
the handlers
- Similar is the case with the balancer.
with current behaviour once we are in the postEnable(for eg) we any way need
to wait for the main enable handler to
be completed.
We should ensure that we dont wait in the main thread so again we need to
spawn a thread and wait on that.
On the other hand if the pre and post api is called on the handlers then only
that handler thread will be
used in the pre/post apis
If the above said plan is ok i can prepare a patch for all such related
changes.

[jira] [Created] (HBASE-5616) Make compaction code standalone

2012-03-22 Thread stack (Created) (JIRA)

Make compaction code standalone
---

 Key: HBASE-5616
 URL: https://issues.apache.org/jira/browse/HBASE-5616
 Project: HBase
  Issue Type: Improvement
Reporter: stack
 Attachments: standalone.txt

This is part of hbase-2462.  Make the compaction code standalone so can run it 
independent of hbase.  Will make it easier to profile and try stuff out.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5616) Make compaction code standalone

2012-03-22 Thread stack (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-5616:
-

Attachment: standalone.txt

First cut.  Not finished.

 Make compaction code standalone
 ---

 Key: HBASE-5616
 URL: https://issues.apache.org/jira/browse/HBASE-5616
 Project: HBase
  Issue Type: Improvement
Reporter: stack
 Attachments: standalone.txt


 This is part of hbase-2462.  Make the compaction code standalone so can run 
 it independent of hbase.  Will make it easier to profile and try stuff out.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5615) the master never do balance becauseof balance the parent region


[ 
https://issues.apache.org/jira/browse/HBASE-5615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235624#comment-13235624
 ] 

ramkrishna.s.vasudevan commented on HBASE-5615:
---

+1 on patch for 0.90.  Is this present in other versions also?

 the master never do balance becauseof  balance the parent region
 

 Key: HBASE-5615
 URL: https://issues.apache.org/jira/browse/HBASE-5615
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.7
Reporter: xufeng
Assignee: xufeng
Priority: Critical
 Attachments: HBASE-5615.patch


 the master never do balance becauseof when master do rebuildUserRegions()，it 
 will add the parent region into  AssignmentManager#servers,
 if balancer let the parent region to move,the parent will in RIT forever.thus 
 balance will never be executed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-5617) Provide coprocessor hooks in put flow while rollbackMemstore.

2012-03-22 Thread ramkrishna.s.vasudevan (Created) (JIRA)

Provide coprocessor hooks in put flow while rollbackMemstore.
-

 Key: HBASE-5617
 URL: https://issues.apache.org/jira/browse/HBASE-5617
 Project: HBase
  Issue Type: Improvement
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Fix For: 0.96.0


With coprocessors hooks while put happens we have the provision to create new 
puts to other tables or regions.  These puts can be done with writeToWal as 
false.
In 0.94 and above the puts are first written to memstore and then to WAL.  If 
any failure in the WAL append or sync the memstore is rollbacked.  
Now the problem is that if the put that happens in the main flow fails there is 
no way to rollback the 
puts that happened in the prePut.

We can add coprocessor hooks to like pre/postRoolBackMemStore.  Is any one hook 
enough here?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5617) Provide coprocessor hooks in put flow while rollbackMemstore.

2012-03-22 Thread ramkrishna.s.vasudevan (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-5617:
--

Component/s: coprocessors

 Provide coprocessor hooks in put flow while rollbackMemstore.
 -

 Key: HBASE-5617
 URL: https://issues.apache.org/jira/browse/HBASE-5617
 Project: HBase
  Issue Type: Improvement
  Components: coprocessors
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Fix For: 0.96.0


 With coprocessors hooks while put happens we have the provision to create new 
 puts to other tables or regions.  These puts can be done with writeToWal as 
 false.
 In 0.94 and above the puts are first written to memstore and then to WAL.  If 
 any failure in the WAL append or sync the memstore is rollbacked.  
 Now the problem is that if the put that happens in the main flow fails there 
 is no way to rollback the 
 puts that happened in the prePut.
 We can add coprocessor hooks to like pre/postRoolBackMemStore.  Is any one 
 hook enough here?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5604) HLog replay tool that generates HFiles for use by LoadIncrementalHFiles.

2012-03-22 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235640#comment-13235640
]

Lars Hofhansl commented on HBASE-5604:
--

This is definitely not to replace distributed log splitting as result of a
crash but for dealing with accidentally deleted data.

Relational databases usually support point in time recovery from a backup by
taking periodic baseline backups and archiving the WAL. Upon recovery the base
backup closest before the PIT is used and then the logs are replayed to the
desired to PIT.

Since HBase has not snapshotting, yet, any backup solution will necessary lead
to an inconsistent copy that can only be made consistent by replaying some of
the logs (to cover the duration the backup took).

Log replay in HBase is either slow (standalone client using the highlevel API)
or can only be used for crash recovery (log splitting, because the logs are
split by region names, wouldn't be able to deal with split regions).

This would take the part of log replaying for a thje log replay part in a PITR
scenario.
Look at this as an M/R version of HBASE-3752.

HLog replay tool that generates HFiles for use by LoadIncrementalHFiles.

Key: HBASE-5604
URL: https://issues.apache.org/jira/browse/HBASE-5604
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl

[jira] [Commented] (HBASE-5328) Small changes to Master to make it more testable


[ 
https://issues.apache.org/jira/browse/HBASE-5328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235641#comment-13235641
 ] 

jirapos...@reviews.apache.org commented on HBASE-5328:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4436/#review6223
---

Ship it!


Sorry, meant to have this said this after the first review.

- jmhsieh


On 2012-03-21 23:18:23, Michael Stack wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4436/
bq.  ---
bq.  
bq.  (Updated 2012-03-21 23:18:23)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  M src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java
bq.Make this class public so its waitForRoot(long) can be used by HMaster.
bq.Remove the stalling waitForRoot no arg.
bq.  M src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
bq.Javadoc.  Add check if stopped flag cycling waiting on assignment.
bq.  M src/main/java/org/apache/hadoop/hbase/master/HMaster.java
bq.Moved check if schema change flag out to a method rather than
bq.have it on tail of constructor.
bq.Moved other initialization stuff like get of assignment manager
bq.and server manager out into methods so could be intercepted by
bq.tests and mocking.
bq.Change how we wait on root so we sleep 100ms at a time and always
bq.check stopped flag rather than block for ever.
bq.Added more checking if stopped flag.
bq.Added flag for when rpc server is up, mostly for tests.
bq.  M src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java
bq.Unused import.
bq.  M src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
bq.Comment.
bq.  M src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
bq.Remove unused code.
bq.  M src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java
bq.Change how we wait on root.  DOn't use removed method.
bq.  A src/test/java/org/apache/hadoop/hbase/master/MockRegionS
bq.  
bq.  
bq.  This addresses bug hbase-5328.
bq.  https://issues.apache.org/jira/browse/hbase-5328
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java 
beaff97 
bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 
d47ef10 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java cd1755f 
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java 
a929e31 
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 7f97b01 
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 
79b6604 
bq.src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java 
90fa45a 
bq.src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 
PRE-CREATION 
bq.src/test/java/org/apache/hadoop/hbase/master/Mocking.java PRE-CREATION 
bq.src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 
841649a 
bq.src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java 
PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/4436/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Michael
bq.  
bq.



 Small changes to Master to make it more testable
 

 Key: HBASE-5328
 URL: https://issues.apache.org/jira/browse/HBASE-5328
 Project: HBase
  Issue Type: Task
Reporter: stack
Assignee: stack
 Fix For: 0.96.0

 Attachments: 5328-addendum.txt, 5328.txt, 5328v12.txt, 5328v2.txt, 
 5328v2.txt, 5328v3.txt, 5328v4.txt, 5328v8.txt


 Here are some small changes in Master that make it more testable.  Included 
 tests stand up a Master and then fake it into thinking that three 
 regionservers are registering making master assign root and meta, etc.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-3134) [replication] Add the ability to enable/disable streams


 [ 
https://issues.apache.org/jira/browse/HBASE-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-3134:
-

Fix Version/s: (was: 0.94.0)
   0.94.1

Thanks Teruyoshi, moving it to 0.94.1 for now.

 [replication] Add the ability to enable/disable streams
 ---

 Key: HBASE-3134
 URL: https://issues.apache.org/jira/browse/HBASE-3134
 Project: HBase
  Issue Type: New Feature
  Components: replication
Reporter: Jean-Daniel Cryans
Assignee: Teruyoshi Zenmyo
Priority: Minor
  Labels: replication
 Fix For: 0.94.1

 Attachments: 3134-v2.txt, 3134-v3.txt, 3134.txt, HBASE-3134.patch, 
 HBASE-3134.patch, HBASE-3134.patch, HBASE-3134.patch


 This jira was initially in the scope of HBASE-2201, but was pushed out since 
 it has low value compared to the required effort (and when want to ship 
 0.90.0 rather soonish).
 We need to design a way to enable/disable replication streams in a 
 determinate fashion.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3776) Add Bloom Filter Support to HFileOutputFormat


[ 
https://issues.apache.org/jira/browse/HBASE-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235646#comment-13235646
 ] 

Lars Hofhansl commented on HBASE-3776:
--

Please comment today on why this is a blocker for 0.94. Otherwise I'll move 
this out of 0.94.

 Add Bloom Filter Support to HFileOutputFormat
 -

 Key: HBASE-3776
 URL: https://issues.apache.org/jira/browse/HBASE-3776
 Project: HBase
  Issue Type: Sub-task
Reporter: Nicolas Spiegelberg
Assignee: Jerry Chen
Priority: Blocker
  Labels: hbase
 Fix For: 0.94.0


 Add Bloom Filter support for bulk imports.  Lacking a bloom filter, even on a 
 single imported file, can cause perf degradation.  Since we now set our 
 compression type based on the HBase CF configuration, it would be good to 
 follow this path for the bloom filter addition.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3967) Support deletes in HFileOutputFormat based bulk import mechanism


[ 
https://issues.apache.org/jira/browse/HBASE-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235648#comment-13235648
 ] 

Lars Hofhansl commented on HBASE-3967:
--

Please comment today on why this is a blocker for 0.94. Otherwise I'll move 
this out of 0.94.

 Support deletes in HFileOutputFormat based bulk import mechanism
 

 Key: HBASE-3967
 URL: https://issues.apache.org/jira/browse/HBASE-3967
 Project: HBase
  Issue Type: Sub-task
Reporter: Kannan Muthukkaruppan
Priority: Blocker
 Fix For: 0.94.0

 Attachments: diff.patch


 During bulk imports, it'll be useful to be able to do delete mutations 
 (either to delete data that already exists in HBase or was inserted earlier 
 during this run of the import). 
 For example, we have a use case, where we are processing a log of data which 
 may have both inserts and deletes in the mix and we want to upload that into 
 HBase using the bulk import mechanism.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4914) Enhance MapReduce TableInputFormat to Support N-mappers per Region


[ 
https://issues.apache.org/jira/browse/HBASE-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235650#comment-13235650
 ] 

Lars Hofhansl commented on HBASE-4914:
--

Please comment today on why this is a blocker for 0.94. Otherwise I'll move 
this out of 0.94.

 Enhance MapReduce TableInputFormat to Support N-mappers per Region
 --

 Key: HBASE-4914
 URL: https://issues.apache.org/jira/browse/HBASE-4914
 Project: HBase
  Issue Type: Sub-task
  Components: client, regionserver
Reporter: Nicolas Spiegelberg
Priority: Blocker
 Fix For: 0.94.0


 Current TableInputFormat based MR jobs create exactly one mapper per region 
 where each mapper sets one Scan with appropriate start/stop row keys. This 
 change allows jobs to be run with any number of mappers per region, so that 
 when a mapper fails, there will be less data to be reprocessed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4565) Maven HBase build broken on cygwin with copynativelib.sh call.


 [ 
https://issues.apache.org/jira/browse/HBASE-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-4565:
-

Fix Version/s: (was: 0.94.0)
   0.96.0

Moving out of 0.94, pull back if you feel differently.

 Maven HBase build broken on cygwin with copynativelib.sh call.
 --

 Key: HBASE-4565
 URL: https://issues.apache.org/jira/browse/HBASE-4565
 Project: HBase
  Issue Type: Bug
  Components: build
Affects Versions: 0.92.0
 Environment: cygwin (on xp and win7)
Reporter: Suraj Varma
Assignee: Suraj Varma
  Labels: build, maven
 Fix For: 0.96.0

 Attachments: HBASE-4565-0.92.patch, HBASE-4565-v2.patch, 
 HBASE-4565-v3-0.92.patch, HBASE-4565-v3.patch, HBASE-4565.patch


 This is broken in both 0.92 as well as trunk pom.xml
 Here's a sample maven log snippet from trunk (from Mayuresh on user mailing 
 list)
 [INFO] [antrun:run {execution: package}]
 [INFO] Executing tasks
 main:
[mkdir] Created dir: 
 D:\workspace\mkshirsa\hbase-trunk\target\hbase-0.93-SNAPSHOT\hbase-0.93-SNAPSHOT\lib\native\${build.platform}
 [exec] ls: cannot access D:workspacemkshirsahbase-trunktarget/nativelib: 
 No such file or directory
 [exec] tar (child): Cannot connect to D: resolve failed
 [INFO] 
 
 [ERROR] BUILD ERROR
 [INFO] 
 
 [INFO] An Ant BuildException has occured: exec returned: 3328
 There are two issues: 
 1) The ant run task below doesn't resolve the windows file separator returned 
 by the project.build.directory - this causes the above resolve failed.
 !-- Using Unix cp to preserve symlinks, using script to handle wildcards --
 echo file=${project.build.directory}/copynativelibs.sh
 if [ `ls ${project.build.directory}/nativelib | wc -l` -ne 0]; then
 2) The tar argument value below also has a similar issue in that the path arg 
 doesn't resolve right.
 !-- Using Unix tar to preserve symlinks --
 exec executable=tar failonerror=yes 
 dir=${project.build.directory}/${project.artifactId}-${project.version}
 arg value=czf/
 arg 
 value=/cygdrive/c/workspaces/hbase-0.92-svn/target/${project.artifactId}-${project.version}.tar.gz/
 arg value=./
 /exec
 In both cases, the fix would probably be to use a cross-platform way to 
 handle the directory locations. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4909) Detailed Block Cache Metrics


 [ 
https://issues.apache.org/jira/browse/HBASE-4909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-4909:
-

Fix Version/s: (was: 0.94.0)
   0.96.0

Moving out of 0.94, pull back if you disagree.

 Detailed Block Cache Metrics
 

 Key: HBASE-4909
 URL: https://issues.apache.org/jira/browse/HBASE-4909
 Project: HBase
  Issue Type: Sub-task
  Components: client, regionserver
Reporter: Nicolas Spiegelberg
 Fix For: 0.96.0




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4916) LoadTest MR Job


 [ 
https://issues.apache.org/jira/browse/HBASE-4916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-4916:
-

Fix Version/s: (was: 0.94.0)
   0.96.0

Moving out of 0.94, pull back if you disagree.

 LoadTest MR Job
 ---

 Key: HBASE-4916
 URL: https://issues.apache.org/jira/browse/HBASE-4916
 Project: HBase
  Issue Type: Sub-task
  Components: client, regionserver
Reporter: Nicolas Spiegelberg
Assignee: Karthik Ranganathan
 Fix For: 0.96.0

 Attachments: HBASE-4916.D741.1.patch


 Add a script to start a streaming map-reduce job where each map tasks runs an 
 instance of the load tester for a partition of the key-space. Ensure that the 
 load tester takes a parameter indicating the start key for write operations.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5586) [replication] NPE in ReplicationSource when creating a stream to an inexistent cluster


 [ 
https://issues.apache.org/jira/browse/HBASE-5586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-5586:
-

Attachment: 5586-v2.txt

Patch that does this :)

 [replication] NPE in ReplicationSource when creating a stream to an 
 inexistent cluster
 --

 Key: HBASE-5586
 URL: https://issues.apache.org/jira/browse/HBASE-5586
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.5
Reporter: Jean-Daniel Cryans
Assignee: Jean-Daniel Cryans
 Fix For: 0.90.7, 0.92.2, 0.94.0

 Attachments: 5586-v2.txt, HBASE-5586.java, HBASE-5586.java


 This is from 0.92.1-ish:
 {noformat}
 2012-03-15 09:52:16,589 ERROR
 org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
 Unexpected exception in ReplicationSource, currentPath=null
 java.lang.NullPointerException
at 
 org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.chooseSinks(ReplicationSource.java:223)
at 
 org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.connectToPeers(ReplicationSource.java:442)
at 
 org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:246)
 {noformat}
 I wanted to add a replication stream to a cluster that wasn't existing yet so 
 that the logs would be buffered until then. This should just be treated as if 
 there was no region servers.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5044) Clarify solution for problem described on http://hbase.apache.org/book/trouble.mapreduce.html


 [ 
https://issues.apache.org/jira/browse/HBASE-5044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-5044:
-

Fix Version/s: (was: 0.94.0)
   (was: 0.90.4)
   0.96.0

Hmm, ok, moving to 0.96 then.

 Clarify solution for problem described on 
 http://hbase.apache.org/book/trouble.mapreduce.html
 -

 Key: HBASE-5044
 URL: https://issues.apache.org/jira/browse/HBASE-5044
 Project: HBase
  Issue Type: Improvement
  Components: documentation
Reporter: Eugene Koontz
Assignee: Eugene Koontz
Priority: Trivial
 Fix For: 0.96.0

 Attachments: HBASE-5044.patch


 Add some documentation regarding how to fix the problem described on :
 http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/package-summary.html#classpath
 Should be some text like: 
 {quote}
 You should run your mapreduce job with your {{HADOOP_CLASSPATH}} set to 
 include the HBase jar and HBase's configured classpath. For example 
 (substitute your own hbase jar location for is {{hbase-0.90.0-SNAPSHOT.jar}}):
 {quote}
 {code}
 HADOOP_CLASSPATH=${HBASE_HOME}/target/hbase-0.90.0-SNAPSHOT.jar:`${HBASE_HOME}/bin/hbase
  classpath` ${HADOOP_HOME}/bin/hadoop jar 
 ${HBASE_HOME}/target/hbase-0.90.0-SNAPSHOT.jar rowcounter usertable
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5586) [replication] NPE in ReplicationSource when creating a stream to an inexistent cluster


 [ 
https://issues.apache.org/jira/browse/HBASE-5586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-5586:
-

Status: Patch Available  (was: Open)

 [replication] NPE in ReplicationSource when creating a stream to an 
 inexistent cluster
 --

 Key: HBASE-5586
 URL: https://issues.apache.org/jira/browse/HBASE-5586
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.5
Reporter: Jean-Daniel Cryans
Assignee: Jean-Daniel Cryans
 Fix For: 0.90.7, 0.92.2, 0.94.0

 Attachments: 5586-v2.txt, HBASE-5586.java, HBASE-5586.java


 This is from 0.92.1-ish:
 {noformat}
 2012-03-15 09:52:16,589 ERROR
 org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
 Unexpected exception in ReplicationSource, currentPath=null
 java.lang.NullPointerException
at 
 org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.chooseSinks(ReplicationSource.java:223)
at 
 org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.connectToPeers(ReplicationSource.java:442)
at 
 org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:246)
 {noformat}
 I wanted to add a replication stream to a cluster that wasn't existing yet so 
 that the logs would be buffered until then. This should just be treated as if 
 there was no region servers.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4113) Add createAsync and splits by start and end key to the shell


 [ 
https://issues.apache.org/jira/browse/HBASE-4113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-4113:
-

Fix Version/s: (was: 0.94.0)
   0.96.0

Moving out of 0.94. Pull back if you feel differently.

 Add createAsync and splits by start and end key to the shell
 

 Key: HBASE-4113
 URL: https://issues.apache.org/jira/browse/HBASE-4113
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.92.0
Reporter: Lars George
Priority: Minor
 Fix For: 0.96.0

 Attachments: HBASE-4113-v2.patch, HBASE-4113.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5128) [uber hbck] Enable hbck to automatically repair table integrity problems as well as region consistency problems while online.

[
https://issues.apache.org/jira/browse/HBASE-5128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235667#comment-13235667
]

Lars Hofhansl commented on HBASE-5128:
--

We'll be trying to get this into 0.94.

[uber hbck] Enable hbck to automatically repair table integrity problems as
well as region consistency problems while online.
-

Key: HBASE-5128
URL: https://issues.apache.org/jira/browse/HBASE-5128
Project: HBase
Issue Type: New Feature
Components: hbck
Affects Versions: 0.90.5, 0.92.0, 0.94.0, 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0

Attachments: hbase-5128-0.90-v2.patch, hbase-5128-0.90-v2b.patch,
hbase-5128-0.92-v2.patch, hbase-5128-0.94-v2.patch,
hbase-5128-trunk-v2.patch, hbase-5128-trunk.patch

The current (0.90.5, 0.92.0rc2) versions of hbck detects most of region
consistency and table integrity invariant violations. However with '-fix' it
can only automatically repair region consistency cases having to do with
deployment problems. This updated version should be able to handle all cases
(including a new orphan regiondir case). When complete will likely deprecate
the OfflineMetaRepair tool and subsume several open META-hole related issue.
Here's the approach (from the comment of at the top of the new version of the
file).
{code}
/**
* HBaseFsck (hbck) is a tool for checking and repairing region consistency
and
* table integrity.
*
* Region consistency checks verify that META, region deployment on
* region servers and the state of data in HDFS (.regioninfo files) all are in
* accordance.
*
* Table integrity checks verify that that all possible row keys can resolve
to
* exactly one region of a table. This means there are no individual
degenerate
* or backwards regions; no holes between regions; and that there no
overlapping
* regions.
*
* The general repair strategy works in these steps.
* 1) Repair Table Integrity on HDFS. (merge or fabricate regions)
* 2) Repair Region Consistency with META and assignments
*
* For table integrity repairs, the tables their region directories are
scanned
* for .regioninfo files. Each table's integrity is then verified. If there
* are any orphan regions (regions with no .regioninfo files), or holes, new
* regions are fabricated. Backwards regions are sidelined as well as empty
* degenerate (endkey==startkey) regions. If there are any overlapping
regions,
* a new region is created and all data is merged into the new region.
*
* Table integrity repairs deal solely with HDFS and can be done offline --
the
* hbase region servers or master do not need to be running. These phase can
be
* use to completely reconstruct the META table in an offline fashion.
*
* Region consistency requires three conditions -- 1) valid .regioninfo file
* present in an hdfs region dir, 2) valid row with .regioninfo data in META,
* and 3) a region is deployed only at the regionserver that is was assigned
to.
*
* Region consistency requires hbck to contact the HBase master and region
* servers, so the connect() must first be called successfully. Much of the
* region consistency information is transient and less risky to repair.
*/
{code}

[jira] [Assigned] (HBASE-4657) Improve the efficiency of our MR jobs with a few configurations

2012-03-22 Thread Lars Hofhansl (Assigned) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl reassigned HBASE-4657:


Assignee: Lars Hofhansl

 Improve the efficiency of our MR jobs with a few configurations
 ---

 Key: HBASE-4657
 URL: https://issues.apache.org/jira/browse/HBASE-4657
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.90.4
Reporter: Jean-Daniel Cryans
Assignee: Lars Hofhansl
 Fix For: 0.94.0


 This is a low hanging fruit, some of our MR jobs like RowCounter and 
 CopyTable don't even setCacheBlocks on the scan object which out of the box 
 completely screws up a running system. Another thing would be to disable 
 speculative execution.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4657) Improve the efficiency of our MR jobs with a few configurations


[ 
https://issues.apache.org/jira/browse/HBASE-4657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235669#comment-13235669
 ] 

Lars Hofhansl commented on HBASE-4657:
--

I'll do this today. Seems silly not to do this for a performance release.

 Improve the efficiency of our MR jobs with a few configurations
 ---

 Key: HBASE-4657
 URL: https://issues.apache.org/jira/browse/HBASE-4657
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.90.4
Reporter: Jean-Daniel Cryans
Assignee: Lars Hofhansl
 Fix For: 0.94.0


 This is a low hanging fruit, some of our MR jobs like RowCounter and 
 CopyTable don't even setCacheBlocks on the scan object which out of the box 
 completely screws up a running system. Another thing would be to disable 
 speculative execution.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4720) Implement atomic update operations (checkAndPut, checkAndDelete) for REST client/server


[ 
https://issues.apache.org/jira/browse/HBASE-4720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235668#comment-13235668
 ] 

Lars Hofhansl commented on HBASE-4720:
--

Needs to be reviewed today to get into 0.94.0. Can still aim for 0.94.1.

 Implement atomic update operations (checkAndPut, checkAndDelete) for REST 
 client/server 
 

 Key: HBASE-4720
 URL: https://issues.apache.org/jira/browse/HBASE-4720
 Project: HBase
  Issue Type: Improvement
Reporter: Daniel Lord
Assignee: Mubarak Seyed
 Fix For: 0.94.0

 Attachments: HBASE-4720.trunk.v1.patch, HBASE-4720.trunk.v2.patch, 
 HBASE-4720.trunk.v3.patch, HBASE-4720.trunk.v4.patch, 
 HBASE-4720.trunk.v5.patch, HBASE-4720.trunk.v6.patch, 
 HBASE-4720.trunk.v7.patch, HBASE-4720.v1.patch, HBASE-4720.v3.patch


 I have several large application/HBase clusters where an application node 
 will occasionally need to talk to HBase from a different cluster.  In order 
 to help ensure some of my consistency guarantees I have a sentinel table that 
 is updated atomically as users interact with the system.  This works quite 
 well for the regular hbase client but the REST client does not implement 
 the checkAndPut and checkAndDelete operations.  This exposes the application 
 to some race conditions that have to be worked around.  It would be ideal if 
 the same checkAndPut/checkAndDelete operations could be supported by the REST 
 client.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5615) the master never do balance becauseof balance the parent region

2012-03-22 Thread Zhihong Yu (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235674#comment-13235674
 ] 

Zhihong Yu commented on HBASE-5615:
---

+1 as well.

Please fix indentation:
{code}
   String tableName = regionInfo.getTableDesc().getNameAsString();
+ if (regionInfo.isOffline()  regionInfo.isSplit()) continue;
{code}

 the master never do balance becauseof  balance the parent region
 

 Key: HBASE-5615
 URL: https://issues.apache.org/jira/browse/HBASE-5615
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.7
Reporter: xufeng
Assignee: xufeng
Priority: Critical
 Attachments: HBASE-5615.patch


 the master never do balance becauseof when master do rebuildUserRegions()，it 
 will add the parent region into  AssignmentManager#servers,
 if balancer let the parent region to move,the parent will in RIT forever.thus 
 balance will never be executed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5548) Add ability to get a table in the shell

2012-03-22 Thread Jesse Yates (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235683#comment-13235683
]

Jesse Yates commented on HBASE-5548:

bq. You are doing call by String methodname. It makes it all a bit more brittle

Indeed, but its how we do everything else in the shell, so I didn't think it
would be so bad.

bq. Whats an exception look like if you pass an unknown 'command'. Is it
intelligible?

Ends up spitting out this:
{code}
NoMethodError: undefined method `new' for nil:NilClass
{code}
So not really - I'll clean it up.

bq. Needs lots of comments explaining whats going on.

I'll work on it - anything in particular odd?

Add ability to get a table in the shell
---

Key: HBASE-5548
URL: https://issues.apache.org/jira/browse/HBASE-5548
Project: HBase
Issue Type: Improvement
Components: shell
Reporter: Jesse Yates
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.1

Attachments: ruby_HBASE-5528-v0.patch, ruby_HBASE-5548-v1.patch,
ruby_HBASE-5548-v2.patch

Currently, all the commands that operate on a table in the shell first have
to take the table as name as input.
There are two main considerations:
* It is annoying to have to write the table name every time, when you should
just be able to get a reference to a table
* the current implementation is very wasteful - it creates a new HTable for
each call (but reuses the connection since it uses the same configuration)
We should be able to get a handle to a single HTable and then operate on that.

[jira] [Commented] (HBASE-5586) [replication] NPE in ReplicationSource when creating a stream to an inexistent cluster

[
https://issues.apache.org/jira/browse/HBASE-5586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235696#comment-13235696
]

Hadoop QA commented on HBASE-5586:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12519453/5586-v2.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:

org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
org.apache.hadoop.hbase.mapreduce.TestImportTsv
org.apache.hadoop.hbase.mapred.TestTableMapReduce
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1257//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1257//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1257//console

This message is automatically generated.

[replication] NPE in ReplicationSource when creating a stream to an
inexistent cluster
--

Key: HBASE-5586
URL: https://issues.apache.org/jira/browse/HBASE-5586
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.5
Reporter: Jean-Daniel Cryans
Assignee: Jean-Daniel Cryans
Fix For: 0.90.7, 0.92.2, 0.94.0

Attachments: 5586-v2.txt, HBASE-5586.java, HBASE-5586.java

This is from 0.92.1-ish:
{noformat}
2012-03-15 09:52:16,589 ERROR
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
Unexpected exception in ReplicationSource, currentPath=null
java.lang.NullPointerException
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.chooseSinks(ReplicationSource.java:223)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.connectToPeers(ReplicationSource.java:442)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:246)
{noformat}
I wanted to add a replication stream to a cluster that wasn't existing yet so
that the logs would be buffered until then. This should just be treated as if
there was no region servers.

[jira] [Created] (HBASE-5618) SplitLogManager - prevent unnecessary attempts to resubmits

2012-03-22 Thread Prakash Khemani (Created) (JIRA)

SplitLogManager - prevent unnecessary attempts to resubmits
---

 Key: HBASE-5618
 URL: https://issues.apache.org/jira/browse/HBASE-5618
 Project: HBase
  Issue Type: Improvement
Reporter: Prakash Khemani


Currently once a watch fires that the task node has been updated (hearbeated) 
by the worker, the splitlogmanager still quite some time before it updates the 
last heard from time. This is because the manager currently schedules another 
getDataSetWatch() and only after that finishes will it update the task's last 
heard from time.

This leads to a large number of zk-BadVersion warnings when resubmission is 
continuously attempted and it fails.


Two changes should be made
(1) On a resubmission failure because of BadVersion the task's lastUpdate time 
should get upped.
(2) The task's lastUpdate time should get upped as soon as the 
nodeDataChanged() watch fires and without waiting for getDataSetWatch() to 
complete.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4932) Block cache can be mistakenly instantiated by tools


 [ 
https://issues.apache.org/jira/browse/HBASE-4932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-4932:
-

   Resolution: Fixed
Fix Version/s: 0.96.0
   Status: Resolved  (was: Patch Available)

Ah, NM, was already committed, just not marked fixed.

 Block cache can be mistakenly instantiated by tools
 ---

 Key: HBASE-4932
 URL: https://issues.apache.org/jira/browse/HBASE-4932
 Project: HBase
  Issue Type: Bug
Reporter: Prakash Khemani
Assignee: Prakash Khemani
 Fix For: 0.94.0, 0.96.0

 Attachments: HBASE-4932.patch


 Map Reduce tasks that create a writer to write HFiles inadvertently end up 
 creating block cache.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4910) thrift scannerstopwithfilter not honoring stop row


 [ 
https://issues.apache.org/jira/browse/HBASE-4910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-4910:
-

Fix Version/s: (was: 0.94.0)
   0.96.0

Moving out of 0.94. Pull back if you disagree.

 thrift scannerstopwithfilter not honoring stop row
 --

 Key: HBASE-4910
 URL: https://issues.apache.org/jira/browse/HBASE-4910
 Project: HBase
  Issue Type: Sub-task
  Components: client, regionserver
Reporter: Nicolas Spiegelberg
 Fix For: 0.96.0




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-3577) enables Thrift client to get the Region location

2012-03-22 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-3577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-3577:
-

Fix Version/s: (was: 0.92.2)
   (was: 0.94.0)
   0.96.0

Moving to 0.96.

 enables Thrift client to get the Region location
 

 Key: HBASE-3577
 URL: https://issues.apache.org/jira/browse/HBASE-3577
 Project: HBase
  Issue Type: Improvement
  Components: thrift
Reporter: Kazuki Ohta
 Fix For: 0.96.0

 Attachments: HBASE3577-1.patch, HBASE3577-2.patch


 The current thrift interface has the getTableRegions() interface like below.
 {code}
   listTRegionInfo getTableRegions(
 /** table name */
 1:Text tableName)
 throws (1:IOError io)
 {code}
 {code}
 struct TRegionInfo {
   1:Text startKey,
   2:Text endKey,
   3:i64 id,
   4:Text name,
   5:byte version
 }
 {code}
 But the method don't have the region location information (where the region 
 is located).
 I want to add the Thrift interfaces like below in HTable.java.
 {code}
 public MapHRegionInfo, HServerAddress getRegionsInfo() throws IOException
 {code}
 {code}
 public HRegionLocation getRegionLocation(final String row)
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5128) [uber hbck] Enable hbck to automatically repair table integrity problems as well as region consistency problems while online.


[ 
https://issues.apache.org/jira/browse/HBASE-5128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235719#comment-13235719
 ] 

jirapos...@reviews.apache.org commented on HBASE-5128:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4280/#review6224
---



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4280/#comment13407

Please log some information about this region



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4280/#comment13408

Redundant 'with'



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4280/#comment13409

'reassigned' - 'reassign'



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4280/#comment13410

It would be nice to create method so that this block of code and lines 
1271-1289 can be unified.



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4280/#comment13411

Please remove unused code.



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4280/#comment13413

(inMeta  inHdfs) appears more than once above, is there a chance that 
this case mistakenly falls into one of them ?



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4280/#comment13414

Looking at fixDupeAssignment(), it really does region closing and offlining.
Can we give it a better name ?



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4280/#comment13415

checkRegionChain() is synchronous.
Can we share one TableIntegrityErrorHandler and set its tInfo in the loop ?



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4280/#comment13416

This would eclipse the global counter, right ?



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4280/#comment13420

This class can be private



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4280/#comment13421

This class can be private



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4280/#comment13422

The following four lines are repeated 3 times in this class.
Refactor and create a new method.


- Ted


On 2012-03-21 23:24:13, jmhsieh wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4280/
bq.  ---
bq.  
bq.  (Updated 2012-03-21 23:24:13)
bq.  
bq.  
bq.  Review request for hbase, Todd Lipcon, Ted Yu, and Lars Hofhansl.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This version is similar to the 0.90.x version posted a few months back, 
but has a few new features and some minor differences.
bq.  
bq.  1) No trackHTD method needed since we can read from the file system.
bq.  2) Added safeguards to prevent mega merges, and to isolate repairs to 
particular tables.
bq.  3) Fixed comparator in HRegionInfo
bq.  4) Fixed TestRegionObserverInterface so that it doesn't rely on bug in 
HRegionInfo comparator.
bq.  
bq.  I'll backport to 0.94/0.92 (which should be very similar) and update the 
0.90 versions after this patch has mostly cleared.
bq.  
bq.  This version is not perfect (there are definitely cases not covered) but 
it think it is worth trying to get this in so that future reviews are more 
manageable.
bq.  
bq.  
bq.  This addresses bug HBASE-5128.
bq.  https://issues.apache.org/jira/browse/HBASE-5128
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 3c635d4 
bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 
d47ef10 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java cd1755f 
bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java c0aaf65 
bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java 5916d9c 
bq.src/main/java/org/apache/hadoop/hbase/util/hbck/OfflineMetaRepair.java 
d57bb6b 
bq.
src/main/java/org/apache/hadoop/hbase/util/hbck/TableIntegrityErrorHandler.java 
PRE-CREATION 
bq.
src/main/java/org/apache/hadoop/hbase/util/hbck/TableIntegrityErrorHandlerImpl.java
 PRE-CREATION 
bq.src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java d9a2a02 
bq.src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java 937781d 
bq.src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsckComparator.java 
0599da1 
bq.

[jira] [Updated] (HBASE-4802) Disable show table metrics in bulk loader

2012-03-22 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-4802:
-

Fix Version/s: (was: 0.94.0)
   0.96.0

Moving out of 0.94. Pull back if you disagree.

 Disable show table metrics in bulk loader
 -

 Key: HBASE-4802
 URL: https://issues.apache.org/jira/browse/HBASE-4802
 Project: HBase
  Issue Type: Bug
Reporter: Nicolas Spiegelberg
Assignee: Liyin Tang
Priority: Trivial
 Fix For: 0.96.0

 Attachments: HBASE-4802.patch


 During bulk load, the Configuration object may be set to null.  This caused 
 an NPE in per-CF metrics because it consults the Configuration to determine 
 whether to show the Table name.  Need to add simple change to allow the conf 
 to be null  not specify table name in that instance.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5128) [uber hbck] Enable hbck to automatically repair table integrity problems as well as region consistency problems while online.


[ 
https://issues.apache.org/jira/browse/HBASE-5128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235727#comment-13235727
 ] 

jirapos...@reviews.apache.org commented on HBASE-5128:
--



bq.  On 2012-03-22 05:21:28, Ted Yu wrote:
bq.   src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 172
bq.   https://reviews.apache.org/r/4280/diff/2/?file=94413#file94413line172
bq.  
bq.   I think tablesToFix would be a better name for this member.

agreed.


bq.  On 2012-03-22 05:21:28, Ted Yu wrote:
bq.   src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 186
bq.   https://reviews.apache.org/r/4280/diff/2/?file=94413#file94413line186
bq.  
bq.   'encoded region name' would be clearer.

It maps from encoded region name to HbckInfo structure. 


bq.  On 2012-03-22 05:21:28, Ted Yu wrote:
bq.   src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 198
bq.   https://reviews.apache.org/r/4280/diff/2/?file=94413#file94413line198
bq.  
bq.   TInfo should be TableInfo

done


bq.  On 2012-03-22 05:21:28, Ted Yu wrote:
bq.   src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 363
bq.   https://reviews.apache.org/r/4280/diff/2/?file=94413#file94413line363
bq.  
bq.   fixes is a global variable.
bq.   I think the loop condition should check that fixes increases across 
iterations.
bq.   If the count doesn't increase, we can break out of the loop.

clearState() reset's the fixes count.  I'll add a comment.


bq.  On 2012-03-22 05:21:28, Ted Yu wrote:
bq.   src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 365
bq.   https://reviews.apache.org/r/4280/diff/2/?file=94413#file94413line365
bq.  
bq.   Why is 2 specially treated here ?

iteration 1 does repairs, iteration 2 verifies things are fixed.  If there are 
more something funny has happened.  adding comment.

Changed success logging message to info.


bq.  On 2012-03-22 05:21:28, Ted Yu wrote:
bq.   src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 396
bq.   https://reviews.apache.org/r/4280/diff/2/?file=94413#file94413line396
bq.  
bq.   Ideally a different return value (say -2) should be used.

done


- jmhsieh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4280/#review6205
---


On 2012-03-21 23:24:13, jmhsieh wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4280/
bq.  ---
bq.  
bq.  (Updated 2012-03-21 23:24:13)
bq.  
bq.  
bq.  Review request for hbase, Todd Lipcon, Ted Yu, and Lars Hofhansl.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This version is similar to the 0.90.x version posted a few months back, 
but has a few new features and some minor differences.
bq.  
bq.  1) No trackHTD method needed since we can read from the file system.
bq.  2) Added safeguards to prevent mega merges, and to isolate repairs to 
particular tables.
bq.  3) Fixed comparator in HRegionInfo
bq.  4) Fixed TestRegionObserverInterface so that it doesn't rely on bug in 
HRegionInfo comparator.
bq.  
bq.  I'll backport to 0.94/0.92 (which should be very similar) and update the 
0.90 versions after this patch has mostly cleared.
bq.  
bq.  This version is not perfect (there are definitely cases not covered) but 
it think it is worth trying to get this in so that future reviews are more 
manageable.
bq.  
bq.  
bq.  This addresses bug HBASE-5128.
bq.  https://issues.apache.org/jira/browse/HBASE-5128
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 3c635d4 
bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 
d47ef10 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java cd1755f 
bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java c0aaf65 
bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java 5916d9c 
bq.src/main/java/org/apache/hadoop/hbase/util/hbck/OfflineMetaRepair.java 
d57bb6b 
bq.
src/main/java/org/apache/hadoop/hbase/util/hbck/TableIntegrityErrorHandler.java 
PRE-CREATION 
bq.
src/main/java/org/apache/hadoop/hbase/util/hbck/TableIntegrityErrorHandlerImpl.java
 PRE-CREATION 
bq.src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java d9a2a02 
bq.src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java 937781d 
bq.src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsckComparator.java 
0599da1 
bq.src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java 
dbb97f8 
bq.
src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildBase.java 
2b4cac8 
bq.

[jira] [Commented] (HBASE-4607) Split log worker should terminate properly when waiting for znode


[ 
https://issues.apache.org/jira/browse/HBASE-4607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235736#comment-13235736
 ] 

Lars Hofhansl commented on HBASE-4607:
--

+1 on v2 even though most of it are whitespace changes.

 Split log worker should terminate properly when waiting for znode
 -

 Key: HBASE-4607
 URL: https://issues.apache.org/jira/browse/HBASE-4607
 Project: HBase
  Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Priority: Minor
 Fix For: 0.94.0

 Attachments: 
 HBASE-4607_SplitLogWorker_should_correct-20111017231456-47a82ef3.patch


 This is an attempt to fix the fact that SplitLogWorker threads are not being 
 terminated properly in some unit tests. This probably does not happen in 
 production because the master always creates the log-splitting ZK node, but 
 it does happen in 89-fb. Thanks to Prakash Khemani for help on this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5565) Refactoring doMiniBatchPut()


[ 
https://issues.apache.org/jira/browse/HBASE-5565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235739#comment-13235739
 ] 

Scott Chen commented on HBASE-5565:
---

@Lars:
Feel free to assign this back to me if you don't have time for this.
I will be more than happy to review your patch also.

 Refactoring doMiniBatchPut()
 

 Key: HBASE-5565
 URL: https://issues.apache.org/jira/browse/HBASE-5565
 Project: HBase
  Issue Type: Improvement
Reporter: Scott Chen
Assignee: Lars Hofhansl
 Fix For: 0.96.0


 HBASE-5542 introduce a RowProcessor interface that can handle multi-row 
 transaction.
 It will make the code cleaner if we can make doMiniBatchPut() to be covered 
 by RowProcessor.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5613) ThriftServer getTableRegions does not return serverName and port

2012-03-22 Thread Scott Chen (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Chen updated HBASE-5613:
--

Status: Patch Available  (was: Open)

 ThriftServer getTableRegions does not return serverName and port
 

 Key: HBASE-5613
 URL: https://issues.apache.org/jira/browse/HBASE-5613
 Project: HBase
  Issue Type: Bug
  Components: thrift
Reporter: Scott Chen
Assignee: Scott Chen
Priority: Minor
 Attachments: HBASE-5613.D2403.1.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5586) [replication] NPE in ReplicationSource when creating a stream to an inexistent cluster

2012-03-22 Thread Jean-Daniel Cryans (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Daniel Cryans updated HBASE-5586:
--

Attachment: HBASE-5586-trunk.patch

Thanks for the heavy lifting Lars.

Here's a patch for trunk, I set the test to Medium since it uses a ZK cluster, 
added a teardown, and a missing license.

 [replication] NPE in ReplicationSource when creating a stream to an 
 inexistent cluster
 --

 Key: HBASE-5586
 URL: https://issues.apache.org/jira/browse/HBASE-5586
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.5
Reporter: Jean-Daniel Cryans
Assignee: Jean-Daniel Cryans
 Fix For: 0.90.7, 0.92.2, 0.94.0

 Attachments: 5586-v2.txt, HBASE-5586-trunk.patch, HBASE-5586.java, 
 HBASE-5586.java


 This is from 0.92.1-ish:
 {noformat}
 2012-03-15 09:52:16,589 ERROR
 org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
 Unexpected exception in ReplicationSource, currentPath=null
 java.lang.NullPointerException
at 
 org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.chooseSinks(ReplicationSource.java:223)
at 
 org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.connectToPeers(ReplicationSource.java:442)
at 
 org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:246)
 {noformat}
 I wanted to add a replication stream to a cluster that wasn't existing yet so 
 that the logs would be buffered until then. This should just be treated as if 
 there was no region servers.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5443) Add PB-based calls to HRegionInterface

2012-03-22 Thread Jimmy Xiang (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235780#comment-13235780
 ] 

Jimmy Xiang commented on HBASE-5443:


I have done some code changes, and some tests failed.  It is very hard to look 
into them.  So I'd like to break it into small pieces and tag them one by one.

 Add PB-based calls to HRegionInterface
 --

 Key: HBASE-5443
 URL: https://issues.apache.org/jira/browse/HBASE-5443
 Project: HBase
  Issue Type: Sub-task
  Components: ipc, master, migration, regionserver
Reporter: Todd Lipcon
Assignee: Jimmy Xiang
 Fix For: 0.96.0

 Attachments: region_java-proto-mapping.pdf




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5443) Add PB-based calls to HRegionInterface

2012-03-22 Thread Jimmy Xiang (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang updated HBASE-5443:
---

Issue Type: Task  (was: Sub-task)
Parent: (was: HBASE-5305)

 Add PB-based calls to HRegionInterface
 --

 Key: HBASE-5443
 URL: https://issues.apache.org/jira/browse/HBASE-5443
 Project: HBase
  Issue Type: Task
  Components: ipc, master, migration, regionserver
Reporter: Todd Lipcon
Assignee: Jimmy Xiang
 Fix For: 0.96.0

 Attachments: region_java-proto-mapping.pdf




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-5619) Create PB protocols for HRegionInterface

2012-03-22 Thread Jimmy Xiang (Created) (JIRA)

Create PB protocols for HRegionInterface


 Key: HBASE-5619
 URL: https://issues.apache.org/jira/browse/HBASE-5619
 Project: HBase
  Issue Type: Sub-task
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0


Subtask of HBase-5443, separate HRegionInterface into admin protocol and client 
protocol, create the PB protocol buffer files

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5222) Stopping replication via the stop_replication command in hbase shell on a slave cluster isn't acknowledged in the replication sink

2012-03-22 Thread Jean-Daniel Cryans (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235795#comment-13235795
 ] 

Jean-Daniel Cryans commented on HBASE-5222:
---

{{stop_replication}} is a kill switch that should normally kill everything 
that's related to replication. In this case, it's not stopping the region 
servers from accepting incoming replication traffic.

 Stopping replication via the stop_replication command in hbase shell on a 
 slave cluster isn't acknowledged in the replication sink
 

 Key: HBASE-5222
 URL: https://issues.apache.org/jira/browse/HBASE-5222
 Project: HBase
  Issue Type: Bug
  Components: replication, shell
Affects Versions: 0.90.4
Reporter: Josh Wymer

 After running stop_replication in the hbase shell on our slave cluster we 
 saw replication continue for weeks. Turns out that the replication sink is 
 missing a check to get the replication state and therefore continued to write.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5542) Unify HRegion.mutateRowsWithLocks() and HRegion.processRow()


 [ 
https://issues.apache.org/jira/browse/HBASE-5542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-5542:
-

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Committed to trunk.
Forgot new and deleted files in first commit.

Scott, could you please sync trunk and double check and I fixed it all 
correctly with my 2nd commit?

 Unify HRegion.mutateRowsWithLocks() and HRegion.processRow()
 

 Key: HBASE-5542
 URL: https://issues.apache.org/jira/browse/HBASE-5542
 Project: HBase
  Issue Type: Improvement
Reporter: Scott Chen
Assignee: Scott Chen
 Fix For: 0.96.0

 Attachments: HBASE-5542.2.txt, HBASE-5542.3.txt, HBASE-5542.4.txt, 
 HBASE-5542.4.txt, HBASE-5542.4.txt, HBASE-5542.D2217.1.patch, 
 HBASE-5542.D2217.10.patch, HBASE-5542.D2217.11.patch, 
 HBASE-5542.D2217.12.patch, HBASE-5542.D2217.13.patch, 
 HBASE-5542.D2217.14.patch, HBASE-5542.D2217.15.patch, 
 HBASE-5542.D2217.2.patch, HBASE-5542.D2217.3.patch, HBASE-5542.D2217.4.patch, 
 HBASE-5542.D2217.5.patch, HBASE-5542.D2217.6.patch, HBASE-5542.D2217.7.patch, 
 HBASE-5542.D2217.8.patch, HBASE-5542.D2217.9.patch, HBASE-5542.txt


 mutateRowsWithLocks() does atomic mutations on multiple rows.
 processRow() does atomic read-modify-writes on a single row.
 It will be useful to generalize both and have a
 processRowsWithLocks() that does atomic read-modify-writes on multiple rows.
 This also helps reduce some redundancy in the codes.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4814) Starting an online alter when regions are splitting can leave their daughters unaltered


 [ 
https://issues.apache.org/jira/browse/HBASE-4814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-4814:
-

Fix Version/s: (was: 0.94.0)
   0.96.0

I don't think this is going to happen by tomorrow :)

 Starting an online alter when regions are splitting can leave their daughters 
 unaltered
 ---

 Key: HBASE-4814
 URL: https://issues.apache.org/jira/browse/HBASE-4814
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
 Fix For: 0.96.0


 I've seen a situation where regions were splitting almost exactly at the same 
 time as an alter command was issued and those regions' daughters were left 
 unaltered. It would even seem that the daughters' daughters also share this 
 situation.
 Reopening all the regions fixes the problem.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-5620) Convert the client protocol of HRegionInterface to PB

2012-03-22 Thread Jimmy Xiang (Created) (JIRA)

Convert the client protocol of HRegionInterface to PB
-

 Key: HBASE-5620
 URL: https://issues.apache.org/jira/browse/HBASE-5620
 Project: HBase
  Issue Type: Sub-task
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5542) Unify HRegion.mutateRowsWithLocks() and HRegion.processRow()


[ 
https://issues.apache.org/jira/browse/HBASE-5542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235807#comment-13235807
 ] 

Scott Chen commented on HBASE-5542:
---

@Lars: No problem. Let me do that right away.

 Unify HRegion.mutateRowsWithLocks() and HRegion.processRow()
 

 Key: HBASE-5542
 URL: https://issues.apache.org/jira/browse/HBASE-5542
 Project: HBase
  Issue Type: Improvement
Reporter: Scott Chen
Assignee: Scott Chen
 Fix For: 0.96.0

 Attachments: HBASE-5542.2.txt, HBASE-5542.3.txt, HBASE-5542.4.txt, 
 HBASE-5542.4.txt, HBASE-5542.4.txt, HBASE-5542.D2217.1.patch, 
 HBASE-5542.D2217.10.patch, HBASE-5542.D2217.11.patch, 
 HBASE-5542.D2217.12.patch, HBASE-5542.D2217.13.patch, 
 HBASE-5542.D2217.14.patch, HBASE-5542.D2217.15.patch, 
 HBASE-5542.D2217.2.patch, HBASE-5542.D2217.3.patch, HBASE-5542.D2217.4.patch, 
 HBASE-5542.D2217.5.patch, HBASE-5542.D2217.6.patch, HBASE-5542.D2217.7.patch, 
 HBASE-5542.D2217.8.patch, HBASE-5542.D2217.9.patch, HBASE-5542.txt


 mutateRowsWithLocks() does atomic mutations on multiple rows.
 processRow() does atomic read-modify-writes on a single row.
 It will be useful to generalize both and have a
 processRowsWithLocks() that does atomic read-modify-writes on multiple rows.
 This also helps reduce some redundancy in the codes.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5542) Unify HRegion.mutateRowsWithLocks() and HRegion.processRow()


[ 
https://issues.apache.org/jira/browse/HBASE-5542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235808#comment-13235808
 ] 

Scott Chen commented on HBASE-5542:
---

Thanks for committing this :)

 Unify HRegion.mutateRowsWithLocks() and HRegion.processRow()
 

 Key: HBASE-5542
 URL: https://issues.apache.org/jira/browse/HBASE-5542
 Project: HBase
  Issue Type: Improvement
Reporter: Scott Chen
Assignee: Scott Chen
 Fix For: 0.96.0

 Attachments: HBASE-5542.2.txt, HBASE-5542.3.txt, HBASE-5542.4.txt, 
 HBASE-5542.4.txt, HBASE-5542.4.txt, HBASE-5542.D2217.1.patch, 
 HBASE-5542.D2217.10.patch, HBASE-5542.D2217.11.patch, 
 HBASE-5542.D2217.12.patch, HBASE-5542.D2217.13.patch, 
 HBASE-5542.D2217.14.patch, HBASE-5542.D2217.15.patch, 
 HBASE-5542.D2217.2.patch, HBASE-5542.D2217.3.patch, HBASE-5542.D2217.4.patch, 
 HBASE-5542.D2217.5.patch, HBASE-5542.D2217.6.patch, HBASE-5542.D2217.7.patch, 
 HBASE-5542.D2217.8.patch, HBASE-5542.D2217.9.patch, HBASE-5542.txt


 mutateRowsWithLocks() does atomic mutations on multiple rows.
 processRow() does atomic read-modify-writes on a single row.
 It will be useful to generalize both and have a
 processRowsWithLocks() that does atomic read-modify-writes on multiple rows.
 This also helps reduce some redundancy in the codes.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-5621) Convert admin protocol of HRegionInterface to PB

2012-03-22 Thread Jimmy Xiang (Created) (JIRA)

Convert admin protocol of HRegionInterface to PB


 Key: HBASE-5621
 URL: https://issues.apache.org/jira/browse/HBASE-5621
 Project: HBase
  Issue Type: Sub-task
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5542) Unify HRegion.mutateRowsWithLocks() and HRegion.processRow()


[ 
https://issues.apache.org/jira/browse/HBASE-5542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235821#comment-13235821
 ] 

Scott Chen commented on HBASE-5542:
---

@Lars: Synced and run TestAtomicOperation and TestRowProcessorEndpoint. It 
works fine. Thanks!

 Unify HRegion.mutateRowsWithLocks() and HRegion.processRow()
 

 Key: HBASE-5542
 URL: https://issues.apache.org/jira/browse/HBASE-5542
 Project: HBase
  Issue Type: Improvement
Reporter: Scott Chen
Assignee: Scott Chen
 Fix For: 0.96.0

 Attachments: HBASE-5542.2.txt, HBASE-5542.3.txt, HBASE-5542.4.txt, 
 HBASE-5542.4.txt, HBASE-5542.4.txt, HBASE-5542.D2217.1.patch, 
 HBASE-5542.D2217.10.patch, HBASE-5542.D2217.11.patch, 
 HBASE-5542.D2217.12.patch, HBASE-5542.D2217.13.patch, 
 HBASE-5542.D2217.14.patch, HBASE-5542.D2217.15.patch, 
 HBASE-5542.D2217.2.patch, HBASE-5542.D2217.3.patch, HBASE-5542.D2217.4.patch, 
 HBASE-5542.D2217.5.patch, HBASE-5542.D2217.6.patch, HBASE-5542.D2217.7.patch, 
 HBASE-5542.D2217.8.patch, HBASE-5542.D2217.9.patch, HBASE-5542.txt


 mutateRowsWithLocks() does atomic mutations on multiple rows.
 processRow() does atomic read-modify-writes on a single row.
 It will be useful to generalize both and have a
 processRowsWithLocks() that does atomic read-modify-writes on multiple rows.
 This also helps reduce some redundancy in the codes.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5613) ThriftServer getTableRegions does not return serverName and port

[
https://issues.apache.org/jira/browse/HBASE-5613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235822#comment-13235822
]

Hadoop QA commented on HBASE-5613:
--

-1 overall. Here are the results of testing the latest attachment

http://issues.apache.org/jira/secure/attachment/12519334/HBASE-5613.D2403.1.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.mapreduce.TestImportTsv
org.apache.hadoop.hbase.mapred.TestTableMapReduce
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
org.apache.hadoop.hbase.master.TestSplitLogManager
org.apache.hadoop.hbase.thrift.TestThriftServerCmdLine

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1258//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1258//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1258//console

This message is automatically generated.

ThriftServer getTableRegions does not return serverName and port

Key: HBASE-5613
URL: https://issues.apache.org/jira/browse/HBASE-5613
Project: HBase
Issue Type: Bug
Components: thrift
Reporter: Scott Chen
Assignee: Scott Chen
Priority: Minor
Attachments: HBASE-5613.D2403.1.patch

[jira] [Commented] (HBASE-5613) ThriftServer getTableRegions does not return serverName and port

2012-03-22 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235828#comment-13235828
 ] 

Scott Chen commented on HBASE-5613:
---

TestThriftServerCmdLine should be related. I will fix it.

 ThriftServer getTableRegions does not return serverName and port
 

 Key: HBASE-5613
 URL: https://issues.apache.org/jira/browse/HBASE-5613
 Project: HBase
  Issue Type: Bug
  Components: thrift
Reporter: Scott Chen
Assignee: Scott Chen
Priority: Minor
 Attachments: HBASE-5613.D2403.1.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5128) [uber hbck] Enable hbck to automatically repair table integrity problems as well as region consistency problems while online.


[ 
https://issues.apache.org/jira/browse/HBASE-5128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235830#comment-13235830
 ] 

jirapos...@reviews.apache.org commented on HBASE-5128:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4280/#review6229
---



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4280/#comment13451

I think we should distinguish the return value in this case (0) from that 
returned on line 1515.
See comment on line 1792



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4280/#comment13428

I suggest renaming holeStart as startRow and renaming holeStop as stopRow.
Then you don't need the comment on 1700.



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4280/#comment13434

Should include maxMerge in the log.



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4280/#comment13439

I wonder whether we should bail if there have been two IOE's, one on 1759 
and one here.



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4280/#comment13440

'Creating' - 'Created'



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4280/#comment13442

Are newRegion and region representing the same entity ?



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4280/#comment13453

If mergeRegionDirs() returns 0 (or less), should we note (partial) failure 
in merging ?



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4280/#comment13456

Should say 'unable to get regions from master' or something similar



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4280/#comment13458

Please remove this.



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4280/#comment13460

'with not' - 'without'
Should also include some info on the entry.



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4280/#comment13459

Please remove this.



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4280/#comment13464

Nit: name hdfsRegiondirModtime as hdfsRegionDirModTime



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4280/#comment13465

Typo: maximum


- Ted


On 2012-03-21 23:24:13, jmhsieh wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4280/
bq.  ---
bq.  
bq.  (Updated 2012-03-21 23:24:13)
bq.  
bq.  
bq.  Review request for hbase, Todd Lipcon, Ted Yu, and Lars Hofhansl.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This version is similar to the 0.90.x version posted a few months back, 
but has a few new features and some minor differences.
bq.  
bq.  1) No trackHTD method needed since we can read from the file system.
bq.  2) Added safeguards to prevent mega merges, and to isolate repairs to 
particular tables.
bq.  3) Fixed comparator in HRegionInfo
bq.  4) Fixed TestRegionObserverInterface so that it doesn't rely on bug in 
HRegionInfo comparator.
bq.  
bq.  I'll backport to 0.94/0.92 (which should be very similar) and update the 
0.90 versions after this patch has mostly cleared.
bq.  
bq.  This version is not perfect (there are definitely cases not covered) but 
it think it is worth trying to get this in so that future reviews are more 
manageable.
bq.  
bq.  
bq.  This addresses bug HBASE-5128.
bq.  https://issues.apache.org/jira/browse/HBASE-5128
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 3c635d4 
bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 
d47ef10 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java cd1755f 
bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java c0aaf65 
bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java 5916d9c 
bq.src/main/java/org/apache/hadoop/hbase/util/hbck/OfflineMetaRepair.java 
d57bb6b 
bq.
src/main/java/org/apache/hadoop/hbase/util/hbck/TableIntegrityErrorHandler.java 
PRE-CREATION 
bq.
src/main/java/org/apache/hadoop/hbase/util/hbck/TableIntegrityErrorHandlerImpl.java
 PRE-CREATION 
bq.src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java d9a2a02 
bq.src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java 937781d 
bq.

[jira] [Updated] (HBASE-5586) [replication] NPE in ReplicationSource when creating a stream to an inexistent cluster

2012-03-22 Thread Jean-Daniel Cryans (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Daniel Cryans updated HBASE-5586:
--

   Resolution: Fixed
Fix Version/s: (was: 0.90.7)
   0.96.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Committed to 0.92, 0.94, and 0.96. I skipped 0.90, should be easy enough to 
backport should anyone really need it (my guess is it won't be).

Thanks for the review Lars.

 [replication] NPE in ReplicationSource when creating a stream to an 
 inexistent cluster
 --

 Key: HBASE-5586
 URL: https://issues.apache.org/jira/browse/HBASE-5586
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.5
Reporter: Jean-Daniel Cryans
Assignee: Jean-Daniel Cryans
 Fix For: 0.92.2, 0.94.0, 0.96.0

 Attachments: 5586-v2.txt, HBASE-5586-trunk.patch, HBASE-5586.java, 
 HBASE-5586.java


 This is from 0.92.1-ish:
 {noformat}
 2012-03-15 09:52:16,589 ERROR
 org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
 Unexpected exception in ReplicationSource, currentPath=null
 java.lang.NullPointerException
at 
 org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.chooseSinks(ReplicationSource.java:223)
at 
 org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.connectToPeers(ReplicationSource.java:442)
at 
 org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:246)
 {noformat}
 I wanted to add a replication stream to a cluster that wasn't existing yet so 
 that the logs would be buffered until then. This should just be treated as if 
 there was no region servers.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5542) Unify HRegion.mutateRowsWithLocks() and HRegion.processRow()


[ 
https://issues.apache.org/jira/browse/HBASE-5542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235839#comment-13235839
 ] 

Lars Hofhansl commented on HBASE-5542:
--

Thanks Scott (for patch and verification :) )!

 Unify HRegion.mutateRowsWithLocks() and HRegion.processRow()
 

 Key: HBASE-5542
 URL: https://issues.apache.org/jira/browse/HBASE-5542
 Project: HBase
  Issue Type: Improvement
Reporter: Scott Chen
Assignee: Scott Chen
 Fix For: 0.96.0

 Attachments: HBASE-5542.2.txt, HBASE-5542.3.txt, HBASE-5542.4.txt, 
 HBASE-5542.4.txt, HBASE-5542.4.txt, HBASE-5542.D2217.1.patch, 
 HBASE-5542.D2217.10.patch, HBASE-5542.D2217.11.patch, 
 HBASE-5542.D2217.12.patch, HBASE-5542.D2217.13.patch, 
 HBASE-5542.D2217.14.patch, HBASE-5542.D2217.15.patch, 
 HBASE-5542.D2217.2.patch, HBASE-5542.D2217.3.patch, HBASE-5542.D2217.4.patch, 
 HBASE-5542.D2217.5.patch, HBASE-5542.D2217.6.patch, HBASE-5542.D2217.7.patch, 
 HBASE-5542.D2217.8.patch, HBASE-5542.D2217.9.patch, HBASE-5542.txt


 mutateRowsWithLocks() does atomic mutations on multiple rows.
 processRow() does atomic read-modify-writes on a single row.
 It will be useful to generalize both and have a
 processRowsWithLocks() that does atomic read-modify-writes on multiple rows.
 This also helps reduce some redundancy in the codes.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5586) [replication] NPE in ReplicationSource when creating a stream to an inexistent cluster

[
https://issues.apache.org/jira/browse/HBASE-5586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235843#comment-13235843
]

Hadoop QA commented on HBASE-5586:
--

-1 overall. Here are the results of testing the latest attachment

http://issues.apache.org/jira/secure/attachment/12519471/HBASE-5586-trunk.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in .

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1259//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1259//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1259//console

This message is automatically generated.

[replication] NPE in ReplicationSource when creating a stream to an
inexistent cluster
--

Attachments: 5586-v2.txt, HBASE-5586-trunk.patch, HBASE-5586.java,
HBASE-5586.java

[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size

2012-03-22 Thread Jean-Daniel Cryans (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235852#comment-13235852
]

Jean-Daniel Cryans commented on HBASE-5190:
---

The current patch works, I've tested it extensively through massive imports.
Current concerns:

- I haven't done a performance comparison, like is it going to slow down
traffic because of additional checks? Most of my testing was done so that I'm
hitting the limit all the time, so that does definitely slow down my throughput
but it's expected :)
- The exception Call queue already full doesn't make it to the client, what
happens is that it's being printed server-side and the client gets an EOF.
That's bad.
- What default should we use? In my testing I saw that 100MB might be too
small, but ideally that needs to scale with the amount of memory.

I don't mind finishing this for 0.94 if there's demand/motivation for it.

Limit the IPC queue size based on calls' payload size
-

Key: HBASE-5190
URL: https://issues.apache.org/jira/browse/HBASE-5190
Project: HBase
Issue Type: Improvement
Affects Versions: 0.90.5
Reporter: Jean-Daniel Cryans
Assignee: Jean-Daniel Cryans
Fix For: 0.94.0

Attachments: HBASE-5190.patch

Currently we limit the number of calls in the IPC queue only on their count.
It used to be really high and was dropped down recently to num_handlers * 10
(so 100 by default) because it was easy to OOME yourself when huge calls were
being queued. It's still possible to hit this problem if you use really big
values and/or a lot of handlers, so the idea is that we should take into
account the payload size. I can see 3 solutions:
- Do the accounting outside of the queue itself for all calls coming in and
out and when a call doesn't fit, throw a retryable exception.
- Same accounting but instead block the call when it comes in until space is
made available.
- Add a new parameter for the maximum size (in bytes) of a Call and then set
the size the IPC queue (in terms of the number of items) so that it could
only contain as many items as some predefined maximum size (in bytes) for the
whole queue.

[jira] [Commented] (HBASE-5542) Unify HRegion.mutateRowsWithLocks() and HRegion.processRow()


[ 
https://issues.apache.org/jira/browse/HBASE-5542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235857#comment-13235857
 ] 

Hudson commented on HBASE-5542:
---

Integrated in HBase-TRUNK #2692 (See 
[https://builds.apache.org/job/HBase-TRUNK/2692/])
HBASE-5542 Unify HRegion.mutateRowsWithLocks() and HRegion.processRow() 
(Scott Chen) part 2 (Revision 1303920)
HBASE-5542 Unify HRegion.mutateRowsWithLocks() and HRegion.processRow() (Scott 
Chen) (Revision 1303915)

 Result = SUCCESS
larsh : 
Files : 
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRowProcessorEndpoint.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/RowProcessor.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/RowProcessorProtocol.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/BaseRowProcessor.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/MultiRowMutationProcessor.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/RowProcessor.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java.orig
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestProcessRowEndpoint.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRowProcessorEndpoint.java

larsh : 
Files : 
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java


 Unify HRegion.mutateRowsWithLocks() and HRegion.processRow()
 

 Key: HBASE-5542
 URL: https://issues.apache.org/jira/browse/HBASE-5542
 Project: HBase
  Issue Type: Improvement
Reporter: Scott Chen
Assignee: Scott Chen
 Fix For: 0.96.0

 Attachments: HBASE-5542.2.txt, HBASE-5542.3.txt, HBASE-5542.4.txt, 
 HBASE-5542.4.txt, HBASE-5542.4.txt, HBASE-5542.D2217.1.patch, 
 HBASE-5542.D2217.10.patch, HBASE-5542.D2217.11.patch, 
 HBASE-5542.D2217.12.patch, HBASE-5542.D2217.13.patch, 
 HBASE-5542.D2217.14.patch, HBASE-5542.D2217.15.patch, 
 HBASE-5542.D2217.2.patch, HBASE-5542.D2217.3.patch, HBASE-5542.D2217.4.patch, 
 HBASE-5542.D2217.5.patch, HBASE-5542.D2217.6.patch, HBASE-5542.D2217.7.patch, 
 HBASE-5542.D2217.8.patch, HBASE-5542.D2217.9.patch, HBASE-5542.txt


 mutateRowsWithLocks() does atomic mutations on multiple rows.
 processRow() does atomic read-modify-writes on a single row.
 It will be useful to generalize both and have a
 processRowsWithLocks() that does atomic read-modify-writes on multiple rows.
 This also helps reduce some redundancy in the codes.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5328) Small changes to Master to make it more testable


[ 
https://issues.apache.org/jira/browse/HBASE-5328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235858#comment-13235858
 ] 

Hudson commented on HBASE-5328:
---

Integrated in HBase-TRUNK #2692 (See 
[https://builds.apache.org/job/HBase-TRUNK/2692/])
HBASE-5328 Small changes to Master to make it more testable (Revision 
1303648)
HBASE-5328 Small changes to Master to make it more testable (Revision 1303626)

 Result = SUCCESS
stack : 
Files : 
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java

stack : 
Files : 
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/Mocking.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java


 Small changes to Master to make it more testable
 

 Key: HBASE-5328
 URL: https://issues.apache.org/jira/browse/HBASE-5328
 Project: HBase
  Issue Type: Task
Reporter: stack
Assignee: stack
 Fix For: 0.96.0

 Attachments: 5328-addendum.txt, 5328.txt, 5328v12.txt, 5328v2.txt, 
 5328v2.txt, 5328v3.txt, 5328v4.txt, 5328v8.txt


 Here are some small changes in Master that make it more testable.  Included 
 tests stand up a Master and then fake it into thinking that three 
 regionservers are registering making master assign root and meta, etc.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5433) [REST] Add metrics to keep track of success/failure count


[ 
https://issues.apache.org/jira/browse/HBASE-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235860#comment-13235860
 ] 

Hudson commented on HBASE-5433:
---

Integrated in HBase-TRUNK #2692 (See 
[https://builds.apache.org/job/HBase-TRUNK/2692/])
HBASE-5433 [REST] Add metrics to keep track of success/failure count 
(Mubarak Seyed) (Revision 1303536)

 Result = SUCCESS
larsh : 
Files : 
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/MultiRowResource.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/RegionsResource.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/RootResource.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/RowResource.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/ScannerInstanceResource.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/ScannerResource.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/SchemaResource.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/StorageClusterStatusResource.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/StorageClusterVersionResource.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/VersionResource.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/metrics/RESTMetrics.java


 [REST] Add metrics to keep track of success/failure count
 -

 Key: HBASE-5433
 URL: https://issues.apache.org/jira/browse/HBASE-5433
 Project: HBase
  Issue Type: Improvement
  Components: metrics, rest
Affects Versions: 0.94.0
Reporter: Mubarak Seyed
Assignee: Mubarak Seyed
  Labels: noob
 Fix For: 0.94.0, 0.96.0

 Attachments: HBASE-5433.trunk.v1.patch


 In a production environment, the visibility of successful REST request(s) are 
 not getting exposed to metric system as we have only one metric (requests) 
 today.
 Proposing to add more metrics such as successful_get_count, failed_get_count, 
 successful_put_count, failed_put_count
 The current implementation increases the request count at the beginning of 
 the method implementation and it is very hard to monitor requests (unless 
 turn on debug, find the row_key and validate it in get/scan using hbase 
 shell), it will be very useful to ops to keep an eye as requests from cross 
 data-centers are trying to write data to one cluster using REST gateway 
 through load balancer (and there is no visibility of which REST-server/RS 
 failed to write data)
 {code}
  Response update(final CellSetModel model, final boolean replace) {
 // for requests
 servlet.getMetrics().incrementRequests(1);
..  
..
   table.put(puts);
   table.flushCommits();
   ResponseBuilder response = Response.ok();
   // for successful_get_count
   servlet.getMetrics().incrementSuccessfulGetRequests(1);
   return response.build();
 } catch (IOException e) {
   // for failed_get_count
   servlet.getMetrics().incrementFailedGetRequests(1);
   throw new WebApplicationException(e,
   Response.Status.SERVICE_UNAVAILABLE);
 } finally {
 }
   }
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4940) hadoop-metrics.properties can include configuration of the rest context for ganglia


[ 
https://issues.apache.org/jira/browse/HBASE-4940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235859#comment-13235859
 ] 

Hudson commented on HBASE-4940:
---

Integrated in HBase-TRUNK #2692 (See 
[https://builds.apache.org/job/HBase-TRUNK/2692/])
HBASE-4940 hadoop-metrics.properties can include configuration of the rest 
context for ganglia (Mubarak Seyed) (Revision 1303553)

 Result = SUCCESS
larsh : 
Files : 
* /hbase/trunk/conf/hadoop-metrics.properties


 hadoop-metrics.properties can include configuration of the rest context for 
 ganglia
 -

 Key: HBASE-4940
 URL: https://issues.apache.org/jira/browse/HBASE-4940
 Project: HBase
  Issue Type: Improvement
  Components: metrics
Affects Versions: 0.90.5
 Environment: HBase-0.90.1
Reporter: Mubarak Seyed
Assignee: Mubarak Seyed
Priority: Minor
  Labels: hbase-rest
 Fix For: 0.94.0, 0.96.0

 Attachments: HBASE-4940.patch, HBASE-4940.trunk.v1.patch, 
 HBASE-4940.trunk.v2.patch


 It appears from hadoop-metrics.properties that configuration for rest context 
 is missing. It would be good if we add the rest context and commented out 
 them, if anyone is using rest-server and if they want to monitor using 
 ganglia context then they can uncomment the rest context and use them for 
 rest-server monitoring using ganglia.
 {code}
 # Configuration of the rest context for ganglia
 #rest.class=org.apache.hadoop.metrics.ganglia.GangliaContext
 #rest.period=10
 #rest.servers=ganglia-metad-hostname:port
 {code}
 Working on the patch, will submit it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5586) [replication] NPE in ReplicationSource when creating a stream to an inexistent cluster


[ 
https://issues.apache.org/jira/browse/HBASE-5586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235888#comment-13235888
 ] 

Hudson commented on HBASE-5586:
---

Integrated in HBase-0.94 #46 (See 
[https://builds.apache.org/job/HBase-0.94/46/])
HBASE-5586  [replication] NPE in ReplicationSource when creating a stream
   to an inexistent cluster (Revision 1303944)

 Result = SUCCESS
jdcryans : 
Files : 
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
* 
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationZookeeper.java


 [replication] NPE in ReplicationSource when creating a stream to an 
 inexistent cluster
 --

 Key: HBASE-5586
 URL: https://issues.apache.org/jira/browse/HBASE-5586
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.5
Reporter: Jean-Daniel Cryans
Assignee: Jean-Daniel Cryans
 Fix For: 0.92.2, 0.94.0, 0.96.0

 Attachments: 5586-v2.txt, HBASE-5586-trunk.patch, HBASE-5586.java, 
 HBASE-5586.java


 This is from 0.92.1-ish:
 {noformat}
 2012-03-15 09:52:16,589 ERROR
 org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
 Unexpected exception in ReplicationSource, currentPath=null
 java.lang.NullPointerException
at 
 org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.chooseSinks(ReplicationSource.java:223)
at 
 org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.connectToPeers(ReplicationSource.java:442)
at 
 org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:246)
 {noformat}
 I wanted to add a replication stream to a cluster that wasn't existing yet so 
 that the logs would be buffered until then. This should just be treated as if 
 there was no region servers.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4607) Split log worker should terminate properly when waiting for znode

2012-03-22 Thread stack (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235890#comment-13235890
 ] 

stack commented on HBASE-4607:
--

Seems fine to me.  +1

 Split log worker should terminate properly when waiting for znode
 -

 Key: HBASE-4607
 URL: https://issues.apache.org/jira/browse/HBASE-4607
 Project: HBase
  Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Priority: Minor
 Fix For: 0.94.0

 Attachments: 
 HBASE-4607_SplitLogWorker_should_correct-20111017231456-47a82ef3.patch


 This is an attempt to fix the fact that SplitLogWorker threads are not being 
 terminated properly in some unit tests. This probably does not happen in 
 production because the master always creates the log-splitting ZK node, but 
 it does happen in 89-fb. Thanks to Prakash Khemani for help on this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size

2012-03-22 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235895#comment-13235895
]

Lars Hofhansl commented on HBASE-5190:
--

I doubt you'll see a slowdown from this. Then again AtomicLong needs a memory
barrier as far as I know, so could potentially add a noticeable slowdown. Hmm...

100MB is per RegionServer, right? Does seem a bit small. Maybe 1G?

We really need a streaming API for large rows, but that's a different story.

I take it that maybe this is something to consider for 0.96. Agreed?

Limit the IPC queue size based on calls' payload size
-

Attachments: HBASE-5190.patch

[jira] [Commented] (HBASE-5128) [uber hbck] Enable hbck to automatically repair table integrity problems as well as region consistency problems while online.


[ 
https://issues.apache.org/jira/browse/HBASE-5128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235896#comment-13235896
 ] 

jirapos...@reviews.apache.org commented on HBASE-5128:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4280/#review6239
---



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4280/#comment13498

Is @Override missing ?



src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java
https://reviews.apache.org/r/4280/#comment13474

Renaming this method is desirable as I mentioned earlier.



src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java
https://reviews.apache.org/r/4280/#comment13473

Typo: assume



src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java
https://reviews.apache.org/r/4280/#comment13475

This method is called in two places where HBaseAdmin is available.

Please change the method signature to avoid creating HBaseAdmin every time.



src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java
https://reviews.apache.org/r/4280/#comment13478

Typo: handleHBCK



src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java
https://reviews.apache.org/r/4280/#comment13485

This is called in a loop in checkMetaRegion().
It would be nice for this method to take a list of regions and wait for 
them to come out of RIT.



src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java
https://reviews.apache.org/r/4280/#comment13483

Why ?



src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java
https://reviews.apache.org/r/4280/#comment13484

success is no longer set in this method.
This can be removed.



src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java
https://reviews.apache.org/r/4280/#comment13486

Shall we return directly here ?
The new exception would be caught at line 182



src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java
https://reviews.apache.org/r/4280/#comment13487

It would be nice to cache meta for subsequent calls.
Can be done in another JIRA.



src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java
https://reviews.apache.org/r/4280/#comment13489

Please use this method in the three places of HBaseFsck I mentioned.



src/main/java/org/apache/hadoop/hbase/util/hbck/TableIntegrityErrorHandler.java
https://reviews.apache.org/r/4280/#comment13494

Javadoc for parameters.



src/main/java/org/apache/hadoop/hbase/util/hbck/TableIntegrityErrorHandler.java
https://reviews.apache.org/r/4280/#comment13495

Javadoc for parameters.



src/main/java/org/apache/hadoop/hbase/util/hbck/TableIntegrityErrorHandler.java
https://reviews.apache.org/r/4280/#comment13496

Javadoc for parameters.



src/main/java/org/apache/hadoop/hbase/util/hbck/TableIntegrityErrorHandlerImpl.java
https://reviews.apache.org/r/4280/#comment13497

Can this class be package-private ?



src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
https://reviews.apache.org/r/4280/#comment13501

This check was added because of failed test ?



src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
https://reviews.apache.org/r/4280/#comment13502

Can we reuse the method from HBaseFsck ?


- Ted


On 2012-03-21 23:24:13, jmhsieh wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4280/
bq.  ---
bq.  
bq.  (Updated 2012-03-21 23:24:13)
bq.  
bq.  
bq.  Review request for hbase, Todd Lipcon, Ted Yu, and Lars Hofhansl.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This version is similar to the 0.90.x version posted a few months back, 
but has a few new features and some minor differences.
bq.  
bq.  1) No trackHTD method needed since we can read from the file system.
bq.  2) Added safeguards to prevent mega merges, and to isolate repairs to 
particular tables.
bq.  3) Fixed comparator in HRegionInfo
bq.  4) Fixed TestRegionObserverInterface so that it doesn't rely on bug in 
HRegionInfo comparator.
bq.  
bq.  I'll backport to 0.94/0.92 (which should be very similar) and update the 
0.90 versions after this patch has mostly cleared.
bq.  
bq.  This version is not perfect (there are definitely cases not covered) but 
it think it is worth trying to get this in so that future reviews are more 
manageable.
bq.  
bq.  
bq.  This addresses bug HBASE-5128.
bq.  https://issues.apache.org/jira/browse/HBASE-5128
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 3c635d4 
bq.

[jira] [Commented] (HBASE-4957) Clean up some log messages, code in RecoverableZooKeeper

2012-03-22 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235901#comment-13235901
 ] 

Lars Hofhansl commented on HBASE-4957:
--

Let's commit if hadoopqa is (mostly) cool with it :)

 Clean up some log messages, code in RecoverableZooKeeper
 

 Key: HBASE-4957
 URL: https://issues.apache.org/jira/browse/HBASE-4957
 Project: HBase
  Issue Type: Improvement
  Components: zookeeper
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Minor
 Fix For: 0.94.0

 Attachments: hbase-4957.txt, hbase-4957.txt, hbase-4957.txt


 In RecoverableZooKeeper, there are a number of log messages and comments 
 which don't really read correctly, and some other pieces of code that can be 
 cleaned up. Simple cleanup - shouldn't be any actual behavioral changes.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4607) Split log worker should terminate properly when waiting for znode

2012-03-22 Thread Zhihong Yu (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Yu updated HBASE-4607:
--

Hadoop Flags: Reviewed
  Status: Patch Available  (was: Open)

 Split log worker should terminate properly when waiting for znode
 -

 Key: HBASE-4607
 URL: https://issues.apache.org/jira/browse/HBASE-4607
 Project: HBase
  Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Priority: Minor
 Fix For: 0.94.0

 Attachments: 
 HBASE-4607_SplitLogWorker_should_correct-20111017231456-47a82ef3.patch


 This is an attempt to fix the fact that SplitLogWorker threads are not being 
 terminated properly in some unit tests. This probably does not happen in 
 production because the master always creates the log-splitting ZK node, but 
 it does happen in 89-fb. Thanks to Prakash Khemani for help on this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5190) Limit the IPC queue size based on calls' payload size

2012-03-22 Thread Jean-Daniel Cryans (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235909#comment-13235909
]

Jean-Daniel Cryans commented on HBASE-5190:
---

bq. 100MB is per RegionServer, right? Does seem a bit small. Maybe 1G?

Might be a good default, those with the default heap will definitely not get
any help here though.

bq. I take it that maybe this is something to consider for 0.96. Agreed?

The more I think about it, the more I want this in 0.94 because it can really
give us a better understanding of those issues we see on the mailing list.

Limit the IPC queue size based on calls' payload size
-

Attachments: HBASE-5190.patch

[jira] [Commented] (HBASE-5128) [uber hbck] Enable hbck to automatically repair table integrity problems as well as region consistency problems while online.


[ 
https://issues.apache.org/jira/browse/HBASE-5128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235912#comment-13235912
 ] 

jirapos...@reviews.apache.org commented on HBASE-5128:
--



bq.  On 2012-03-22 06:33:20, Ted Yu wrote:
bq.   src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 554
bq.   https://reviews.apache.org/r/4280/diff/2/?file=94413#file94413line554
bq.  
bq.   Can we do this in the current JIRA ?
bq.   
bq.   Why do we need to reload for every type of fix ?

I'd rather do it in a follow on issue.  Correctness first, then performance.  
This patch is massive already.


bq.  On 2012-03-22 06:33:20, Ted Yu wrote:
bq.   src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 404
bq.   https://reviews.apache.org/r/4280/diff/2/?file=94413#file94413line404
bq.  
bq.   Should be 'what are online'

get regions according to what is online on each RegionServer


bq.  On 2012-03-22 06:33:20, Ted Yu wrote:
bq.   src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 418
bq.   https://reviews.apache.org/r/4280/diff/2/?file=94413#file94413line418
bq.  
bq.   checkAndRestoreConsistency() would be a better name.

every other variable is fix* so I think it seems ok to keep this fix as well.


bq.  On 2012-03-22 06:33:20, Ted Yu wrote:
bq.   src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 435
bq.   https://reviews.apache.org/r/4280/diff/2/?file=94413#file94413line435
bq.  
bq.   I think master.synchronousBalanceSwitch() is better candidate for 
this action.

I agree, but since this method is only in the trunk/0.94 branches I'll file a 
follow on issue for this.


bq.  On 2012-03-22 06:33:20, Ted Yu wrote:
bq.   src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 457
bq.   https://reviews.apache.org/r/4280/diff/2/?file=94413#file94413line457
bq.  
bq.   the trailing s of '.regioninfos' should be removed.

Orphaned regions are regions without a .regioninfo file in them.


bq.  On 2012-03-22 06:33:20, Ted Yu wrote:
bq.   src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 484
bq.   https://reviews.apache.org/r/4280/diff/2/?file=94413#file94413line484
bq.  
bq.   I don't see where the hf is closed.

good catch!


bq.  On 2012-03-22 06:33:20, Ted Yu wrote:
bq.   src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 488
bq.   https://reviews.apache.org/r/4280/diff/2/?file=94413#file94413line488
bq.  
bq.   Should hfile be added to a list so that we can report them 
collectively ?
bq.   
bq.   Currently user has to search the output of hbck.

bq. From my point of view it is easier to keep these all on separate lines so 
we can grep the output.  Adding word orphan to log message.  


bq.  On 2012-03-22 06:33:20, Ted Yu wrote:
bq.   src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 489
bq.   https://reviews.apache.org/r/4280/diff/2/?file=94413#file94413line489
bq.  
bq.   Shall we continue with the remaining HFiles ?

good point. changed break to continue.


bq.  On 2012-03-22 06:33:20, Ted Yu wrote:
bq.   src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 501
bq.   https://reviews.apache.org/r/4280/diff/2/?file=94413#file94413line501
bq.  
bq.   Help me understand this comparison:
bq.   are we shrinking the range here ?

Good catch! 

The goal here is to indeed expand the region to cover the range of all the 
hfiles.  


bq.  On 2012-03-22 06:33:20, Ted Yu wrote:
bq.   src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 531
bq.   https://reviews.apache.org/r/4280/diff/2/?file=94413#file94413line531
bq.  
bq.   Should read 'If there are errors to be fixed'

   * This method determines if there are table integrity errors in HDFS.  If
   * there are errors and the appropriate fix options are enabled, the method
   * will first correct orphan regions making them into legit regiondirs, and
   * then reload to merge potentially overlapping regions.


bq.  On 2012-03-22 06:33:20, Ted Yu wrote:
bq.   src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 567
bq.   https://reviews.apache.org/r/4280/diff/2/?file=94413#file94413line567
bq.  
bq.   Some assertion here for the declared state (no holes) ?

removed no orphans, no holes from comment - the overlap repairs could happen if 
the hdfs hole fix options are off.


bq.  On 2012-03-22 06:33:20, Ted Yu wrote:
bq.   src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 655
bq.   https://reviews.apache.org/r/4280/diff/2/?file=94413#file94413line655
bq.  
bq.   This exception isn't used.
bq.   Do we need it ?

not needed and removed.  I believe this is in the 0.90 version and a remnant of 
porting back and forth between versions.


bq.  On 2012-03-22 06:33:20, Ted Yu wrote:
bq.   src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 702
bq.

[jira] [Updated] (HBASE-4957) Clean up some log messages, code in RecoverableZooKeeper

2012-03-22 Thread stack (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4957:
-

Status: Patch Available  (was: Open)

 Clean up some log messages, code in RecoverableZooKeeper
 

 Key: HBASE-4957
 URL: https://issues.apache.org/jira/browse/HBASE-4957
 Project: HBase
  Issue Type: Improvement
  Components: zookeeper
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Minor
 Fix For: 0.94.0

 Attachments: hbase-4957.txt, hbase-4957.txt, hbase-4957.txt


 In RecoverableZooKeeper, there are a number of log messages and comments 
 which don't really read correctly, and some other pieces of code that can be 
 cleaned up. Simple cleanup - shouldn't be any actual behavioral changes.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4607) Split log worker should terminate properly when waiting for znode

2012-03-22 Thread Mikhail Bautin (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235926#comment-13235926
 ] 

Mikhail Bautin commented on HBASE-4607:
---

Tried to apply the patch, but it shows as already applied. Upon further 
analysis, very similar changes to SplitLogWorker have been integrated as part 
of HBASE-5542:

svn diff -r 1303648:1303920  
https://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java


 Split log worker should terminate properly when waiting for znode
 -

 Key: HBASE-4607
 URL: https://issues.apache.org/jira/browse/HBASE-4607
 Project: HBase
  Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Priority: Minor
 Fix For: 0.94.0

 Attachments: 
 HBASE-4607_SplitLogWorker_should_correct-20111017231456-47a82ef3.patch


 This is an attempt to fix the fact that SplitLogWorker threads are not being 
 terminated properly in some unit tests. This probably does not happen in 
 production because the master always creates the log-splitting ZK node, but 
 it does happen in 89-fb. Thanks to Prakash Khemani for help on this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5591) ThiftServerRunner.HBaseHandler.toBytes() is identical to Bytes.getBytes()

2012-03-22 Thread Scott Chen (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Chen updated HBASE-5591:
--

Status: Patch Available  (was: Open)

 ThiftServerRunner.HBaseHandler.toBytes() is identical to Bytes.getBytes()
 -

 Key: HBASE-5591
 URL: https://issues.apache.org/jira/browse/HBASE-5591
 Project: HBase
  Issue Type: Improvement
Reporter: Scott Chen
Assignee: Scott Chen
Priority: Trivial
 Attachments: HBASE-5591.D2355.1.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4607) Split log worker should terminate properly when waiting for znode

2012-03-22 Thread Mikhail Bautin (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Bautin updated HBASE-4607:
--

Resolution: Fixed
Status: Resolved  (was: Patch Available)

The same changes committed in HBASE-5542.

 Split log worker should terminate properly when waiting for znode
 -

 Key: HBASE-4607
 URL: https://issues.apache.org/jira/browse/HBASE-4607
 Project: HBase
  Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Priority: Minor
 Fix For: 0.94.0

 Attachments: 
 HBASE-4607_SplitLogWorker_should_correct-20111017231456-47a82ef3.patch


 This is an attempt to fix the fact that SplitLogWorker threads are not being 
 terminated properly in some unit tests. This probably does not happen in 
 production because the master always creates the log-splitting ZK node, but 
 it does happen in 89-fb. Thanks to Prakash Khemani for help on this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4657) Improve the efficiency of our MR jobs with a few configurations


 [ 
https://issues.apache.org/jira/browse/HBASE-4657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-4657:
-

Attachment: 4657.txt

How about something as simple as this.
Set cacheBlocks to false where appropriate, and also advises how to disable 
speculative execution and to set scanner caching to 100.

 Improve the efficiency of our MR jobs with a few configurations
 ---

 Key: HBASE-4657
 URL: https://issues.apache.org/jira/browse/HBASE-4657
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.90.4
Reporter: Jean-Daniel Cryans
Assignee: Lars Hofhansl
 Fix For: 0.94.0

 Attachments: 4657.txt


 This is a low hanging fruit, some of our MR jobs like RowCounter and 
 CopyTable don't even setCacheBlocks on the scan object which out of the box 
 completely screws up a running system. Another thing would be to disable 
 speculative execution.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5548) Add ability to get a table in the shell

2012-03-22 Thread Jesse Yates (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesse Yates updated HBASE-5548:
---

Attachment: ruby_HBASE-5548-v3.patch

Updated patch with:
* more commenting!
* updates for all the table commands
* addition of admin utilities on a table (e.g. flush, enable, disable and drop)

Think this is ready (and large enough to RB). thoughts?

 Add ability to get a table in the shell
 ---

 Key: HBASE-5548
 URL: https://issues.apache.org/jira/browse/HBASE-5548
 Project: HBase
  Issue Type: Improvement
  Components: shell
Reporter: Jesse Yates
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.1

 Attachments: ruby_HBASE-5528-v0.patch, ruby_HBASE-5548-v1.patch, 
 ruby_HBASE-5548-v2.patch, ruby_HBASE-5548-v3.patch


 Currently, all the commands that operate on a table in the shell first have 
 to take the table as name as input. 
 There are two main considerations:
 * It is annoying to have to write the table name every time, when you should 
 just be able to get a reference to a table
 * the current implementation is very wasteful - it creates a new HTable for 
 each call (but reuses the connection since it uses the same configuration)
 We should be able to get a handle to a single HTable and then operate on that.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5613) ThriftServer getTableRegions does not return serverName and port

2012-03-22 Thread Phabricator (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HBASE-5613:
---

Attachment: HBASE-5613.D2403.2.patch

sc updated the revision HBASE-5613 [jira] ThriftServer getTableRegions does 
not return serverName and port.
Reviewers: dhruba, tedyu, JIRA

  Fixed unit test

REVISION DETAIL
  https://reviews.facebook.net/D2403

AFFECTED FILES
  src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java
  src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServer.java


 ThriftServer getTableRegions does not return serverName and port
 

 Key: HBASE-5613
 URL: https://issues.apache.org/jira/browse/HBASE-5613
 Project: HBase
  Issue Type: Bug
  Components: thrift
Reporter: Scott Chen
Assignee: Scott Chen
Priority: Minor
 Attachments: HBASE-5613.D2403.1.patch, HBASE-5613.D2403.2.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4607) Split log worker should terminate properly when waiting for znode