[jira] [Commented] (HBASE-5837) hbase shell deleteall to .META. allows insertion of malformed rowkey.

2012-04-20 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13258344#comment-13258344
 ] 

Jonathan Hsieh commented on HBASE-5837:
---

Here's an example shell command that will leave you with a borked hbase:

{code}
deleteall '.META.', 'table,not a comma'
{code}


 hbase shell deleteall to .META. allows insertion of malformed rowkey.
 -

 Key: HBASE-5837
 URL: https://issues.apache.org/jira/browse/HBASE-5837
 Project: HBase
  Issue Type: Bug
  Components: master, shell
Affects Versions: 0.90.6
Reporter: Jonathan Hsieh

 When using the hbase shell to manipulate meta entries, one is allowed to 
 'delete' malformed rows (entries with less than 2 ascii 44 ',' chars).  When 
 this happens HBase servers may go down and the cluster will not be 
 restartable without manual intervention.  
 The delete results in a durable malformed rowkey in .META.'s memstore, 
 .META.'s HLog, and eventually .META.'s HFiles.  Subsequent scans to meta 
 (such as when a HMaster starts) fail in the scanner because the comparator 
 fails.  In the case of an HMaster startup, it causes an abort that kills the 
 HMaster process.
 {code}
 12/04/18 22:07:34 FATAL master.HMaster: Unhandled exception. Starting 
 shutdown.
 org.apache.hadoop.ipc.RemoteException: java.io.IOException: 
 java.lang.IllegalArgumentException: No 44 in 
 blah,1334744821162.81f2df35c332dd2d3bb966fb5b419568., length=47, offset=54
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:990)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:979)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1894)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1834)
 at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
 at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
 at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
 Caused by: java.lang.IllegalArgumentException: No 44 in 
 blah,1334744821162.81f2df35c332dd2d3bb966fb5b419568., length=47, offset=54
 at 
 org.apache.hadoop.hbase.KeyValue.getRequiredDelimiterInReverse(KeyValue.java:1300)
 at 
 org.apache.hadoop.hbase.KeyValue$MetaKeyComparator.compareRows(KeyValue.java:1846)
 at 
 org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.match(ScanQueryMatcher.java:130)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:257)
 at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2435)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2391)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2408)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1870)
 ... 6 more
 at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
 at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
 at $Proxy9.next(Unknown Source)
 at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:264)
 at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:237)
 at 
 org.apache.hadoop.hbase.catalog.MetaReader.fullScanOfResults(MetaReader.java:220)
 at 
 org.apache.hadoop.hbase.master.AssignmentManager.rebuildUserRegions(AssignmentManager.java:1580)
 at 
 org.apache.hadoop.hbase.master.AssignmentManager.processFailover(AssignmentManager.java:221)
 at 
 org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:422)
 at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:295)
 12/04/18 22:07:34 INFO master.HMaster: Aborting 
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5842) Passing shell commands as an argument

2012-04-20 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13258360#comment-13258360
 ] 

Jonathan Hsieh commented on HBASE-5842:
---

Harsh,  I would add that the output from the shell of that particular example 
is too clever with formatting and ends up being cumbersome to use with standard 
unix parsing tools.

For that particular command, I've hacked the HFile tool do dump contents in an 
grep'able format.  

Are you looking for that command in particular or are there more cases?

Maybe we should add a hbase-admin command that has shortcuts to utility methods 
like HLog, HFile, and something like DumpMeta?



 Passing shell commands as an argument
 -

 Key: HBASE-5842
 URL: https://issues.apache.org/jira/browse/HBASE-5842
 Project: HBase
  Issue Type: Improvement
  Components: shell
Affects Versions: 0.94.0
Reporter: Harsh J
Priority: Minor

 Many times we've required scans of .META. to analyze issues with the cluster 
 we work on, and to have the result in a file we can pass around we usually 
 end up doing something like:
 {{echo scan '.META.'| hbase shell  meta-scan.txt}}
 This can rather be simplified as something like the following instead, with 
 support for a commands reading argument:
 {{hbase shell -c scan '.META.'}}
 [Note though: File reading is possible already, i.e. {{hbase shell file.hs}}, 
 but then thats two steps and we usually don't keep a file around for just a 
 meta table scan.]

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5823) Hbck should be able to print help

2012-04-19 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13257277#comment-13257277
 ] 

Jonathan Hsieh commented on HBASE-5823:
---

Thanks Enis!  (this got lost along when I added table restrictions).  



 Hbck should be able to print help
 -

 Key: HBASE-5823
 URL: https://issues.apache.org/jira/browse/HBASE-5823
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.92.1, 0.96.0, 0.94.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Priority: Minor
 Fix For: 0.92.2, 0.94.0

 Attachments: hbase-hbck.patch


 bin/hbase hbck -h and -help should print the help message. It used to print 
 help when unrecognized options are passed. We can backport this to 0.92/0.94 
 branches as well. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5654) [findbugs] Address dodgy bugs

2012-04-19 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13257652#comment-13257652
 ] 

Jonathan Hsieh commented on HBASE-5654:
---

bq. -1 core tests. The patch failed these unit tests:

This usually indicates that there was a hung test somewhere -- there is a 
script in ./dev-* that should help you find it.

 [findbugs] Address dodgy bugs
 -

 Key: HBASE-5654
 URL: https://issues.apache.org/jira/browse/HBASE-5654
 Project: HBase
  Issue Type: Sub-task
  Components: scripts
Affects Versions: 0.96.0
Reporter: Jonathan Hsieh
Assignee: Ashutosh Jindal
  Labels: patch
 Fix For: 0.96.0

 Attachments: Hbase 5654_v3.patch, Hbase-5654.patch, 
 Hbase_5654_V2.patch


 See 
 https://builds.apache.org/job/PreCommit-HBASE-Build/1313//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html#Warnings_STYLE
 This may be broken down further.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5654) [findbugs] Address dodgy bugs

2012-04-19 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13257871#comment-13257871
 ] 

Jonathan Hsieh commented on HBASE-5654:
---

@stack.  Got it.  Thanks.  

 [findbugs] Address dodgy bugs
 -

 Key: HBASE-5654
 URL: https://issues.apache.org/jira/browse/HBASE-5654
 Project: HBase
  Issue Type: Sub-task
  Components: scripts
Affects Versions: 0.96.0
Reporter: Jonathan Hsieh
Assignee: Ashutosh Jindal
  Labels: patch
 Fix For: 0.96.0

 Attachments: Hbase 5654_v3.patch, Hbase-5654.patch, 
 Hbase_5654_V2.patch


 See 
 https://builds.apache.org/job/PreCommit-HBASE-Build/1313//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html#Warnings_STYLE
 This may be broken down further.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5831) hadoopqa builds not completing

2012-04-19 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13257916#comment-13257916
 ] 

Jonathan Hsieh commented on HBASE-5831:
---

I tried the tested in question on trunk and it passes in eclipse.  It fails in 
mvn.  I've encountered this kind of problem previously as well with HBASE-5793 
-- my eclipse setup differs from mvn by not using 
./src/test/resources/hbase-site.xml in the resources dir.

Since the test expects and error, my guess is this may have to do with the 
client retry number getting bumped up to 100 in the resource hbase-site.xml 
file in HBASE-5747.

When I edited that file back down client retries to 10 it passes in about 2 
minutes.

 hadoopqa builds not completing
 --

 Key: HBASE-5831
 URL: https://issues.apache.org/jira/browse/HBASE-5831
 Project: HBase
  Issue Type: Bug
  Components: test
Reporter: stack
Assignee: stack
Priority: Blocker
 Attachments: 5831.remove.TestLoadIncrementalHFilesSplitRecovery.txt, 
 5831.remove.TestLoadIncrementalHFilesSplitRecovery.txt, 
 5831.remove.TestLoadIncrementalHFilesSplitRecovery.txt, 
 5831.remove.TestLoadIncrementalHFilesSplitRecovery.txt, 
 5831.remove.TestLoadIncrementalHFilesSplitRecovery.txt, 
 5831.remove.all.mapreduce.txt


 No test failures but build complains it has failed.  trunk build seems to 
 have the same affliction:
 {code}
 Results :
 Tests run: 909, Failures: 0, Errors: 0, Skipped: 9
 [INFO] 
 
 [INFO] BUILD FAILURE
 [INFO] 
 
 [INFO] Total time: 41:19.273s
 [INFO] Finished at: Wed Apr 18 21:54:31 UTC 2012
 [INFO] Final Memory: 59M/451M
 [INFO] 
 
 [ERROR] Failed to execute goal 
 org.apache.maven.plugins:maven-surefire-plugin:2.12-TRUNK-HBASE-2:test 
 (secondPartTestsExecution) on project hbase: Failure or timeout - [Help 1]
 [ERROR] 
 [ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
 switch.
 [ERROR] Re-run Maven using the -X switch to enable full debug logging.
 [ERROR] 
 [ERROR] For more information about the errors and possible solutions, please 
 read the following articles:
 [ERROR] [Help 1] 
 http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
 -1 overall.  Here are the results of testing the latest attachment 
   http://issues.apache.org/jira/secure/attachment/12523250/5811+%281%29.txt
   against trunk revision .
 +1 @author.  The patch does not contain any @author tags.
 +1 tests included.  The patch appears to include 3 new or modified tests.
 +1 javadoc.  The javadoc tool did not generate any warning messages.
 +1 javac.  The applied patch does not increase the total number of javac 
 compiler warnings.
 -1 findbugs.  The patch appears to introduce 6 new Findbugs (version 
 1.3.9) warnings.
 +1 release audit.  The applied patch does not increase the total number 
 of release audit warnings.
  -1 core tests.  The patch failed these unit tests:
 {code}
 Its not apparent that any particular test is not finishing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5831) hadoopqa builds not completing

2012-04-19 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13257991#comment-13257991
 ] 

Jonathan Hsieh commented on HBASE-5831:
---

@Stack, it seems from the HBASE-5794 patch the mapreduce tests, including 
TestLoadIncrementalHFilesSplitRecovery, were passing. 

https://builds.apache.org/job/PreCommit-HBASE-Build/1587//testReport/org.apache.hadoop.hbase.mapreduce/

The other failures on HBASE-5794 seem somewhat odd however.




 hadoopqa builds not completing
 --

 Key: HBASE-5831
 URL: https://issues.apache.org/jira/browse/HBASE-5831
 Project: HBase
  Issue Type: Bug
  Components: test
Reporter: stack
Assignee: stack
Priority: Blocker
 Attachments: 5831.remove.TestLoadIncrementalHFilesSplitRecovery.txt, 
 5831.remove.TestLoadIncrementalHFilesSplitRecovery.txt, 
 5831.remove.TestLoadIncrementalHFilesSplitRecovery.txt, 
 5831.remove.TestLoadIncrementalHFilesSplitRecovery.txt, 
 5831.remove.TestLoadIncrementalHFilesSplitRecovery.txt, 
 5831.remove.all.mapreduce.txt, 5831.remove.all.mapreduce.txt


 No test failures but build complains it has failed.  trunk build seems to 
 have the same affliction:
 {code}
 Results :
 Tests run: 909, Failures: 0, Errors: 0, Skipped: 9
 [INFO] 
 
 [INFO] BUILD FAILURE
 [INFO] 
 
 [INFO] Total time: 41:19.273s
 [INFO] Finished at: Wed Apr 18 21:54:31 UTC 2012
 [INFO] Final Memory: 59M/451M
 [INFO] 
 
 [ERROR] Failed to execute goal 
 org.apache.maven.plugins:maven-surefire-plugin:2.12-TRUNK-HBASE-2:test 
 (secondPartTestsExecution) on project hbase: Failure or timeout - [Help 1]
 [ERROR] 
 [ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
 switch.
 [ERROR] Re-run Maven using the -X switch to enable full debug logging.
 [ERROR] 
 [ERROR] For more information about the errors and possible solutions, please 
 read the following articles:
 [ERROR] [Help 1] 
 http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
 -1 overall.  Here are the results of testing the latest attachment 
   http://issues.apache.org/jira/secure/attachment/12523250/5811+%281%29.txt
   against trunk revision .
 +1 @author.  The patch does not contain any @author tags.
 +1 tests included.  The patch appears to include 3 new or modified tests.
 +1 javadoc.  The javadoc tool did not generate any warning messages.
 +1 javac.  The applied patch does not increase the total number of javac 
 compiler warnings.
 -1 findbugs.  The patch appears to introduce 6 new Findbugs (version 
 1.3.9) warnings.
 +1 release audit.  The applied patch does not increase the total number 
 of release audit warnings.
  -1 core tests.  The patch failed these unit tests:
 {code}
 Its not apparent that any particular test is not finishing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5654) [findbugs] Address dodgy bugs

2012-04-17 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255719#comment-13255719
 ] 

Jonathan Hsieh commented on HBASE-5654:
---

There are a few match related entries:

Here's how one could likely be fixed.

{code}
  public static long calculateOverhead(long maxSize, long blockSize, int 
concurrency){
// FindBugs ICAST_INTEGER_MULTIPLY_CAST_TO_LONG
return CACHE_FIXED_OVERHEAD + ClassSize.CONCURRENT_HASHMAP +
((long)Math.ceil(maxSize*1.2/blockSize)
* ClassSize.CONCURRENT_HASHMAP_ENTRY) +
  //  (concurrency * ClassSize.CONCURRENT_HASHMAP_SEGMENT);
  ((long)concurrency * (long)ClassSize.CONCURRENT_HASHMAP_SEGMENT);
  }
{code}

For bounds checks, I think there was another where a double gets converted to a 
long (what happens is double is larger than max long?)

 [findbugs] Address dodgy bugs
 -

 Key: HBASE-5654
 URL: https://issues.apache.org/jira/browse/HBASE-5654
 Project: HBase
  Issue Type: Sub-task
  Components: scripts
Affects Versions: 0.96.0
Reporter: Jonathan Hsieh
Assignee: Ashutosh Jindal
  Labels: patch
 Fix For: 0.96.0

 Attachments: Hbase-5654.patch


 See 
 https://builds.apache.org/job/PreCommit-HBASE-Build/1313//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html#Warnings_STYLE
 This may be broken down further.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5792) HLog Performance Evaluation Tool

2012-04-16 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254650#comment-13254650
 ] 

Jonathan Hsieh commented on HBASE-5792:
---

When I ran a previous version of this I believe it wanted a hbase cluster up, 
we should make sure this works without that.

 HLog Performance Evaluation Tool
 

 Key: HBASE-5792
 URL: https://issues.apache.org/jira/browse/HBASE-5792
 Project: HBase
  Issue Type: Test
  Components: wal
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
  Labels: performance, wal
 Attachments: HBASE-5792-v0.patch, HBASE-5792-v1.patch


 Related to HDFS-3280 and the HBase WAL slowdown on 0.23+
 It would be nice to have a simple tool like HFilePerformanceEvaluation, ...
 to be able to check easily the HLog performance.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5654) [findbugs] Address dodgy bugs

2012-04-16 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254683#comment-13254683
 ] 

Jonathan Hsieh commented on HBASE-5654:
---

Ashutosh:  Thanks for taking this on.

I'm about half way through the excludes.  Just starting the INT category.

Questions:
* why no fix?: WritableRcpEngine#getProxy
* why no fix?: Ambiguous invocations in RounRobinPool/ReusablePool  (specify 
scope of get() by changing to this.get() or ReusablePool.this.get())
* add bounds checks?Result of integer multiplication cast to long in 
org.apache.hadoop.hbase.io.hfile.LruBlockCache.calculateOverhead(long, long, 
int)
Result of integer multiplication cast to long in 
org.apache.hadoop.hbase.regionserver.IncreasingToUpperBoundRegionSplitPolicy.getSizeToCheck(int)


 [findbugs] Address dodgy bugs
 -

 Key: HBASE-5654
 URL: https://issues.apache.org/jira/browse/HBASE-5654
 Project: HBase
  Issue Type: Sub-task
  Components: scripts
Affects Versions: 0.96.0
Reporter: Jonathan Hsieh
Assignee: Ashutosh Jindal
  Labels: patch
 Fix For: 0.96.0

 Attachments: Hbase-5654.patch


 See 
 https://builds.apache.org/job/PreCommit-HBASE-Build/1313//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html#Warnings_STYLE
 This may be broken down further.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5654) [findbugs] Address dodgy bugs

2012-04-16 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254802#comment-13254802
 ] 

Jonathan Hsieh commented on HBASE-5654:
---

I'm being pretty strict with excludes -- for an excludes it should be explained 
(because they will be ignored forever more.. )

{code}
+Match
+   Class name=org.apache.hadoop.hbase.regionserver.HRegion/
+  Or
+ Method name=exec /
+   /Or
+   Bug pattern=NP_LOAD_OF_KNOWN_NULL_VALUE /
+ /Match
{code}

I think this one may have been handled in anther patch -- maybe this exclude 
isn't needed.

{code}
+Match
+   Class name=org.apache.hadoop.hbase.regionserver.StoreFile$Reader/
+  Or
+ Method name=passesGeneralBloomFilter /
+   /Or
+   Bug pattern=RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE /
+ /Match
{code}

This one is definitely fixable in code instead of exclude.

{code}

+Match
+   Class name=org.apache.hadoop.hbase.filter.FilterList/
+  Or
+ Method name=filterKeyValue /
+   /Or
+   Bug pattern=SF_SWITCH_NO_DEFAULT /
+ /Match
+


+
+Match
+   Class name=org.apache.hadoop.hbase.master.AssignmentManager/
+  Or
+ Method name=processRegionsInTransition /
+Method name=handleRegion /
+   /Or
+   Bug pattern=SF_SWITCH_NO_DEFAULT /
+ /Match
+
...
+
+Match
+   Class 
name=org.apache.hadoop.hbase.master.AssignmentManager$TimeoutMonitor/
+  Or
+ Method name=actOnTimeOut /
+   /Or
+   Bug pattern=SF_SWITCH_NO_DEFAULT /
+ /Match
+
...
+
+Match
+   Class 
name=org.apache.hadoop.hbase.master.AssignmentManager$TimeoutMonitor/
+  Or
+ Method name=actOnTimeOut /
+   /Or
+   Bug pattern=SF_SWITCH_NO_DEFAULT /
+ /Match
+
...
+ Match
+   Class name=org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher/
+  Or
+ Method name=connectionEvent /
+   /Or
+   Bug pattern=SF_SWITCH_NO_DEFAULT /
+ /Match
 
{code}
Definitely fixable in code without exclude -- (add default case that does 
nothing or enumrate all cases and have default throws some sort of runtime 
exception such as IllegalStateException to catch cases where new enums created).


{code}
 Match
+   Class name=org.apache.hadoop.hbase.util.HBaseFsck/
+  Or
+ Method name=setSummary /
+ Method name=setDisplayFullReport /
+   /Or
+   Bug pattern=ST_WRITE_TO_STATIC_FROM_INSTANCE_METHOD /
+ /Match
+
{code}
File a separate jira for this one and I'll take care of it.  Please to do not 
exclude this warning.

{code}
 Match
+   Class name= org.apache.hadoop.hbase.regionserver.metrics/
+  Or
+ Method name=SchemaConfigured /
+   /Or
+   Bug pattern=NP_LOAD_OF_KNOWN_NULL_VALUE /
+ /Match
+
+Match
Class name=org.apache.hadoop.hbase.regionserver.HRegion/
{code}

Does this do anything?  metrics is a package, SchemaConfigured is a class... 

{code}
+
+Match
+   Class name=org.apache.hadoop.hbase.util.ByteBloomFilter/
+  Or
+ Method name=optimalFunctionCount /
+   /Or
+   Bug pattern=ICAST_IDIV_CAST_TO_DOUBLE /
+ /Match
+
{code}
Probably wants bitSize and maxKeys to be cased to doubles before div.






 [findbugs] Address dodgy bugs
 -

 Key: HBASE-5654
 URL: https://issues.apache.org/jira/browse/HBASE-5654
 Project: HBase
  Issue Type: Sub-task
  Components: scripts
Affects Versions: 0.96.0
Reporter: Jonathan Hsieh
Assignee: Ashutosh Jindal
  Labels: patch
 Fix For: 0.96.0

 Attachments: Hbase-5654.patch


 See 
 https://builds.apache.org/job/PreCommit-HBASE-Build/1313//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html#Warnings_STYLE
 This may be broken down further.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5798) NPE running hbck on 0.94 out of reportTablesInFlux

2012-04-16 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254843#comment-13254843
 ] 

Jonathan Hsieh commented on HBASE-5798:
---

I started a run of the unit test suite testing this fix -- for a method like 
this, I prefer returning empty arrays instead of null arrays.

{code}
diff --git src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 
src/main/java/org/apache/hadoop/hbase/cli
index ee16e72..44b7c11 100644
--- src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
+++ src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
@@ -1691,7 +1691,7 @@ public class HBaseAdmin implements Abortable, Closeable {
  /**
  * Get tableDescriptors
  * @param tableNames List of table names
- * @return HTD[] the tableDescriptor
+ * @return HTD[] the tableDescriptor (never null)
  * @throws IOException if a remote or network exception occurs
  */
   public HTableDescriptor[] getTableDescriptors(ListString tableNames)
diff --git src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 
src/main/java/org/apache/hadoop/h
index 820e2a9..f183b15 100644
--- src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
+++ src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
@@ -2195,7 +2195,7 @@ public class HConnectionManager {
 
 @Override
 public HTableDescriptor[] getHTableDescriptors(ListString tableNames) 
throws IOException {
-  if (tableNames == null || tableNames.isEmpty()) return null;
+  if (tableNames == null || tableNames.isEmpty()) return new 
HTableDescriptor[0];
   MasterKeepAliveConnection master = getKeepAliveMaster();
   try {
 return master.getHTableDescriptors(tableNames);
{code}

 NPE running hbck on 0.94 out of reportTablesInFlux
 --

 Key: HBASE-5798
 URL: https://issues.apache.org/jira/browse/HBASE-5798
 Project: HBase
  Issue Type: Bug
Reporter: stack
Assignee: Jonathan Hsieh

 Got this playing w/ hbck going against the 0.94RC:
 {code}
 12/04/16 17:03:14 INFO util.HBaseFsck: getHTableDescriptors == tableNames = 
 []
 Exception in thread main java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.util.HBaseFsck.reportTablesInFlux(HBaseFsck.java:553)
 at 
 org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:344)
 at 
 org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:380)
 at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3033)
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5798) NPE running hbck on 0.94 out of reportTablesInFlux

2012-04-16 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254862#comment-13254862
 ] 

Jonathan Hsieh commented on HBASE-5798:
---

I think #2 makes sense, but would need to be tested to verify (it is a legacy 
of the original hbck -- I didn't change this).  



 NPE running hbck on 0.94 out of reportTablesInFlux
 --

 Key: HBASE-5798
 URL: https://issues.apache.org/jira/browse/HBASE-5798
 Project: HBase
  Issue Type: Bug
  Components: hbck
Affects Versions: 0.94.0, 0.96.0
Reporter: stack
Assignee: Jonathan Hsieh

 Got this playing w/ hbck going against the 0.94RC:
 {code}
 12/04/16 17:03:14 INFO util.HBaseFsck: getHTableDescriptors == tableNames = 
 []
 Exception in thread main java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.util.HBaseFsck.reportTablesInFlux(HBaseFsck.java:553)
 at 
 org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:344)
 at 
 org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:380)
 at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3033)
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5798) NPE running hbck on 0.94 out of reportTablesInFlux

2012-04-16 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254867#comment-13254867
 ] 

Jonathan Hsieh commented on HBASE-5798:
---

Anoop -- do you guys want to take this on or should I?

 NPE running hbck on 0.94 out of reportTablesInFlux
 --

 Key: HBASE-5798
 URL: https://issues.apache.org/jira/browse/HBASE-5798
 Project: HBase
  Issue Type: Bug
  Components: hbck
Affects Versions: 0.94.0, 0.96.0
Reporter: stack
Assignee: Jonathan Hsieh

 Got this playing w/ hbck going against the 0.94RC:
 {code}
 12/04/16 17:03:14 INFO util.HBaseFsck: getHTableDescriptors == tableNames = 
 []
 Exception in thread main java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.util.HBaseFsck.reportTablesInFlux(HBaseFsck.java:553)
 at 
 org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:344)
 at 
 org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:380)
 at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3033)
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5798) NPE running hbck on 0.94 out of reportTablesInFlux

2012-04-16 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254888#comment-13254888
 ] 

Jonathan Hsieh commented on HBASE-5798:
---

Returning empty array is valid.  I dug a little into the master side as well -- 
it returns an empty array in the case where an invalid set of table names is 
passed.

 NPE running hbck on 0.94 out of reportTablesInFlux
 --

 Key: HBASE-5798
 URL: https://issues.apache.org/jira/browse/HBASE-5798
 Project: HBase
  Issue Type: Bug
  Components: hbck
Affects Versions: 0.94.0, 0.96.0
Reporter: stack
Assignee: Anoop Sam John

 Got this playing w/ hbck going against the 0.94RC:
 {code}
 12/04/16 17:03:14 INFO util.HBaseFsck: getHTableDescriptors == tableNames = 
 []
 Exception in thread main java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.util.HBaseFsck.reportTablesInFlux(HBaseFsck.java:553)
 at 
 org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:344)
 at 
 org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:380)
 at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3033)
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5781) Zookeeper session got closed while trying to assign the region to RS using hbck -fix

2012-04-15 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254287#comment-13254287
 ] 

Jonathan Hsieh commented on HBASE-5781:
---

Oh -- but it is committed to trunk as well.

 Zookeeper session got closed while trying to assign the region to RS using 
 hbck -fix
 

 Key: HBASE-5781
 URL: https://issues.apache.org/jira/browse/HBASE-5781
 Project: HBase
  Issue Type: Bug
  Components: hbck
Affects Versions: 0.90.7, 0.92.1, 0.94.0, 0.96.0
Reporter: Kristam Subba Swathi
Assignee: Jonathan Hsieh
Priority: Critical
 Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0

 Attachments: hbase-5781.patch


 After running the hbck in the cluster ,it is found that one region is not 
 assigned
 So the hbck -fix is used to fix this 
 But the assignment didnt happen since the zookeeper session is closed
 Please find the attached trace for more details
 -
 Trying to fix unassigned region...
 12/04/03 11:02:57 INFO util.HBaseFsckRepair: Region still in transition, 
 waiting for it to become assigned: {NAME = 
 'ufdr,002300,179123498.00871fbd7583512e12c4eb38e900be8d.', STARTKEY = 
 '002300', ENDKEY = '002311', ENCODED = 00871fbd7583512e12c4eb38e900be8d,}
 12/04/03 11:02:58 INFO client.HConnectionManager$HConnectionImplementation: 
 Closed zookeeper sessionid=0x236738a263a
 12/04/03 11:02:58 INFO zookeeper.ZooKeeper: Session: 0x236738a263a closed
 ERROR: Region { meta = 
 ufdr,010444,179123857.01594219211d0035b9586f98954462e1., hdfs = 
 hdfs://10.18.40.25:9000/hbase/ufdr/01594219211d0035b9586f98954462e1, deployed 
 = } not deployed on any region server.
 Trying to fix unassigned region...
 12/04/03 11:02:58 INFO zookeeper.ClientCnxn: EventThread shut down
 12/04/03 11:02:58 WARN zookeeper.ZKUtil: hconnection-0x236738a263a Unable 
 to set watcher on znode (/hbase)
 org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode 
 = Session expired for /hbase
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
 at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
 at 
 org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150)
 at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:263)
 at 
 org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.checkIfBaseNodeAvailable(ZooKeeperNodeTracker.java:208)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(HConnectionManager.java:695)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:626)
 at org.apache.hadoop.hbase.client.HBaseAdmin.getMaster(HBaseAdmin.java:211)
 at org.apache.hadoop.hbase.client.HBaseAdmin.assign(HBaseAdmin.java:1325)
 at 
 org.apache.hadoop.hbase.util.HBaseFsckRepair.forceOfflineInZK(HBaseFsckRepair.java:109)
 at 
 org.apache.hadoop.hbase.util.HBaseFsckRepair.fixUnassigned(HBaseFsckRepair.java:92)
 at 
 org.apache.hadoop.hbase.util.HBaseFsck.tryAssignmentRepair(HBaseFsck.java:1235)
 at 
 org.apache.hadoop.hbase.util.HBaseFsck.checkRegionConsistency(HBaseFsck.java:1351)
 at 
 org.apache.hadoop.hbase.util.HBaseFsck.checkAndFixConsistency(HBaseFsck.java:1114)
 at 
 org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:356)
 at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:375)
 at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:2894)
 12/04/03 11:02:58 ERROR zookeeper.ZooKeeperWatcher: 
 hconnection-0x236738a263a Received unexpected KeeperException, 
 re-throwing exception
 org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode 
 = Session expired for /hbase
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
 at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
 at 
 org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150)
 at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:263)
 at 
 org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.checkIfBaseNodeAvailable(ZooKeeperNodeTracker.java:208)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(HConnectionManager.java:695)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:626)
 at org.apache.hadoop.hbase.client.HBaseAdmin.getMaster(HBaseAdmin.java:211)
 at 

[jira] [Commented] (HBASE-5793) TestHBaseFsck#TestNoHdfsTable test hangs after HBASE-5747

2012-04-15 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254420#comment-13254420
 ] 

Jonathan Hsieh commented on HBASE-5793:
---

Zhihong yeah, I broke this out to a separate issue because I was fixing other 
TestHBaseFsck problems, wanted test to pass, but didn't yet spend the effort to 
figure out the rest of the HBASE-5747 problems.

Since I got your confirmation and since this is a minor test update, I'm just 
going to commit.

 TestHBaseFsck#TestNoHdfsTable test hangs after HBASE-5747
 -

 Key: HBASE-5793
 URL: https://issues.apache.org/jira/browse/HBASE-5793
 Project: HBase
  Issue Type: Sub-task
Affects Versions: 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Attachments: hbase-5793.patch


 After the HBASE-5747 modification, this one particular case hangs.
 {code}
 mvn test -PlocalTests -Dtest=TestHBaseFsck
 {code}
 It was hanging on a scan of a table that the test deleted. It expected a call 
 to thrown an exception after a timeout.  HBASE-5747 changed the timeout to a 
 larger number of retries which caused mvn to fail the test.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5747) Forward port hbase-5708 [89-fb] Make MiniMapRedCluster directory a subdirectory of target/test

2012-04-15 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254448#comment-13254448
 ] 

Jonathan Hsieh commented on HBASE-5747:
---

@stack Does removing the retries cause more errors to show up does it remain 
the same?  

 Forward port hbase-5708 [89-fb] Make MiniMapRedCluster directory a 
 subdirectory of target/test
 

 Key: HBASE-5747
 URL: https://issues.apache.org/jira/browse/HBASE-5747
 Project: HBase
  Issue Type: Task
Reporter: stack
Assignee: stack
Priority: Blocker
 Fix For: 0.96.0

 Attachments: 5474.txt, 5474v2.txt, 5474v3 (1).txt, 5474v3.txt, 
 5708v4.txt, 5708v4.txt


 Forward port as much as we can of Mikhail's hard-won test cleanups over on 
 0.89 branch  Will improve our being able to run unit tests in //.  He also 
 found a few bugs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5747) Forward port hbase-5708 [89-fb] Make MiniMapRedCluster directory a subdirectory of target/test

2012-04-14 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254155#comment-13254155
 ] 

Jonathan Hsieh commented on HBASE-5747:
---

Just did a git bisect and this patch seems to make TestHBaseFsck hang.  
Investigating.

 Forward port hbase-5708 [89-fb] Make MiniMapRedCluster directory a 
 subdirectory of target/test
 

 Key: HBASE-5747
 URL: https://issues.apache.org/jira/browse/HBASE-5747
 Project: HBase
  Issue Type: Task
Reporter: stack
Assignee: stack
Priority: Blocker
 Fix For: 0.96.0

 Attachments: 5474.txt, 5474v2.txt, 5474v3 (1).txt, 5474v3.txt, 
 5708v4.txt, 5708v4.txt


 Forward port as much as we can of Mikhail's hard-won test cleanups over on 
 0.89 branch  Will improve our being able to run unit tests in //.  He also 
 found a few bugs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5781) Zookeeper session got closed while trying to assign the region to RS using hbck -fix

2012-04-14 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254159#comment-13254159
 ] 

Jonathan Hsieh commented on HBASE-5781:
---

I looked into this briefly.  The version I've used on production systems 
doesn't have this finally/close portion in it.   HBASE-3777 (on 
trunk/0.92/0.94) and HBASE-4508 (backport of HBASE-3777) on 0.90 added this.  

I think if you remove the extra try-finally-close the -fix will work again, 
(but may leak resources). Can you give the modification a try?


 Zookeeper session got closed while trying to assign the region to RS using 
 hbck -fix
 

 Key: HBASE-5781
 URL: https://issues.apache.org/jira/browse/HBASE-5781
 Project: HBase
  Issue Type: Bug
  Components: hbck
Affects Versions: 0.94.0
Reporter: Kristam Subba Swathi
Assignee: Jonathan Hsieh

 After running the hbck in the cluster ,it is found that one region is not 
 assigned
 So the hbck -fix is used to fix this 
 But the assignment didnt happen since the zookeeper session is closed
 Please find the attached trace for more details
 -
 Trying to fix unassigned region...
 12/04/03 11:02:57 INFO util.HBaseFsckRepair: Region still in transition, 
 waiting for it to become assigned: {NAME = 
 'ufdr,002300,179123498.00871fbd7583512e12c4eb38e900be8d.', STARTKEY = 
 '002300', ENDKEY = '002311', ENCODED = 00871fbd7583512e12c4eb38e900be8d,}
 12/04/03 11:02:58 INFO client.HConnectionManager$HConnectionImplementation: 
 Closed zookeeper sessionid=0x236738a263a
 12/04/03 11:02:58 INFO zookeeper.ZooKeeper: Session: 0x236738a263a closed
 ERROR: Region { meta = 
 ufdr,010444,179123857.01594219211d0035b9586f98954462e1., hdfs = 
 hdfs://10.18.40.25:9000/hbase/ufdr/01594219211d0035b9586f98954462e1, deployed 
 = } not deployed on any region server.
 Trying to fix unassigned region...
 12/04/03 11:02:58 INFO zookeeper.ClientCnxn: EventThread shut down
 12/04/03 11:02:58 WARN zookeeper.ZKUtil: hconnection-0x236738a263a Unable 
 to set watcher on znode (/hbase)
 org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode 
 = Session expired for /hbase
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
 at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
 at 
 org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150)
 at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:263)
 at 
 org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.checkIfBaseNodeAvailable(ZooKeeperNodeTracker.java:208)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(HConnectionManager.java:695)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:626)
 at org.apache.hadoop.hbase.client.HBaseAdmin.getMaster(HBaseAdmin.java:211)
 at org.apache.hadoop.hbase.client.HBaseAdmin.assign(HBaseAdmin.java:1325)
 at 
 org.apache.hadoop.hbase.util.HBaseFsckRepair.forceOfflineInZK(HBaseFsckRepair.java:109)
 at 
 org.apache.hadoop.hbase.util.HBaseFsckRepair.fixUnassigned(HBaseFsckRepair.java:92)
 at 
 org.apache.hadoop.hbase.util.HBaseFsck.tryAssignmentRepair(HBaseFsck.java:1235)
 at 
 org.apache.hadoop.hbase.util.HBaseFsck.checkRegionConsistency(HBaseFsck.java:1351)
 at 
 org.apache.hadoop.hbase.util.HBaseFsck.checkAndFixConsistency(HBaseFsck.java:1114)
 at 
 org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:356)
 at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:375)
 at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:2894)
 12/04/03 11:02:58 ERROR zookeeper.ZooKeeperWatcher: 
 hconnection-0x236738a263a Received unexpected KeeperException, 
 re-throwing exception
 org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode 
 = Session expired for /hbase
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
 at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
 at 
 org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150)
 at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:263)
 at 
 org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.checkIfBaseNodeAvailable(ZooKeeperNodeTracker.java:208)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(HConnectionManager.java:695)
 at 
 

[jira] [Commented] (HBASE-5781) Zookeeper session got closed while trying to assign the region to RS using hbck -fix

2012-04-14 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254167#comment-13254167
 ] 

Jonathan Hsieh commented on HBASE-5781:
---

@Anoop.  

If you want to supply a patch that would be great.  We would definitely want to 
get this into 0.94!  Currently, there is a hanging test in trunk's 
TestHBaseFsck (HBASE-5973) that I'm hunting down, so if you can give me the a 
output of a run on 0.94 I'd be happy.

If it ends up being having a resource leak, I'd say that since hbck isn't long 
running, it would probably be ok to later as long as we noted it with a follow 
on jira.  


 Zookeeper session got closed while trying to assign the region to RS using 
 hbck -fix
 

 Key: HBASE-5781
 URL: https://issues.apache.org/jira/browse/HBASE-5781
 Project: HBase
  Issue Type: Bug
  Components: hbck
Affects Versions: 0.94.0
Reporter: Kristam Subba Swathi
Assignee: Jonathan Hsieh

 After running the hbck in the cluster ,it is found that one region is not 
 assigned
 So the hbck -fix is used to fix this 
 But the assignment didnt happen since the zookeeper session is closed
 Please find the attached trace for more details
 -
 Trying to fix unassigned region...
 12/04/03 11:02:57 INFO util.HBaseFsckRepair: Region still in transition, 
 waiting for it to become assigned: {NAME = 
 'ufdr,002300,179123498.00871fbd7583512e12c4eb38e900be8d.', STARTKEY = 
 '002300', ENDKEY = '002311', ENCODED = 00871fbd7583512e12c4eb38e900be8d,}
 12/04/03 11:02:58 INFO client.HConnectionManager$HConnectionImplementation: 
 Closed zookeeper sessionid=0x236738a263a
 12/04/03 11:02:58 INFO zookeeper.ZooKeeper: Session: 0x236738a263a closed
 ERROR: Region { meta = 
 ufdr,010444,179123857.01594219211d0035b9586f98954462e1., hdfs = 
 hdfs://10.18.40.25:9000/hbase/ufdr/01594219211d0035b9586f98954462e1, deployed 
 = } not deployed on any region server.
 Trying to fix unassigned region...
 12/04/03 11:02:58 INFO zookeeper.ClientCnxn: EventThread shut down
 12/04/03 11:02:58 WARN zookeeper.ZKUtil: hconnection-0x236738a263a Unable 
 to set watcher on znode (/hbase)
 org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode 
 = Session expired for /hbase
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
 at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
 at 
 org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150)
 at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:263)
 at 
 org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.checkIfBaseNodeAvailable(ZooKeeperNodeTracker.java:208)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(HConnectionManager.java:695)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:626)
 at org.apache.hadoop.hbase.client.HBaseAdmin.getMaster(HBaseAdmin.java:211)
 at org.apache.hadoop.hbase.client.HBaseAdmin.assign(HBaseAdmin.java:1325)
 at 
 org.apache.hadoop.hbase.util.HBaseFsckRepair.forceOfflineInZK(HBaseFsckRepair.java:109)
 at 
 org.apache.hadoop.hbase.util.HBaseFsckRepair.fixUnassigned(HBaseFsckRepair.java:92)
 at 
 org.apache.hadoop.hbase.util.HBaseFsck.tryAssignmentRepair(HBaseFsck.java:1235)
 at 
 org.apache.hadoop.hbase.util.HBaseFsck.checkRegionConsistency(HBaseFsck.java:1351)
 at 
 org.apache.hadoop.hbase.util.HBaseFsck.checkAndFixConsistency(HBaseFsck.java:1114)
 at 
 org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:356)
 at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:375)
 at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:2894)
 12/04/03 11:02:58 ERROR zookeeper.ZooKeeperWatcher: 
 hconnection-0x236738a263a Received unexpected KeeperException, 
 re-throwing exception
 org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode 
 = Session expired for /hbase
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
 at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
 at 
 org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150)
 at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:263)
 at 
 org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.checkIfBaseNodeAvailable(ZooKeeperNodeTracker.java:208)
 at 
 

[jira] [Commented] (HBASE-5747) Forward port hbase-5708 [89-fb] Make MiniMapRedCluster directory a subdirectory of target/test

2012-04-14 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254185#comment-13254185
 ] 

Jonathan Hsieh commented on HBASE-5747:
---

I've verified that the failure is due to hbase-site.xml's change.  The test 
expects a scan timeout (previously 10 retries) but likely fails out due to mvn 
timeout with 100 retries. 

{code}
diff --git a/src/test/resources/hbase-site.xml 
b/src/test/resources/hbase-site.xml
index 84b5612..e7b7112 100644
--- a/src/test/resources/hbase-site.xml
+++ b/src/test/resources/hbase-site.xml
@@ -142,4 +142,11 @@
 version is X.X.X-SNAPSHOT
 /description
   /property
+   property
+ namehbase.client.retries.number/name
+ value100/value
+ description
+ Use a lot of retries in unit tests.
+ /description
+   /property
 /configuration
{code}

Is this really necessary? 

Regardless of the outcome here, I'm going to modify the test in HBASE-5793 so 
that it checks the same thing but doesn't rely on retries.

 Forward port hbase-5708 [89-fb] Make MiniMapRedCluster directory a 
 subdirectory of target/test
 

 Key: HBASE-5747
 URL: https://issues.apache.org/jira/browse/HBASE-5747
 Project: HBase
  Issue Type: Task
Reporter: stack
Assignee: stack
Priority: Blocker
 Fix For: 0.96.0

 Attachments: 5474.txt, 5474v2.txt, 5474v3 (1).txt, 5474v3.txt, 
 5708v4.txt, 5708v4.txt


 Forward port as much as we can of Mikhail's hard-won test cleanups over on 
 0.89 branch  Will improve our being able to run unit tests in //.  He also 
 found a few bugs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5791) Apache project branding requirements: DOAP file [PATCH]

2012-04-14 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254189#comment-13254189
 ] 

Jonathan Hsieh commented on HBASE-5791:
---

DOAP = Description of a project.

 Apache project branding requirements: DOAP file [PATCH]
 ---

 Key: HBASE-5791
 URL: https://issues.apache.org/jira/browse/HBASE-5791
 Project: HBase
  Issue Type: Improvement
Reporter: Shane Curcuru
  Labels: branding
 Attachments: doap_Hbase.rdf


 Attached.  Re: http://www.apache.org/foundation/marks/pmcs
 See Also: http://projects.apache.org/create.html

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5793) TestHBaseFsck#TestNoHdfsTable test hangs after HBASE-5747

2012-04-14 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254194#comment-13254194
 ] 

Jonathan Hsieh commented on HBASE-5793:
---

patch applies cleanly on trunk/0.94, has minor conflicts in 0.90/0.92 imports 
that can be address on commit.

 TestHBaseFsck#TestNoHdfsTable test hangs after HBASE-5747
 -

 Key: HBASE-5793
 URL: https://issues.apache.org/jira/browse/HBASE-5793
 Project: HBase
  Issue Type: Sub-task
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Attachments: hbase-5793.patch


 After the HBASE-5747 modification, this one particular case hangs.
 {code}
 mvn test -PlocalTests -Dtest=TestHBaseFsck
 {code}
 It was hanging on a scan of a table that the test deleted. It expected a call 
 to thrown an exception after a timeout.  HBASE-5747 changed the timeout to a 
 larger number of retries which caused mvn to fail the test.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5781) Zookeeper session got closed while trying to assign the region to RS using hbck -fix

2012-04-14 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254206#comment-13254206
 ] 

Jonathan Hsieh commented on HBASE-5781:
---

@Lars Yeah, this is a feature regression.  I'm unit testing the suggested fix 
right now, will test on a borked testing cluster I have.

 Zookeeper session got closed while trying to assign the region to RS using 
 hbck -fix
 

 Key: HBASE-5781
 URL: https://issues.apache.org/jira/browse/HBASE-5781
 Project: HBase
  Issue Type: Bug
  Components: hbck
Affects Versions: 0.94.0, 0.96.0
Reporter: Kristam Subba Swathi
Assignee: Jonathan Hsieh
Priority: Critical
 Fix For: 0.94.0


 After running the hbck in the cluster ,it is found that one region is not 
 assigned
 So the hbck -fix is used to fix this 
 But the assignment didnt happen since the zookeeper session is closed
 Please find the attached trace for more details
 -
 Trying to fix unassigned region...
 12/04/03 11:02:57 INFO util.HBaseFsckRepair: Region still in transition, 
 waiting for it to become assigned: {NAME = 
 'ufdr,002300,179123498.00871fbd7583512e12c4eb38e900be8d.', STARTKEY = 
 '002300', ENDKEY = '002311', ENCODED = 00871fbd7583512e12c4eb38e900be8d,}
 12/04/03 11:02:58 INFO client.HConnectionManager$HConnectionImplementation: 
 Closed zookeeper sessionid=0x236738a263a
 12/04/03 11:02:58 INFO zookeeper.ZooKeeper: Session: 0x236738a263a closed
 ERROR: Region { meta = 
 ufdr,010444,179123857.01594219211d0035b9586f98954462e1., hdfs = 
 hdfs://10.18.40.25:9000/hbase/ufdr/01594219211d0035b9586f98954462e1, deployed 
 = } not deployed on any region server.
 Trying to fix unassigned region...
 12/04/03 11:02:58 INFO zookeeper.ClientCnxn: EventThread shut down
 12/04/03 11:02:58 WARN zookeeper.ZKUtil: hconnection-0x236738a263a Unable 
 to set watcher on znode (/hbase)
 org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode 
 = Session expired for /hbase
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
 at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
 at 
 org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150)
 at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:263)
 at 
 org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.checkIfBaseNodeAvailable(ZooKeeperNodeTracker.java:208)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(HConnectionManager.java:695)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:626)
 at org.apache.hadoop.hbase.client.HBaseAdmin.getMaster(HBaseAdmin.java:211)
 at org.apache.hadoop.hbase.client.HBaseAdmin.assign(HBaseAdmin.java:1325)
 at 
 org.apache.hadoop.hbase.util.HBaseFsckRepair.forceOfflineInZK(HBaseFsckRepair.java:109)
 at 
 org.apache.hadoop.hbase.util.HBaseFsckRepair.fixUnassigned(HBaseFsckRepair.java:92)
 at 
 org.apache.hadoop.hbase.util.HBaseFsck.tryAssignmentRepair(HBaseFsck.java:1235)
 at 
 org.apache.hadoop.hbase.util.HBaseFsck.checkRegionConsistency(HBaseFsck.java:1351)
 at 
 org.apache.hadoop.hbase.util.HBaseFsck.checkAndFixConsistency(HBaseFsck.java:1114)
 at 
 org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:356)
 at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:375)
 at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:2894)
 12/04/03 11:02:58 ERROR zookeeper.ZooKeeperWatcher: 
 hconnection-0x236738a263a Received unexpected KeeperException, 
 re-throwing exception
 org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode 
 = Session expired for /hbase
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
 at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
 at 
 org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150)
 at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:263)
 at 
 org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.checkIfBaseNodeAvailable(ZooKeeperNodeTracker.java:208)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(HConnectionManager.java:695)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:626)
 at org.apache.hadoop.hbase.client.HBaseAdmin.getMaster(HBaseAdmin.java:211)
 at 

[jira] [Commented] (HBASE-5781) Zookeeper session got closed while trying to assign the region to RS using hbck -fix

2012-04-14 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254241#comment-13254241
 ] 

Jonathan Hsieh commented on HBASE-5781:
---

@Anoop -- I checked the HConnection code and it seems like there should be no 
connection leaks -- the suggested patch seems clean, and doesn't need follow up 
work.  

This something I missed when I ported HBASE-5128 to trunk branches.

I'll commit this if an reviews +1 this, or if first thing Monday unless there 
are any concerns.

 Zookeeper session got closed while trying to assign the region to RS using 
 hbck -fix
 

 Key: HBASE-5781
 URL: https://issues.apache.org/jira/browse/HBASE-5781
 Project: HBase
  Issue Type: Bug
  Components: hbck
Affects Versions: 0.90.7, 0.92.1, 0.94.0, 0.96.0
Reporter: Kristam Subba Swathi
Assignee: Jonathan Hsieh
Priority: Critical
 Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0

 Attachments: hbase-5781.patch


 After running the hbck in the cluster ,it is found that one region is not 
 assigned
 So the hbck -fix is used to fix this 
 But the assignment didnt happen since the zookeeper session is closed
 Please find the attached trace for more details
 -
 Trying to fix unassigned region...
 12/04/03 11:02:57 INFO util.HBaseFsckRepair: Region still in transition, 
 waiting for it to become assigned: {NAME = 
 'ufdr,002300,179123498.00871fbd7583512e12c4eb38e900be8d.', STARTKEY = 
 '002300', ENDKEY = '002311', ENCODED = 00871fbd7583512e12c4eb38e900be8d,}
 12/04/03 11:02:58 INFO client.HConnectionManager$HConnectionImplementation: 
 Closed zookeeper sessionid=0x236738a263a
 12/04/03 11:02:58 INFO zookeeper.ZooKeeper: Session: 0x236738a263a closed
 ERROR: Region { meta = 
 ufdr,010444,179123857.01594219211d0035b9586f98954462e1., hdfs = 
 hdfs://10.18.40.25:9000/hbase/ufdr/01594219211d0035b9586f98954462e1, deployed 
 = } not deployed on any region server.
 Trying to fix unassigned region...
 12/04/03 11:02:58 INFO zookeeper.ClientCnxn: EventThread shut down
 12/04/03 11:02:58 WARN zookeeper.ZKUtil: hconnection-0x236738a263a Unable 
 to set watcher on znode (/hbase)
 org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode 
 = Session expired for /hbase
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
 at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
 at 
 org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150)
 at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:263)
 at 
 org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.checkIfBaseNodeAvailable(ZooKeeperNodeTracker.java:208)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(HConnectionManager.java:695)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:626)
 at org.apache.hadoop.hbase.client.HBaseAdmin.getMaster(HBaseAdmin.java:211)
 at org.apache.hadoop.hbase.client.HBaseAdmin.assign(HBaseAdmin.java:1325)
 at 
 org.apache.hadoop.hbase.util.HBaseFsckRepair.forceOfflineInZK(HBaseFsckRepair.java:109)
 at 
 org.apache.hadoop.hbase.util.HBaseFsckRepair.fixUnassigned(HBaseFsckRepair.java:92)
 at 
 org.apache.hadoop.hbase.util.HBaseFsck.tryAssignmentRepair(HBaseFsck.java:1235)
 at 
 org.apache.hadoop.hbase.util.HBaseFsck.checkRegionConsistency(HBaseFsck.java:1351)
 at 
 org.apache.hadoop.hbase.util.HBaseFsck.checkAndFixConsistency(HBaseFsck.java:1114)
 at 
 org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:356)
 at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:375)
 at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:2894)
 12/04/03 11:02:58 ERROR zookeeper.ZooKeeperWatcher: 
 hconnection-0x236738a263a Received unexpected KeeperException, 
 re-throwing exception
 org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode 
 = Session expired for /hbase
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
 at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
 at 
 org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150)
 at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:263)
 at 
 org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.checkIfBaseNodeAvailable(ZooKeeperNodeTracker.java:208)
 at 
 

[jira] [Commented] (HBASE-4379) [hbck] Does not complain about tables with no end region [Z,]

2012-04-13 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253152#comment-13253152
 ] 

Jonathan Hsieh commented on HBASE-4379:
---

Yes, we should.  Do you want to take a stab at it?  (If so, just take the jira) 
I need knock off some other things on my plate before I can get back to this.

 [hbck] Does not complain about tables with no end region [Z,]
 -

 Key: HBASE-4379
 URL: https://issues.apache.org/jira/browse/HBASE-4379
 Project: HBase
  Issue Type: Bug
  Components: hbck
Affects Versions: 0.90.5, 0.92.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Attachments: 
 0001-HBASE-4379-hbck-does-not-complain-about-tables-with-.patch, 
 hbase-4379.v2.patch


 hbck does not detect or have an error condition when the last region of a 
 table is missing (end key != '').

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5781) Zookeeper session got closed while trying to assign the region to RS using hbck -fix

2012-04-13 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253184#comment-13253184
 ] 

Jonathan Hsieh commented on HBASE-5781:
---

@Kristam What versions are you using?  (can you fill out the affects version?)

I actually ran into this problem earlier today and have been spending some time 
investigating.


 Zookeeper session got closed while trying to assign the region to RS using 
 hbck -fix
 

 Key: HBASE-5781
 URL: https://issues.apache.org/jira/browse/HBASE-5781
 Project: HBase
  Issue Type: Bug
  Components: hbck
Reporter: Kristam Subba Swathi
Assignee: Jonathan Hsieh

 After running the hbck in the cluster ,it is found that one region is not 
 assigned
 So the hbck -fix is used to fix this 
 But the assignment didnt happen since the zookeeper session is closed
 Please find the attached trace for more details
 -
 Trying to fix unassigned region...
 12/04/03 11:02:57 INFO util.HBaseFsckRepair: Region still in transition, 
 waiting for it to become assigned: {NAME = 
 'ufdr,002300,179123498.00871fbd7583512e12c4eb38e900be8d.', STARTKEY = 
 '002300', ENDKEY = '002311', ENCODED = 00871fbd7583512e12c4eb38e900be8d,}
 12/04/03 11:02:58 INFO client.HConnectionManager$HConnectionImplementation: 
 Closed zookeeper sessionid=0x236738a263a
 12/04/03 11:02:58 INFO zookeeper.ZooKeeper: Session: 0x236738a263a closed
 ERROR: Region { meta = 
 ufdr,010444,179123857.01594219211d0035b9586f98954462e1., hdfs = 
 hdfs://10.18.40.25:9000/hbase/ufdr/01594219211d0035b9586f98954462e1, deployed 
 = } not deployed on any region server.
 Trying to fix unassigned region...
 12/04/03 11:02:58 INFO zookeeper.ClientCnxn: EventThread shut down
 12/04/03 11:02:58 WARN zookeeper.ZKUtil: hconnection-0x236738a263a Unable 
 to set watcher on znode (/hbase)
 org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode 
 = Session expired for /hbase
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
 at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
 at 
 org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150)
 at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:263)
 at 
 org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.checkIfBaseNodeAvailable(ZooKeeperNodeTracker.java:208)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(HConnectionManager.java:695)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:626)
 at org.apache.hadoop.hbase.client.HBaseAdmin.getMaster(HBaseAdmin.java:211)
 at org.apache.hadoop.hbase.client.HBaseAdmin.assign(HBaseAdmin.java:1325)
 at 
 org.apache.hadoop.hbase.util.HBaseFsckRepair.forceOfflineInZK(HBaseFsckRepair.java:109)
 at 
 org.apache.hadoop.hbase.util.HBaseFsckRepair.fixUnassigned(HBaseFsckRepair.java:92)
 at 
 org.apache.hadoop.hbase.util.HBaseFsck.tryAssignmentRepair(HBaseFsck.java:1235)
 at 
 org.apache.hadoop.hbase.util.HBaseFsck.checkRegionConsistency(HBaseFsck.java:1351)
 at 
 org.apache.hadoop.hbase.util.HBaseFsck.checkAndFixConsistency(HBaseFsck.java:1114)
 at 
 org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:356)
 at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:375)
 at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:2894)
 12/04/03 11:02:58 ERROR zookeeper.ZooKeeperWatcher: 
 hconnection-0x236738a263a Received unexpected KeeperException, 
 re-throwing exception
 org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode 
 = Session expired for /hbase
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
 at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
 at 
 org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150)
 at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:263)
 at 
 org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.checkIfBaseNodeAvailable(ZooKeeperNodeTracker.java:208)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(HConnectionManager.java:695)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:626)
 at org.apache.hadoop.hbase.client.HBaseAdmin.getMaster(HBaseAdmin.java:211)
 at org.apache.hadoop.hbase.client.HBaseAdmin.assign(HBaseAdmin.java:1325)
 at 
 

[jira] [Commented] (HBASE-5360) [uberhbck] Add options for how to handle offline split parents.

2012-04-13 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253187#comment-13253187
 ] 

Jonathan Hsieh commented on HBASE-5360:
---

@Anoop.  Partially.  HBASE-5719 sidelines potential overlapping regions but 
doesn't take .META.'s info about if the region is offline into account.  It 
could sideline a live region while it could have been more efficient to 
sideline an offline region.  However, HBASE-5719 was sufficient for the case we 
were dealing with.  

Let's make this a placeholder for ways to improve it.  Some ideas include:
* Taking a region's offline/splitparent state into account if .META. entry is 
present
* Making decisions about sidelining vs merging based on region size and max 
region size (instead of range) xml properties.
* Improving the heuristic used to decide which region are sidelined.



 [uberhbck] Add options for how to handle offline split parents. 
 

 Key: HBASE-5360
 URL: https://issues.apache.org/jira/browse/HBASE-5360
 Project: HBase
  Issue Type: Improvement
  Components: hbck
Affects Versions: 0.90.7, 0.92.1, 0.94.0
Reporter: Jonathan Hsieh

 In a recent case, we attempted to repair a cluster that suffered from 
 HBASE-4238 that had about 6-7 generations of leftover split data.  The hbck 
 repair options in an development version of HBASE-5128 treat HDFS as ground 
 truth but didn't check SPLIT and OFFLINE flags only found in meta.  The net 
 effect was that it essentially attempted to merge many regions back into its 
 eldest geneneration's parent's range.  
 More safe guards to prevent mega-merges are being added on HBASE-5128.
 This issue would automate the handling of the mega-merge avoiding cases 
 such as lingering grandparents.  The strategy here would be to add more 
 checks against .META., and perform part of the catalog janitor's 
 responsibilities for lingering grandparents.  This would potentially include 
 options to sideline regions, deleting grandparent regions, min size for 
 sidelining, and mechanisms for cleaning .META..  
 Note: There already exists an mechanism to reload these regions -- the bulk 
 loaded mechanisms in LoadIncrementalHFiles can be used to re-add grandparents 
 (automatically splitting them if necessary) to HBase.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5488) OfflineMetaRepair doesn't support hadoop 0.20's fs.default.name property

2012-04-13 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253219#comment-13253219
 ] 

Jonathan Hsieh commented on HBASE-5488:
---

This patch shows test failures that seem unrelated , likely due to HBASE-5778.

 OfflineMetaRepair doesn't support hadoop 0.20's fs.default.name property
 

 Key: HBASE-5488
 URL: https://issues.apache.org/jira/browse/HBASE-5488
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.6
Reporter: gaojinchao
Assignee: gaojinchao
Priority: Minor
 Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0

 Attachments: HBASE-5488-branch92.patch, HBASE-5488-trunk.patch, 
 HBASE-5488_branch90.txt, hbase-5488-v2.patch


 I want to use OfflineMetaRepair tools and found onbody fix this bugs. I 
 will make a patch.
  12/01/05 23:23:30 ERROR util.HBaseFsck: Bailed out due to:
  java.lang.IllegalArgumentException: Wrong FS: hdfs:// 
  us01-ciqps1-name01.carrieriq.com:9000/hbase/M2M-INTEGRATION-MM_TION-13
  25190318714/0003d2ede27668737e192d8430dbe5d0/.regioninfo,
  expected: file:///
 at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:352)
 at
  org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:47)
 at
  org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:368)
 at
  org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
 at
  org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.init(ChecksumFileSystem.java:126)
 at
  org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:284)
 at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:398)
 at
  org.apache.hadoop.hbase.util.HBaseFsck.loadMetaEntry(HBaseFsck.java:256)
 at
  org.apache.hadoop.hbase.util.HBaseFsck.loadTableInfo(HBaseFsck.java:284)
 at
  org.apache.hadoop.hbase.util.HBaseFsck.rebuildMeta(HBaseFsck.java:402)
 at
  org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRe

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5654) [findbugs] Address dodgy bugs

2012-04-13 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253552#comment-13253552
 ] 

Jonathan Hsieh commented on HBASE-5654:
---

Hi Ashutosh, I don't see an attachement -- could you attach one that we could 
take a look at?  Thanks!

 [findbugs] Address dodgy bugs
 -

 Key: HBASE-5654
 URL: https://issues.apache.org/jira/browse/HBASE-5654
 Project: HBase
  Issue Type: Sub-task
  Components: scripts
Affects Versions: 0.96.0
Reporter: Jonathan Hsieh
Assignee: Ashutosh Jindal
  Labels: patch
 Fix For: 0.96.0


 See 
 https://builds.apache.org/job/PreCommit-HBASE-Build/1313//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html#Warnings_STYLE
 This may be broken down further.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5786) Implement histogram metrics for flush and compaction latencies and sizes.

2012-04-13 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253861#comment-13253861
 ] 

Jonathan Hsieh commented on HBASE-5786:
---


At least for the constant write load testing I'm doing currently, 1000 entries 
would are exhausted in about 8 hours.  

The histogramming currently uses reservoir sampling (with 100 slots) to keep 
metrics over all time.  Forward biasing makes more recent entries favored.

With the sampling method we could keep reasonable metrics for longer periods of 
time (weeks).

 Implement histogram metrics for flush and compaction latencies and sizes.
 -

 Key: HBASE-5786
 URL: https://issues.apache.org/jira/browse/HBASE-5786
 Project: HBase
  Issue Type: New Feature
  Components: metrics, regionserver
Affects Versions: 0.92.2, 0.94.0, 0.96.0
Reporter: Jonathan Hsieh
Assignee: Shaneal Manek

 Average time for region operations doesn't really tell a useful story when 
 that help diagnose anomalous conditions.
 It would be extremely useful to add histogramming metrics similar to 
 HBASE-5533 for region operations like flush, compaction and splitting.  The 
 probably should be forward biased at a much coarser granularity however 
 (maybe decay every day?) 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5599) [hbck] handle NO_VERSION_FILE and SHOULD_NOT_BE_DEPLOYED inconsistencies

2012-04-10 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250632#comment-13250632
 ] 

Jonathan Hsieh commented on HBASE-5599:
---

Fulin.  Great test, and nice patch.  On trunk, the failed test passes for me 
locally.  On 0.90, failed tests pass for me locally.

I plan on committing to 0.90, 0.92, 0.94, trunk.

Fixed a few spacing and a spelling nit (falut - fault) that I'll fix 
before commit.

Remaining issues: 
* Could you re-upload the same v7 patches but give ASF permission to include?
* Trunk version of patch applies to 0.94, needs a minor tweak for 0.92.  Mind 
doing this?


 [hbck] handle NO_VERSION_FILE and SHOULD_NOT_BE_DEPLOYED inconsistencies
 

 Key: HBASE-5599
 URL: https://issues.apache.org/jira/browse/HBASE-5599
 Project: HBase
  Issue Type: New Feature
  Components: hbck
Affects Versions: 0.90.6
Reporter: fulin wang
Assignee: fulin wang
 Attachments: 0.90-surefire-report-hbck.html, hbase-5599-0.90.patch, 
 hbase-5599-0.90_v2.patch, hbase-5599-0.90_v3.patch, hbase-5599-0.90_v5.patch, 
 hbase-5599-0.90_v6.patch, hbase-5599-0.90_v7.patch, hbase-5599-0.92_v5.patch, 
 hbase-5599-0.94_v5.patch, hbase-5599-trunk_v5.patch, hbase-5599-trunk_v7.patch


 The hbck tool can not fix the six scenarios.
 1. Version file does not exist in root dir.
Fix: I try to create a version file by 'FSUtils.setVersion' method.

 2. [REGIONNAME][KEY] on HDFS, but not listed in META or deployed on any 
 region server.
Fix: I get region info form the hdfs file, this region info write to 
 '.META.' table.

 3. [REGIONNAME][KEY] not in META, but deployed on [SERVERNAME]
Fix: I get region info form the hdfs file, this region info write to 
 '.META.' table.

 4. [REGIONNAME] should not be deployed according to META, but is deployed on 
 [SERVERNAME]
Fix: Close this region.

 5. First region should start with an empty key.  You need to  create a new 
 region and regioninfo in HDFS to plug the hole.
Fix: The region info is not in hdfs and .META., so it create a empty 
 region for this error.
 6. There is a hole in the region chain between [KEY] and [KEY]. You need to 
 create a new regioninfo and region dir in hdfs to plug the hole.
   Fix: The region info is not in hdfs and .META., so it create a empty region 
 for this hole.
   

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5653) [findbugs] fix perf warnings

2012-04-10 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250654#comment-13250654
 ] 

Jonathan Hsieh commented on HBASE-5653:
---

Looks good to me.  One suggestion:

{code}
 totalReplicated++;
   }
-  for(byte [] table : rows.keySet()) {
-batch(table, rows.get(table));
+  for (IteratorEntrybyte[], ListRow iterator = rows.entrySet()
+  .iterator(); iterator.hasNext();) {
+Entrybyte[], ListRow keyValues = iterator.next();
+batch(keyValues.getKey(), keyValues.getValue());
   }
{code}

Maybe write this way instead:

{code}
for (Entrybyte[], ListRow entry : rows.entrySet()) {
  batch(entry.getKey(), entry.getValue());
}
{code}

 [findbugs] fix perf warnings
 

 Key: HBASE-5653
 URL: https://issues.apache.org/jira/browse/HBASE-5653
 Project: HBase
  Issue Type: Sub-task
  Components: scripts
Reporter: Jonathan Hsieh
Assignee: Uma Maheswara Rao G
 Attachments: HBASE-5653.patch


 See 
 https://builds.apache.org/job/PreCommit-HBASE-Build/1313//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html#Warnings_PERFORMANCE

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4336) Convert source tree into maven modules

2012-04-10 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13251200#comment-13251200
 ] 

Jonathan Hsieh commented on HBASE-4336:
---

The original flume project had the current hadoop trunk/0.23 branches have the 
{component}-assembly broken out.  I think that is canonical for maven.  I'll 
ping some of the local mavenistas for their opinion.

+1 to doing something coarse first.  

At least in flume we found it really easy for plugins (like hbase's 
co-processors) to be compiled and tested separately.



 Convert source tree into maven modules
 --

 Key: HBASE-4336
 URL: https://issues.apache.org/jira/browse/HBASE-4336
 Project: HBase
  Issue Type: Task
  Components: build
Reporter: Gary Helmling
Priority: Critical
 Fix For: 0.96.0


 When we originally converted the build to maven we had a single core module 
 defined, but later reverted this to a module-less build for the sake of 
 simplicity.
 It now looks like it's time to re-address this, as we have an actual need for 
 modules to:
 * provide a trimmed down client library that applications can make use of
 * more cleanly support building against different versions of Hadoop, in 
 place of some of the reflection machinations currently required
 * incorporate the secure RPC engine that depends on some secure Hadoop classes
 I propose we start simply by refactoring into two initial modules:
 * core - common classes and utilities, and client-side code and interfaces
 * server - master and region server implementations and supporting code
 This would also lay the groundwork for incorporating the HBase security 
 features that have been developed.  Once the module structure is in place, 
 security-related features could then be incorporated into a third module -- 
 security -- after normal review and approval.  The security module could 
 then depend on secure Hadoop, without modifying the dependencies of the rest 
 of the HBase code.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5599) [hbck] handle NO_VERSION_FILE and SHOULD_NOT_BE_DEPLOYED inconsistencies

2012-04-10 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13251316#comment-13251316
 ] 

Jonathan Hsieh commented on HBASE-5599:
---

I can take care of the tweak for 0.92.

For me to commit the patches you've submitted, you need to Grant license to 
ASF for inclusion in ASF works (as per the Apache License ยง5).  When you 
attach files, there is a selector box at the bottom, and you need to select 
that.

 [hbck] handle NO_VERSION_FILE and SHOULD_NOT_BE_DEPLOYED inconsistencies
 

 Key: HBASE-5599
 URL: https://issues.apache.org/jira/browse/HBASE-5599
 Project: HBase
  Issue Type: New Feature
  Components: hbck
Affects Versions: 0.90.6
Reporter: fulin wang
Assignee: fulin wang
 Attachments: 0.90-surefire-report-hbck.html, hbase-5599-0.90.patch, 
 hbase-5599-0.90_v2.patch, hbase-5599-0.90_v3.patch, hbase-5599-0.90_v5.patch, 
 hbase-5599-0.90_v6.patch, hbase-5599-0.90_v7.patch, hbase-5599-0.92_v5.patch, 
 hbase-5599-0.94_v5.patch, hbase-5599-trunk_v5.patch, 
 hbase-5599-trunk_v7.patch, license.png


 The hbck tool can not fix the six scenarios.
 1. Version file does not exist in root dir.
Fix: I try to create a version file by 'FSUtils.setVersion' method.

 2. [REGIONNAME][KEY] on HDFS, but not listed in META or deployed on any 
 region server.
Fix: I get region info form the hdfs file, this region info write to 
 '.META.' table.

 3. [REGIONNAME][KEY] not in META, but deployed on [SERVERNAME]
Fix: I get region info form the hdfs file, this region info write to 
 '.META.' table.

 4. [REGIONNAME] should not be deployed according to META, but is deployed on 
 [SERVERNAME]
Fix: Close this region.

 5. First region should start with an empty key.  You need to  create a new 
 region and regioninfo in HDFS to plug the hole.
Fix: The region info is not in hdfs and .META., so it create a empty 
 region for this error.
 6. There is a hole in the region chain between [KEY] and [KEY]. You need to 
 create a new regioninfo and region dir in hdfs to plug the hole.
   Fix: The region info is not in hdfs and .META., so it create a empty region 
 for this hole.
   

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5645) [findbugs] Fix correctness warnings

2012-04-10 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13251330#comment-13251330
 ] 

Jonathan Hsieh commented on HBASE-5645:
---

lgtm. 

I just checked in HBASE_5653 that reduced the find bugs number, which likely 
breaks this patch.  I'm going to submit patch to find out the new number and 
the tweak v2 to reflect this new number.

 [findbugs] Fix correctness warnings
 ---

 Key: HBASE-5645
 URL: https://issues.apache.org/jira/browse/HBASE-5645
 Project: HBase
  Issue Type: Sub-task
  Components: scripts
Reporter: Jonathan Hsieh
Assignee: David S. Wang
 Attachments: HBASE-5645-2.patch, HBASE-5645.patch, HBASE-5645.patch


 See 
 https://builds.apache.org/job/PreCommit-HBASE-Build/1313//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
 Fix the warnings in the correctness section.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5645) [findbugs] Fix correctness warnings

2012-04-10 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13251331#comment-13251331
 ] 

Jonathan Hsieh commented on HBASE-5645:
---

lgtm. 

I just checked in HBASE_5653 that reduced the find bugs number, which likely 
breaks this patch.  I'm going to submit patch to find out the new number and 
the tweak v2 to reflect this new number.

 [findbugs] Fix correctness warnings
 ---

 Key: HBASE-5645
 URL: https://issues.apache.org/jira/browse/HBASE-5645
 Project: HBase
  Issue Type: Sub-task
  Components: scripts
Reporter: Jonathan Hsieh
Assignee: David S. Wang
 Attachments: HBASE-5645-2.patch, HBASE-5645.patch, HBASE-5645.patch


 See 
 https://builds.apache.org/job/PreCommit-HBASE-Build/1313//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
 Fix the warnings in the correctness section.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5213) hbase master stop does not bring down backup masters

2012-04-08 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249661#comment-13249661
 ] 

Jonathan Hsieh commented on HBASE-5213:
---

The issue is apparently only in 0.90.  How about we close this issue for 
0.92/0.94/trunk and create a follow on issue for 0.90?  This will unblock this 
for 0.94 and Greg can address this when he gets back.

 hbase master stop does not bring down backup masters
 --

 Key: HBASE-5213
 URL: https://issues.apache.org/jira/browse/HBASE-5213
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.5, 0.92.0, 0.94.0, 0.96.0
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Priority: Minor
 Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0

 Attachments: 5213.jstack, HBASE-5213-v0-trunk.patch, 
 HBASE-5213-v1-trunk.patch, HBASE-5213-v2-90.patch, HBASE-5213-v2-92.patch, 
 HBASE-5213-v2-trunk.patch


 Typing hbase master stop produces the following message:
 stop   Start cluster shutdown; Master signals RegionServer shutdown
 It seems like backup masters should be considered part of the cluster, but 
 they are not brought down by hbase master stop.
 stop-hbase.sh does correctly bring down the backup masters.
 The same behavior is observed when a client app makes use of the client API 
 HBaseAdmin.shutdown() 
 http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html#shutdown()
  -- this isn't too surprising since I think hbase master stop just calls 
 this API.
 It seems like HBASE-1448 address this; perhaps there was a regression?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5680) Hbase94 and Hbase 92.2 is not compatible with the Hadoop 23.1

2012-04-06 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13248103#comment-13248103
 ] 

Jonathan Hsieh commented on HBASE-5680:
---

How to test:

* Setup hadoop hdfs 0.23.x cluster.
* Compile against 1.0.x hadoop. {{mvn clean package -DskipTests}}
* Make HBASE_HOME ./target/hbase-version/hbase-version
* Replace hadoop jars in ./target/hbase-version/hbase-version/lib/ with hadoop 
23 jars.
* start hbase.
* Look at master log.


 Hbase94 and Hbase 92.2 is not compatible with the Hadoop 23.1 
 --

 Key: HBASE-5680
 URL: https://issues.apache.org/jira/browse/HBASE-5680
 Project: HBase
  Issue Type: Bug
  Components: master
Reporter: Kristam Subba Swathi
Assignee: Jonathan Hsieh
 Attachments: hbase-5680-94.patch


 Hmaster is not able to start because of the following error
 Please find the following error 
 
 2012-03-30 11:12:19,487 FATAL org.apache.hadoop.hbase.master.HMaster: 
 Unhandled exception. Starting shutdown.
 java.lang.NoClassDefFoundError: 
 org/apache/hadoop/hdfs/protocol/FSConstants$SafeModeAction
   at org.apache.hadoop.hbase.util.FSUtils.waitOnSafeMode(FSUtils.java:524)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:324)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:127)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.init(MasterFileSystem.java:112)
   at 
 org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:496)
   at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:363)
   at java.lang.Thread.run(Thread.java:662)
 Caused by: java.lang.ClassNotFoundException: 
 org.apache.hadoop.hdfs.protocol.FSConstants$SafeModeAction
   at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
   ... 7 more
 There is a change in the FSConstants

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5680) Hbase94 and Hbase 92.2 is not compatible with the Hadoop 23.1

2012-04-06 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13248104#comment-13248104
 ] 

Jonathan Hsieh commented on HBASE-5680:
---

How to test:

* Setup hadoop hdfs 0.23.x cluster.
* Compile against 1.0.x hadoop. {{mvn clean package -DskipTests}}
* Make HBASE_HOME ./target/hbase-version/hbase-version
* Replace hadoop jars in ./target/hbase-version/hbase-version/lib/ with hadoop 
23 jars.
* start hbase.
* Look at master log.


 Hbase94 and Hbase 92.2 is not compatible with the Hadoop 23.1 
 --

 Key: HBASE-5680
 URL: https://issues.apache.org/jira/browse/HBASE-5680
 Project: HBase
  Issue Type: Bug
  Components: master
Reporter: Kristam Subba Swathi
Assignee: Jonathan Hsieh
 Attachments: hbase-5680-94.patch


 Hmaster is not able to start because of the following error
 Please find the following error 
 
 2012-03-30 11:12:19,487 FATAL org.apache.hadoop.hbase.master.HMaster: 
 Unhandled exception. Starting shutdown.
 java.lang.NoClassDefFoundError: 
 org/apache/hadoop/hdfs/protocol/FSConstants$SafeModeAction
   at org.apache.hadoop.hbase.util.FSUtils.waitOnSafeMode(FSUtils.java:524)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:324)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:127)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.init(MasterFileSystem.java:112)
   at 
 org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:496)
   at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:363)
   at java.lang.Thread.run(Thread.java:662)
 Caused by: java.lang.ClassNotFoundException: 
 org.apache.hadoop.hdfs.protocol.FSConstants$SafeModeAction
   at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
   ... 7 more
 There is a change in the FSConstants

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5680) Hbase94 and Hbase 92.2 is not compatible with the Hadoop 23.1

2012-04-06 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13248107#comment-13248107
 ] 

Jonathan Hsieh commented on HBASE-5680:
---

Patch applies on 0.90/0.92/0.94/0.96.

 Hbase94 and Hbase 92.2 is not compatible with the Hadoop 23.1 
 --

 Key: HBASE-5680
 URL: https://issues.apache.org/jira/browse/HBASE-5680
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.6, 0.92.1, 0.94.0, 0.96.0
Reporter: Kristam Subba Swathi
Assignee: Jonathan Hsieh
 Attachments: hbase-5680-94.patch


 Hmaster is not able to start because of the following error
 Please find the following error 
 
 2012-03-30 11:12:19,487 FATAL org.apache.hadoop.hbase.master.HMaster: 
 Unhandled exception. Starting shutdown.
 java.lang.NoClassDefFoundError: 
 org/apache/hadoop/hdfs/protocol/FSConstants$SafeModeAction
   at org.apache.hadoop.hbase.util.FSUtils.waitOnSafeMode(FSUtils.java:524)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:324)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:127)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.init(MasterFileSystem.java:112)
   at 
 org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:496)
   at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:363)
   at java.lang.Thread.run(Thread.java:662)
 Caused by: java.lang.ClassNotFoundException: 
 org.apache.hadoop.hdfs.protocol.FSConstants$SafeModeAction
   at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
   ... 7 more
 There is a change in the FSConstants

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5680) Hbase94 and Hbase 92.2 is not compatible with the Hadoop 23.1

2012-04-06 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13248128#comment-13248128
 ] 

Jonathan Hsieh commented on HBASE-5680:
---

if you look the patch, it takes the HadoopVersionAnnotation value that's 
compiled into the hadoop jar that is linked in. I tested against CDH Hadoop 23 
in this case but if you use apache Hadoop 0.23.1 it will say that.

Sent from my iPhone




 Hbase94 and Hbase 92.2 is not compatible with the Hadoop 23.1 
 --

 Key: HBASE-5680
 URL: https://issues.apache.org/jira/browse/HBASE-5680
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.6, 0.92.1, 0.94.0, 0.96.0
Reporter: Kristam Subba Swathi
Assignee: Jonathan Hsieh
 Attachments: hbase-5680-94.patch


 Hmaster is not able to start because of the following error
 Please find the following error 
 
 2012-03-30 11:12:19,487 FATAL org.apache.hadoop.hbase.master.HMaster: 
 Unhandled exception. Starting shutdown.
 java.lang.NoClassDefFoundError: 
 org/apache/hadoop/hdfs/protocol/FSConstants$SafeModeAction
   at org.apache.hadoop.hbase.util.FSUtils.waitOnSafeMode(FSUtils.java:524)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:324)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:127)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.init(MasterFileSystem.java:112)
   at 
 org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:496)
   at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:363)
   at java.lang.Thread.run(Thread.java:662)
 Caused by: java.lang.ClassNotFoundException: 
 org.apache.hadoop.hdfs.protocol.FSConstants$SafeModeAction
   at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
   ... 7 more
 There is a change in the FSConstants

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5213) hbase master stop does not bring down backup masters

2012-04-06 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13248467#comment-13248467
 ] 

Jonathan Hsieh commented on HBASE-5213:
---

I'll revert the 0.90 version.  Sorry about this fellas.

 hbase master stop does not bring down backup masters
 --

 Key: HBASE-5213
 URL: https://issues.apache.org/jira/browse/HBASE-5213
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.5, 0.92.0, 0.94.0, 0.96.0
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Priority: Minor
 Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0

 Attachments: 5213.jstack, HBASE-5213-v0-trunk.patch, 
 HBASE-5213-v1-trunk.patch, HBASE-5213-v2-90.patch, HBASE-5213-v2-92.patch, 
 HBASE-5213-v2-trunk.patch


 Typing hbase master stop produces the following message:
 stop   Start cluster shutdown; Master signals RegionServer shutdown
 It seems like backup masters should be considered part of the cluster, but 
 they are not brought down by hbase master stop.
 stop-hbase.sh does correctly bring down the backup masters.
 The same behavior is observed when a client app makes use of the client API 
 HBaseAdmin.shutdown() 
 http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html#shutdown()
  -- this isn't too surprising since I think hbase master stop just calls 
 this API.
 It seems like HBASE-1448 address this; perhaps there was a regression?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5599) The hbck tool can not fix the six scenarios, it is NO_VERSION_FILE, NOT_IN_META_OR_DEPLOYED, NOT_IN_META, SHOULD_NOT_BE_DEPLOYED, FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_

2012-04-06 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13248485#comment-13248485
 ] 

Jonathan Hsieh commented on HBASE-5599:
---

Can you share the test code?  Extra sets of eyes might help find the problem!

 The hbck tool can not fix the six scenarios, it is NO_VERSION_FILE, 
 NOT_IN_META_OR_DEPLOYED, NOT_IN_META, SHOULD_NOT_BE_DEPLOYED, 
 FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_IN_REGION_CHAIN.
 

 Key: HBASE-5599
 URL: https://issues.apache.org/jira/browse/HBASE-5599
 Project: HBase
  Issue Type: New Feature
  Components: hbck
Affects Versions: 0.90.6
Reporter: fulin wang
Assignee: fulin wang
 Fix For: 0.90.6

 Attachments: hbase-5599-0.90.patch, hbase-5599-0.90_v2.patch, 
 hbase-5599-0.90_v3.patch, hbase-5599-0.90_v5.patch, hbase-5599-0.90_v6.patch, 
 hbase-5599-0.92_v5.patch, hbase-5599-0.94_v5.patch, hbase-5599-trunk_v5.patch


 The hbck tool can not fix the six scenarios.
 1. Version file does not exist in root dir.
Fix: I try to create a version file by 'FSUtils.setVersion' method.

 2. [REGIONNAME][KEY] on HDFS, but not listed in META or deployed on any 
 region server.
Fix: I get region info form the hdfs file, this region info write to 
 '.META.' table.

 3. [REGIONNAME][KEY] not in META, but deployed on [SERVERNAME]
Fix: I get region info form the hdfs file, this region info write to 
 '.META.' table.

 4. [REGIONNAME] should not be deployed according to META, but is deployed on 
 [SERVERNAME]
Fix: Close this region.

 5. First region should start with an empty key.  You need to  create a new 
 region and regioninfo in HDFS to plug the hole.
Fix: The region info is not in hdfs and .META., so it create a empty 
 region for this error.
 6. There is a hole in the region chain between [KEY] and [KEY]. You need to 
 create a new regioninfo and region dir in hdfs to plug the hole.
   Fix: The region info is not in hdfs and .META., so it create a empty region 
 for this hole.
   

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5734) Change hbck sideline root

2012-04-06 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249004#comment-13249004
 ] 

Jonathan Hsieh commented on HBASE-5734:
---

Can you followup by writing an example up about what problem this solves in a 
little more detail?  This will help a lot when I write the documentation.  

I'm assuming the failed test is a flakey.  Patch look good to me.


 Change hbck sideline root
 -

 Key: HBASE-5734
 URL: https://issues.apache.org/jira/browse/HBASE-5734
 Project: HBase
  Issue Type: Improvement
  Components: hbck
Affects Versions: 0.94.0, 0.96.0
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Trivial
 Fix For: 0.96.0

 Attachments: hbase-5734.patch


 Currently hbck sideline root is the root which can run into permission issue. 
 We can change it to /hbck

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5734) Change hbck sideline root

2012-04-06 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249066#comment-13249066
 ] 

Jonathan Hsieh commented on HBASE-5734:
---

Docs look great Jimmy, thanks.

Failed test passed fo rme.

Committed to 0.90/0.92/0.94/trunk

 Change hbck sideline root
 -

 Key: HBASE-5734
 URL: https://issues.apache.org/jira/browse/HBASE-5734
 Project: HBase
  Issue Type: Improvement
  Components: hbck
Affects Versions: 0.94.0, 0.96.0
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Trivial
 Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0

 Attachments: hbase-5734-90.patch, hbase-5734.patch


 Currently hbck sideline root is the root which can run into permission issue. 
 We can change it to /hbck

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5644) [findbugs] Fix null pointer warnings.

2012-04-05 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247097#comment-13247097
 ] 

Jonathan Hsieh commented on HBASE-5644:
---

First quick review (I have more to review, will continue hopefully in next day)

Test failure on TestSplitTransactionOnCluster seems to be legit -- it hung on 
my machine locally 2x.  

Have you used Preconditions before?  It is slightly more compact and seems  
little better at conveying intent.  
http://google-collections.googlecode.com/svn/trunk/javadoc/com/google/common/base/Preconditions.html

There was another NP findbugs warnings from build 1365:  Mind handling this one 
too?
Possible null pointer dereference of serverInfo in 
org.apache.hadoop.hbase.tmpl.regionserver.RSStatusTmplImpl.renderNoFlush(Writer)
 on exception path


 [findbugs] Fix null pointer warnings.
 -

 Key: HBASE-5644
 URL: https://issues.apache.org/jira/browse/HBASE-5644
 Project: HBase
  Issue Type: Sub-task
  Components: scripts
Reporter: Jonathan Hsieh
Assignee: Uma Maheswara Rao G
 Attachments: HBASE-5644.patch, NullPointerFindBugs_Analysis.xlsx


 See 
 https://builds.apache.org/job/PreCommit-HBASE-Build/1313//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
 Fix the NP category

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5644) [findbugs] Fix null pointer warnings.

2012-04-05 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247318#comment-13247318
 ] 

Jonathan Hsieh commented on HBASE-5644:
---

Code portion looks good to me.

On the spread sheet, one suggestion:

HTable - row 2, 3
 - delete -- why is the type bool?  Maybre change to ServerCallableVoid? (two 
cases).

Ex:
{code}
  @Override
  public void delete(final Delete delete)
  throws IOException {
new ServerCallableVoid(connection, tableName, delete.getRow(), 
operationTimeout) {
  public Void call() throws IOException {
server.delete(location.getRegionInfo().getRegionName(), delete);
return null; // FindBugs NP_BOOLEAN_RETURN_NULL
  }
}.withRetries();
  }
{code}

4,5,6, 7 smells funny but I buy it. 

Seems like findbugs doesn't handle ?: very well.

bq. Used, but not extensively ( only in Hadoop). are you suggesting not to use 
it here?

What I wrote above was unclear.  You use Preconditions in some places (Store), 
and there are places you don't (ShutdownHook).  Seems like you could us it in a 
few more places?  Not a big deal, but it makes code easier to read by conveying 
more intent IMO.  

Maybe you chose not to use because it wasn't at the top of a method?  

bq. RSStatusTmplImpl is Autogenerated Jamon code. I wanted to fix it but I am 
not familiar in Jamon code generation area . Let some one or you can update 
this change if you are familiar. If not, can I file a separate bug?

You can get this one, it is pretty straightforward.  The source of the autogen 
RSStatusTmplImpl data is here.  Take a look, just modify there and it will just 
percolate that code through to the java version.

https://github.com/apache/hbase/blob/trunk/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon#L48





 [findbugs] Fix null pointer warnings.
 -

 Key: HBASE-5644
 URL: https://issues.apache.org/jira/browse/HBASE-5644
 Project: HBase
  Issue Type: Sub-task
  Components: scripts
Reporter: Jonathan Hsieh
Assignee: Uma Maheswara Rao G
 Attachments: HBASE-5644.patch, NullPointerFindBugs_Analysis.xlsx


 See 
 https://builds.apache.org/job/PreCommit-HBASE-Build/1313//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
 Fix the NP category

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5644) [findbugs] Fix null pointer warnings.

2012-04-05 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247811#comment-13247811
 ] 

Jonathan Hsieh commented on HBASE-5644:
---

Oh yeah, it is nice to see hadoop qa not complain when it ran your patch!

 [findbugs] Fix null pointer warnings.
 -

 Key: HBASE-5644
 URL: https://issues.apache.org/jira/browse/HBASE-5644
 Project: HBase
  Issue Type: Sub-task
  Components: scripts
Reporter: Jonathan Hsieh
Assignee: Uma Maheswara Rao G
 Fix For: 0.96.0

 Attachments: HBASE-5644.patch, HBASE-5644.patch, 
 NullPointerFindBugs_Analysis.xlsx


 See 
 https://builds.apache.org/job/PreCommit-HBASE-Build/1313//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
 Fix the NP category

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5719) Enhance hbck to sideline overlapped mega regions

2012-04-05 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247865#comment-13247865
 ] 

Jonathan Hsieh commented on HBASE-5719:
---

More context:

We ran into a corrupted cluster that had encountered HBASE-4238 and had several 
generations of grandparent and regions lingering in HDFS.  If you looked at a 
region map, we had overlapping regions that looked like this:

[A-I], [A-E], [E-H], [A-C], [A-B], [B-C] ... 

The HBASE-5128 version of hbck would see that all these regions fit inside of 
A-I and then attempt to merge the all into one mega region.  This is 
technically correct but could result merging all the regions in an overlap 
group into one region that was significantly larger than all others (worst case 
all regions of a table could get combined into one region).  HBASE-5128 
includes some safeguards to prevent these mega merges.  In order to fix these 
situations, we sidelined (close, offline, move to different dir) the 
grandparent regions with the largest overlapped with the most other regions.  
This leaves us with many small groups of overlapping regions instead of a 
single large set of overlapping regions.  These smaller regions could be safely 
repaired automatically via merges, and any data from the sidelined grandparent 
regions could be restored via a bulk load later on.

So in the example above, the [A-I], [A-E], [E-H] grandparent regions would get 
sidelined, and leaving us with [A-C], [A-B],[B-C].  These smaller regions could 
get safely merged automatically into a single [A-C]' region.  We'd then bulk 
load [A-I], [A-E], and [E-H] regions back in afterwards to restore data.

The goal of this patch is to automatically id and sideline overlapping 
grandparent regions. 


 Enhance hbck to sideline overlapped mega regions
 

 Key: HBASE-5719
 URL: https://issues.apache.org/jira/browse/HBASE-5719
 Project: HBase
  Issue Type: New Feature
  Components: hbck
Affects Versions: 0.94.0, 0.96.0
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0

 Attachments: hbase-5719.patch


 If there are too many regions in one overlapped group (by default, more than 
 10), hbck currently doesn't merge them since it takes time.
 In this case, we can sideline some regions in the group and break the 
 overlapping to fix the inconsistency.  Later on, sidelined regions can be 
 bulk loaded manually.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5599) The hbck tool can not fix the six scenarios, it is NO_VERSION_FILE, NOT_IN_META_OR_DEPLOYED, NOT_IN_META, SHOULD_NOT_BE_DEPLOYED, FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_

2012-04-05 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247911#comment-13247911
 ] 

Jonathan Hsieh commented on HBASE-5599:
---

Have you encountered this in your cluster before?  (I actually have not)

To test it properly, you would probably disable a table and then do some funny 
direct calls to a region server to set up a the funky internal state.  I 
actually spent a lot of time getting tests working for hbck but it definitely  
caught some important cornercases.

I've suggested this before, but I'd be happy to commit the versionFile part and 
push off this SHOULD_NOT_BE_DEPLOYED case to another issue. (Also, still 
waiting for an answer about the Merge.java edits).

 The hbck tool can not fix the six scenarios, it is NO_VERSION_FILE, 
 NOT_IN_META_OR_DEPLOYED, NOT_IN_META, SHOULD_NOT_BE_DEPLOYED, 
 FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_IN_REGION_CHAIN.
 

 Key: HBASE-5599
 URL: https://issues.apache.org/jira/browse/HBASE-5599
 Project: HBase
  Issue Type: New Feature
  Components: hbck
Affects Versions: 0.90.6
Reporter: fulin wang
Assignee: fulin wang
 Fix For: 0.90.6

 Attachments: hbase-5599-0.90.patch, hbase-5599-0.90_v2.patch, 
 hbase-5599-0.90_v3.patch, hbase-5599-0.90_v5.patch, hbase-5599-0.90_v6.patch, 
 hbase-5599-0.92_v5.patch, hbase-5599-0.94_v5.patch, hbase-5599-trunk_v5.patch


 The hbck tool can not fix the six scenarios.
 1. Version file does not exist in root dir.
Fix: I try to create a version file by 'FSUtils.setVersion' method.

 2. [REGIONNAME][KEY] on HDFS, but not listed in META or deployed on any 
 region server.
Fix: I get region info form the hdfs file, this region info write to 
 '.META.' table.

 3. [REGIONNAME][KEY] not in META, but deployed on [SERVERNAME]
Fix: I get region info form the hdfs file, this region info write to 
 '.META.' table.

 4. [REGIONNAME] should not be deployed according to META, but is deployed on 
 [SERVERNAME]
Fix: Close this region.

 5. First region should start with an empty key.  You need to  create a new 
 region and regioninfo in HDFS to plug the hole.
Fix: The region info is not in hdfs and .META., so it create a empty 
 region for this error.
 6. There is a hole in the region chain between [KEY] and [KEY]. You need to 
 create a new regioninfo and region dir in hdfs to plug the hole.
   Fix: The region info is not in hdfs and .META., so it create a empty region 
 for this hole.
   

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5711) Tests are failing with incorrect data directory permissions.

2012-04-05 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247923#comment-13247923
 ] 

Jonathan Hsieh commented on HBASE-5711:
---

I've been working around this by setting 'umask 022' in my shell before 
executing tests/eclipse, etc.

 Tests are failing with incorrect data directory permissions.
 

 Key: HBASE-5711
 URL: https://issues.apache.org/jira/browse/HBASE-5711
 Project: HBase
  Issue Type: Bug
  Components: test
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
 Fix For: 0.92.2, 0.94.0

 Attachments: HBASE-5711.patch


 When we run some tests in Hbase (TestAdmin), it is failing with following 
 error.
 {quote}
 Starting DataNode 0 with dfs.data.dir: 
 E:\Repositories\Hbase\target\test-data\5ff23198-892e-4f1c-8022-b3d9969fcf0b\dfscluster_0ecc6984-1925-4870-ac7c-439fceede4cb\dfs\data\data1,E:\Repositories\Hbase\target\test-data\5ff23198-892e-4f1c-8022-b3d9969fcf0b\dfscluster_0ecc6984-1925-4870-ac7c-439fceede4cb\dfs\data\data2
 2012-04-04 18:04:51,036 WARN  [main] impl.MetricsSystemImpl(137): Metrics 
 system not started: Cannot locate configuration: tried 
 hadoop-metrics2-datanode.properties, hadoop-metrics2.properties
 2012-04-04 18:04:51,255 WARN  [main] datanode.DataNode(1548): Invalid 
 directory in dfs.data.dir: Incorrect permission for 
 E:/Repositories/Hbase/target/test-data/5ff23198-892e-4f1c-8022-b3d9969fcf0b/dfscluster_0ecc6984-1925-4870-ac7c-439fceede4cb/dfs/data/data1,
  expected: rwxr-xr-x, while actual: rwx--
 2012-04-04 18:04:51,411 WARN  [main] datanode.DataNode(1548): Invalid 
 directory in dfs.data.dir: Incorrect permission for 
 E:/Repositories/Hbase/target/test-data/5ff23198-892e-4f1c-8022-b3d9969fcf0b/dfscluster_0ecc6984-1925-4870-ac7c-439fceede4cb/dfs/data/data2,
  expected: rwxr-xr-x, while actual: rwx--
 2012-04-04 18:04:51,411 ERROR [main] datanode.DataNode(1554): All directories 
 in dfs.data.dir are invalid.
 2012-04-04 18:04:51,411 INFO  [main] hbase.HBaseTestingUtility(684): Shutting 
 down minicluster
 2012-04-04 18:04:51,646 WARN  [main] hbase.HBaseTestingUtility(696): Failed 
 delete of 
 E:\Repositories\Hbase\target\test-data\5ff23198-892e-4f1c-8022-b3d9969fcf0b\dfscluster_0ecc6984-1925-4870-ac7c-439fceede4cb
 2012-04-04 18:04:51,646 INFO  [main] hbase.HBaseTestingUtility(700): 
 Minicluster is down
 {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4348) Add metrics for regions in transition

2012-04-05 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247981#comment-13247981
 ] 

Jonathan Hsieh commented on HBASE-4348:
---

You probably need to update/fetch again.  trunk lives in svn and has a 
different style for commit numbers.  The hash you gave is git -- if you are 
using git.apache.org it often lags and doesn't have the most recent patches 
that svn has.

Looks like the conflict was with HBASE-5715 Revert 'Instant schema alter' for 
now, HBASE-4213.  It's minor so I've updated the patch.  We'll let the bot 
test the slightly updated version and if it is happy I'll commit.


 Add metrics for regions in transition
 -

 Key: HBASE-4348
 URL: https://issues.apache.org/jira/browse/HBASE-4348
 Project: HBase
  Issue Type: Improvement
  Components: metrics
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Assignee: Himanshu Vashishtha
Priority: Minor
  Labels: noob
 Fix For: 0.96.0

 Attachments: 4348-metrics-v3.patch, 4348-v1.patch, 4348-v2.patch, 
 RITs.png, RegionInTransitions2.png, hbase-4348-v4.patch, hbase-4348.patch, 
 metrics-v2.patch


 The following metrics would be useful for monitoring the master:
 - the number of regions in transition
 - the number of regions in transition that have been in transition for more 
 than a minute
 - how many seconds has the oldest region-in-transition been in transition

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4348) Add metrics for regions in transition

2012-04-05 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13248008#comment-13248008
 ] 

Jonathan Hsieh commented on HBASE-4348:
---

The failed test has been flakey recently.  

Committed.  Thanks Himanshu and thanks Ted for reviews.


 Add metrics for regions in transition
 -

 Key: HBASE-4348
 URL: https://issues.apache.org/jira/browse/HBASE-4348
 Project: HBase
  Issue Type: Improvement
  Components: metrics
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Assignee: Himanshu Vashishtha
Priority: Minor
  Labels: noob
 Fix For: 0.96.0

 Attachments: 4348-metrics-v3.patch, 4348-v1.patch, 4348-v2.patch, 
 RITs.png, RegionInTransitions2.png, hbase-4348-v4.patch, hbase-4348.patch, 
 metrics-v2.patch


 The following metrics would be useful for monitoring the master:
 - the number of regions in transition
 - the number of regions in transition that have been in transition for more 
 than a minute
 - how many seconds has the oldest region-in-transition been in transition

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5711) Tests are failing with incorrect data directory permissions.

2012-04-05 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13248087#comment-13248087
 ] 

Jonathan Hsieh commented on HBASE-5711:
---

@Uma 
Yeah, I'm a linux+eclipse guy for dev environment -- it is just in scripts for 
running tests or in my login script before I kick off eclipse.  I think I 
answered question on the mailing list about this the person came back saying 
they were in windows as well.

 Tests are failing with incorrect data directory permissions.
 

 Key: HBASE-5711
 URL: https://issues.apache.org/jira/browse/HBASE-5711
 Project: HBase
  Issue Type: Bug
  Components: test
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
 Fix For: 0.92.2, 0.94.0

 Attachments: HBASE-5711.patch


 When we run some tests in Hbase (TestAdmin), it is failing with following 
 error.
 {quote}
 Starting DataNode 0 with dfs.data.dir: 
 E:\Repositories\Hbase\target\test-data\5ff23198-892e-4f1c-8022-b3d9969fcf0b\dfscluster_0ecc6984-1925-4870-ac7c-439fceede4cb\dfs\data\data1,E:\Repositories\Hbase\target\test-data\5ff23198-892e-4f1c-8022-b3d9969fcf0b\dfscluster_0ecc6984-1925-4870-ac7c-439fceede4cb\dfs\data\data2
 2012-04-04 18:04:51,036 WARN  [main] impl.MetricsSystemImpl(137): Metrics 
 system not started: Cannot locate configuration: tried 
 hadoop-metrics2-datanode.properties, hadoop-metrics2.properties
 2012-04-04 18:04:51,255 WARN  [main] datanode.DataNode(1548): Invalid 
 directory in dfs.data.dir: Incorrect permission for 
 E:/Repositories/Hbase/target/test-data/5ff23198-892e-4f1c-8022-b3d9969fcf0b/dfscluster_0ecc6984-1925-4870-ac7c-439fceede4cb/dfs/data/data1,
  expected: rwxr-xr-x, while actual: rwx--
 2012-04-04 18:04:51,411 WARN  [main] datanode.DataNode(1548): Invalid 
 directory in dfs.data.dir: Incorrect permission for 
 E:/Repositories/Hbase/target/test-data/5ff23198-892e-4f1c-8022-b3d9969fcf0b/dfscluster_0ecc6984-1925-4870-ac7c-439fceede4cb/dfs/data/data2,
  expected: rwxr-xr-x, while actual: rwx--
 2012-04-04 18:04:51,411 ERROR [main] datanode.DataNode(1554): All directories 
 in dfs.data.dir are invalid.
 2012-04-04 18:04:51,411 INFO  [main] hbase.HBaseTestingUtility(684): Shutting 
 down minicluster
 2012-04-04 18:04:51,646 WARN  [main] hbase.HBaseTestingUtility(696): Failed 
 delete of 
 E:\Repositories\Hbase\target\test-data\5ff23198-892e-4f1c-8022-b3d9969fcf0b\dfscluster_0ecc6984-1925-4870-ac7c-439fceede4cb
 2012-04-04 18:04:51,646 INFO  [main] hbase.HBaseTestingUtility(700): 
 Minicluster is down
 {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5680) Hbase94 and Hbase 92.2 is not compatible with the Hadoop 23.1

2012-04-05 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13248097#comment-13248097
 ] 

Jonathan Hsieh commented on HBASE-5680:
---

This is my test result -- new actionable FATAL warning message in master log:  

{code}
2012-04-05 22:51:34,267 FATAL master.HMaster (HMaster.java:abort(1385)) - HBase 
is having a problem with its Hadoop jars.  You may need to recompile HBase 
against Hadoop version 0.23.0-cdh4b1 or change your hadoop jars to start 
properly
java.lang.NoClassDefFoundError: 
org/apache/hadoop/hdfs/protocol/FSConstants$SafeModeAction
at org.apache.hadoop.hbase.util.FSUtils.waitOnSafeMode(FSUtils.java:524)
at 
org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:324)
at 
org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:127)
at 
org.apache.hadoop.hbase.master.MasterFileSystem.init(MasterFileSystem.java:112)
at 
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:479)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:342)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.ClassNotFoundException: 
org.apache.hadoop.hdfs.protocol.FSConstants$SafeModeAction
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 7 more
{code}

 Hbase94 and Hbase 92.2 is not compatible with the Hadoop 23.1 
 --

 Key: HBASE-5680
 URL: https://issues.apache.org/jira/browse/HBASE-5680
 Project: HBase
  Issue Type: Bug
  Components: master
Reporter: Kristam Subba Swathi
Assignee: Jonathan Hsieh

 Hmaster is not able to start because of the following error
 Please find the following error 
 
 2012-03-30 11:12:19,487 FATAL org.apache.hadoop.hbase.master.HMaster: 
 Unhandled exception. Starting shutdown.
 java.lang.NoClassDefFoundError: 
 org/apache/hadoop/hdfs/protocol/FSConstants$SafeModeAction
   at org.apache.hadoop.hbase.util.FSUtils.waitOnSafeMode(FSUtils.java:524)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:324)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:127)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.init(MasterFileSystem.java:112)
   at 
 org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:496)
   at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:363)
   at java.lang.Thread.run(Thread.java:662)
 Caused by: java.lang.ClassNotFoundException: 
 org.apache.hadoop.hdfs.protocol.FSConstants$SafeModeAction
   at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
   ... 7 more
 There is a change in the FSConstants

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5628) Improve performance of uberhbck

2012-04-04 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246248#comment-13246248
 ] 

Jonathan Hsieh commented on HBASE-5628:
---

- parallelize the reads of .regioninfo files in the diagnostic section of 
repairs.

 Improve performance of uberhbck
 ---

 Key: HBASE-5628
 URL: https://issues.apache.org/jira/browse/HBASE-5628
 Project: HBase
  Issue Type: Improvement
  Components: hbck
Affects Versions: 0.90.7, 0.92.2, 0.94.0, 0.96.0
Reporter: Jonathan Hsieh

 During reviews of HBASE-5128 there are several opportunities investigate for 
 improving the performance of the tool.
 - Change regionInfoMap and tablesInfo from TreeMap to HashMap.
 - Change some full region set reloads to be incremental to require fewer 
 passes.
 - Cache meta for subsequent calls of closeRegionSileneglyAndWait

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5628) Improve performance of uberhbck

2012-04-04 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246249#comment-13246249
 ] 

Jonathan Hsieh commented on HBASE-5628:
---

- parallelize the reads of .regioninfo files in the diagnostic section of 
repairs.

 Improve performance of uberhbck
 ---

 Key: HBASE-5628
 URL: https://issues.apache.org/jira/browse/HBASE-5628
 Project: HBase
  Issue Type: Improvement
  Components: hbck
Affects Versions: 0.90.7, 0.92.2, 0.94.0, 0.96.0
Reporter: Jonathan Hsieh

 During reviews of HBASE-5128 there are several opportunities investigate for 
 improving the performance of the tool.
 - Change regionInfoMap and tablesInfo from TreeMap to HashMap.
 - Change some full region set reloads to be incremental to require fewer 
 passes.
 - Cache meta for subsequent calls of closeRegionSileneglyAndWait

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5680) Hbase94 and Hbase 92.2 is not compatible with the Hadoop 23.1

2012-04-03 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13245015#comment-13245015
 ] 

Jonathan Hsieh commented on HBASE-5680:
---

I started one that emits a better warning, need to test still. 

Sent from my iPhone




 Hbase94 and Hbase 92.2 is not compatible with the Hadoop 23.1 
 --

 Key: HBASE-5680
 URL: https://issues.apache.org/jira/browse/HBASE-5680
 Project: HBase
  Issue Type: Bug
  Components: master
Reporter: Kristam Subba Swathi

 Hmaster is not able to start because of the following error
 Please find the following error 
 
 2012-03-30 11:12:19,487 FATAL org.apache.hadoop.hbase.master.HMaster: 
 Unhandled exception. Starting shutdown.
 java.lang.NoClassDefFoundError: 
 org/apache/hadoop/hdfs/protocol/FSConstants$SafeModeAction
   at org.apache.hadoop.hbase.util.FSUtils.waitOnSafeMode(FSUtils.java:524)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:324)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:127)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.init(MasterFileSystem.java:112)
   at 
 org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:496)
   at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:363)
   at java.lang.Thread.run(Thread.java:662)
 Caused by: java.lang.ClassNotFoundException: 
 org.apache.hadoop.hdfs.protocol.FSConstants$SafeModeAction
   at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
   ... 7 more
 There is a change in the FSConstants

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5536) Make it clear that hbase 0.96 requires hadoop 1.0.0 at least; we will no longer work on older versions

2012-04-03 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13245345#comment-13245345
 ] 

Jonathan Hsieh commented on HBASE-5536:
---

bq. That'd be nice. Agree would be sweet if we didn't have to do reflection at 
all going forward.

Or at least to remove reflection require for pre-hadoop 1.0.0.  We will 
probably eventually need it again for hadoop 0.23 hdfs...

 Make it clear that hbase 0.96 requires hadoop 1.0.0 at least; we will no 
 longer work on older versions
 --

 Key: HBASE-5536
 URL: https://issues.apache.org/jira/browse/HBASE-5536
 Project: HBase
  Issue Type: Task
Reporter: stack
Priority: Blocker
 Fix For: 0.96.0


 Looks like there is pretty much consensus that depending on 1.0.0 in 0.96 
 should be fine?  See 
 http://search-hadoop.com/m/dSbVW14EsUb2/discuss+0.96subj=RE+DISCUSS+Have+hbase+require+at+least+hadoop+1+0+0+in+hbase+0+96+0+

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5599) The hbck tool can not fix the six scenarios, it is NO_VERSION_FILE, NOT_IN_META_OR_DEPLOYED, NOT_IN_META, SHOULD_NOT_BE_DEPLOYED, FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_

2012-04-03 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13245688#comment-13245688
 ] 

Jonathan Hsieh commented on HBASE-5599:
---

Thanks for the updates, the new patch looks pretty good.  A few things to 
follow up on before commit:

- If adding a test case is hard, let's file new issue to add HBCK test case 
for SHOULD_NOT_BE_DEPLOYED case.  

- Change the text here to be Try to fix missing hbase.version file in hdfs.
{code}
 System.err.println(   -fixVersionFile   Try to fix the hbase.version 
missing.);
{code} 

- I noticed the Merge.java changes are still present in the newer patch -- is 
that intended? It don't seem necessary or related to this issue. 

This should be good to commit for 0.90, if you address these issues.  

Could you provide an updated version for trunk? 

Thanks!

 The hbck tool can not fix the six scenarios, it is NO_VERSION_FILE, 
 NOT_IN_META_OR_DEPLOYED, NOT_IN_META, SHOULD_NOT_BE_DEPLOYED, 
 FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_IN_REGION_CHAIN.
 

 Key: HBASE-5599
 URL: https://issues.apache.org/jira/browse/HBASE-5599
 Project: HBase
  Issue Type: New Feature
  Components: hbck
Affects Versions: 0.90.6
Reporter: fulin wang
Assignee: fulin wang
 Fix For: 0.90.6

 Attachments: hbase-5599-0.90.patch, hbase-5599-0.90_v2.patch, 
 hbase-5599-0.90_v3.patch, hbase-5599-0.90_v5.patch, hbase-5599-0.90_v6.patch, 
 hbase-5599-0.92_v5.patch, hbase-5599-0.94_v5.patch, hbase-5599-trunk_v5.patch


 The hbck tool can not fix the six scenarios.
 1. Version file does not exist in root dir.
Fix: I try to create a version file by 'FSUtils.setVersion' method.

 2. [REGIONNAME][KEY] on HDFS, but not listed in META or deployed on any 
 region server.
Fix: I get region info form the hdfs file, this region info write to 
 '.META.' table.

 3. [REGIONNAME][KEY] not in META, but deployed on [SERVERNAME]
Fix: I get region info form the hdfs file, this region info write to 
 '.META.' table.

 4. [REGIONNAME] should not be deployed according to META, but is deployed on 
 [SERVERNAME]
Fix: Close this region.

 5. First region should start with an empty key.  You need to  create a new 
 region and regioninfo in HDFS to plug the hole.
Fix: The region info is not in hdfs and .META., so it create a empty 
 region for this error.
 6. There is a hole in the region chain between [KEY] and [KEY]. You need to 
 create a new regioninfo and region dir in hdfs to plug the hole.
   Fix: The region info is not in hdfs and .META., so it create a empty region 
 for this hole.
   

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5680) Hbase94 and Hbase 92.2 is not compatible with the Hadoop 23.1

2012-04-02 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244250#comment-13244250
 ] 

Jonathan Hsieh commented on HBASE-5680:
---

I feel like to resolve this we should give the user some sort of warning about 
needing to recompile against hadoop23 (or vice versa if a version compiled 
against hadoop23 is attempts to run against a hadoop 1.0.0/0.20.x based hdfs).  
Thoughts?

 Hbase94 and Hbase 92.2 is not compatible with the Hadoop 23.1 
 --

 Key: HBASE-5680
 URL: https://issues.apache.org/jira/browse/HBASE-5680
 Project: HBase
  Issue Type: Bug
  Components: master
Reporter: Kristam Subba Swathi

 Hmaster is not able to start because of the following error
 Please find the following error 
 
 2012-03-30 11:12:19,487 FATAL org.apache.hadoop.hbase.master.HMaster: 
 Unhandled exception. Starting shutdown.
 java.lang.NoClassDefFoundError: 
 org/apache/hadoop/hdfs/protocol/FSConstants$SafeModeAction
   at org.apache.hadoop.hbase.util.FSUtils.waitOnSafeMode(FSUtils.java:524)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:324)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:127)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.init(MasterFileSystem.java:112)
   at 
 org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:496)
   at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:363)
   at java.lang.Thread.run(Thread.java:662)
 Caused by: java.lang.ClassNotFoundException: 
 org.apache.hadoop.hdfs.protocol.FSConstants$SafeModeAction
   at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
   ... 7 more
 There is a change in the FSConstants

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5697) Audit HBase for usage of deprecated hadoop 0.20.x property names.

2012-04-02 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244671#comment-13244671
 ] 

Jonathan Hsieh commented on HBASE-5697:
---

Hopefully that is a comprehensive list.  My guess is that only a handful are 
relevant.  I started testing on hadoop 23 and there are definitely some new 
deprecation warnings that show up in logs/console.  Also, some have been bugs 
in previous versions - I've gotten snagged on this one before:

fs.default.name - fs.defaultFS 



 Audit HBase for usage of deprecated hadoop 0.20.x property names.
 -

 Key: HBASE-5697
 URL: https://issues.apache.org/jira/browse/HBASE-5697
 Project: HBase
  Issue Type: Task
Reporter: Jonathan Hsieh

 Many xml config properties in Hadoop have changed in 0.23.  We should audit 
 hbase to insulate it from hadoop property name changes.
 Here is a list of the hadoop property name changes:
 http://hadoop.apache.org/common/docs/r0.23.1/hadoop-project-dist/hadoop-common/DeprecatedProperties.html

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4348) Add metrics for regions in transition

2012-04-01 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13243801#comment-13243801
 ] 

Jonathan Hsieh commented on HBASE-4348:
---

@Otis  I generally like the policy of setting versions on commit, or having a 
release manager set it if they decide it is necessary for a release. 

 Add metrics for regions in transition
 -

 Key: HBASE-4348
 URL: https://issues.apache.org/jira/browse/HBASE-4348
 Project: HBase
  Issue Type: Improvement
  Components: metrics
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Assignee: Himanshu Vashishtha
Priority: Minor
  Labels: noob
 Fix For: 0.96.0

 Attachments: 4348-metrics-v3.patch, 4348-v1.patch, 4348-v2.patch, 
 RITs.png, RegionInTransitions2.png, metrics-v2.patch


 The following metrics would be useful for monitoring the master:
 - the number of regions in transition
 - the number of regions in transition that have been in transition for more 
 than a minute
 - how many seconds has the oldest region-in-transition been in transition

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5680) Hbase94 and Hbase 92.2 is not compatible with the Hadoop 23.1

2012-04-01 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13243942#comment-13243942
 ] 

Jonathan Hsieh commented on HBASE-5680:
---

Tried it again, and actually -- if you recompile using -Dhadoop.profile=23 
without the security profile the Master comes up and does not encounter the 
problem.  

(I probably had the wrong hadoop jars in my hbase classpath). 

So it boils down to needing to recompile hbase against hadoop 23. 

Maybe we should catch this exception and warn the user to recompile HBase, or 
possibly put out yet another package that is compiled against 23. 

 Hbase94 and Hbase 92.2 is not compatible with the Hadoop 23.1 
 --

 Key: HBASE-5680
 URL: https://issues.apache.org/jira/browse/HBASE-5680
 Project: HBase
  Issue Type: Bug
  Components: master
Reporter: Kristam Subba Swathi

 Hmaster is not able to start because of the following error
 Please find the following error 
 
 2012-03-30 11:12:19,487 FATAL org.apache.hadoop.hbase.master.HMaster: 
 Unhandled exception. Starting shutdown.
 java.lang.NoClassDefFoundError: 
 org/apache/hadoop/hdfs/protocol/FSConstants$SafeModeAction
   at org.apache.hadoop.hbase.util.FSUtils.waitOnSafeMode(FSUtils.java:524)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:324)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:127)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.init(MasterFileSystem.java:112)
   at 
 org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:496)
   at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:363)
   at java.lang.Thread.run(Thread.java:662)
 Caused by: java.lang.ClassNotFoundException: 
 org.apache.hadoop.hdfs.protocol.FSConstants$SafeModeAction
   at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
   ... 7 more
 There is a change in the FSConstants

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5680) Hbase94 and Hbase 92.2 is not compatible with the Hadoop 23.1

2012-04-01 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13243970#comment-13243970
 ] 

Jonathan Hsieh commented on HBASE-5680:
---

@Ram -- when I did it i used 'mvn package -DskipTests -Dhadoop.profile=23' and 
then ran from a copy of the dir generated in target/hbase-xxx/hbase-xxx.  If 
you run from the directory you ran the mvn command in, I think hbase will 
scripts will picked up hbase from that dir, or possibly the 1.0.0 version from 
the ~/.m2 dir. I think this is what caught me the first time I tried this.

 Hbase94 and Hbase 92.2 is not compatible with the Hadoop 23.1 
 --

 Key: HBASE-5680
 URL: https://issues.apache.org/jira/browse/HBASE-5680
 Project: HBase
  Issue Type: Bug
  Components: master
Reporter: Kristam Subba Swathi

 Hmaster is not able to start because of the following error
 Please find the following error 
 
 2012-03-30 11:12:19,487 FATAL org.apache.hadoop.hbase.master.HMaster: 
 Unhandled exception. Starting shutdown.
 java.lang.NoClassDefFoundError: 
 org/apache/hadoop/hdfs/protocol/FSConstants$SafeModeAction
   at org.apache.hadoop.hbase.util.FSUtils.waitOnSafeMode(FSUtils.java:524)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:324)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:127)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.init(MasterFileSystem.java:112)
   at 
 org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:496)
   at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:363)
   at java.lang.Thread.run(Thread.java:662)
 Caused by: java.lang.ClassNotFoundException: 
 org.apache.hadoop.hdfs.protocol.FSConstants$SafeModeAction
   at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
   ... 7 more
 There is a change in the FSConstants

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5680) Hbase94 and Hbase 92.2 is not compatible with the Hadoop 23.1

2012-04-01 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13243976#comment-13243976
 ] 

Jonathan Hsieh commented on HBASE-5680:
---

Hm.. I didn't add all the hadoop 23 jars -- I added:

hadoop-auth-*.jar
hadoop-common-*.jar
hadoop-hdfs-*.jar

I didn't add any of the mapreduce or yarn jars maybe that has something to do 
with it.  A lot of the recompile was due to changes in MR2 (classes turned into 
interfaces, and shims to allow compilation).  Can you try moving those jars out 
of the way?

By just including those jars, I've run some recompiled mr jobs using a command 
line like this:

HADOOP_HOME=`hbase classpath` hadoop jar xxx.jar Class args..



 Hbase94 and Hbase 92.2 is not compatible with the Hadoop 23.1 
 --

 Key: HBASE-5680
 URL: https://issues.apache.org/jira/browse/HBASE-5680
 Project: HBase
  Issue Type: Bug
  Components: master
Reporter: Kristam Subba Swathi

 Hmaster is not able to start because of the following error
 Please find the following error 
 
 2012-03-30 11:12:19,487 FATAL org.apache.hadoop.hbase.master.HMaster: 
 Unhandled exception. Starting shutdown.
 java.lang.NoClassDefFoundError: 
 org/apache/hadoop/hdfs/protocol/FSConstants$SafeModeAction
   at org.apache.hadoop.hbase.util.FSUtils.waitOnSafeMode(FSUtils.java:524)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:324)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:127)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.init(MasterFileSystem.java:112)
   at 
 org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:496)
   at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:363)
   at java.lang.Thread.run(Thread.java:662)
 Caused by: java.lang.ClassNotFoundException: 
 org.apache.hadoop.hdfs.protocol.FSConstants$SafeModeAction
   at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
   ... 7 more
 There is a change in the FSConstants

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5680) Hbase94 and Hbase 92.2 is not compatible with the Hadoop 23.1

2012-03-31 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13243593#comment-13243593
 ] 

Jonathan Hsieh commented on HBASE-5680:
---

The master starts on top of hadoop 0.23.x
apache 0.92.1 recompiled with the security profile *on* (-Psecurity) and with 
-Dhadoop.profile=23
apache 0.94.0rc0 recompiled with the security profile *on* (-Psecurity) and  
with -Dhadoop.profile=23 

The master fails on top of hadoop 0.23.x with the class not found error in 
these cases:
apache 0.92.1 right out of tarball
apache 0.92.1 with the security profile *off* with -Dhadoop.profile=23
apache 0.94.0rc0 right out of tarball.
apache 0.94.0rc0 security right out of tarball.
apache 0.94.0rc0 with the security profile *off* with -Dhadoop.profile=23



 Hbase94 and Hbase 92.2 is not compatible with the Hadoop 23.1 
 --

 Key: HBASE-5680
 URL: https://issues.apache.org/jira/browse/HBASE-5680
 Project: HBase
  Issue Type: Bug
  Components: master
Reporter: Kristam Subba Swathi

 Hmaster is not able to start because of the following error
 Please find the following error 
 
 2012-03-30 11:12:19,487 FATAL org.apache.hadoop.hbase.master.HMaster: 
 Unhandled exception. Starting shutdown.
 java.lang.NoClassDefFoundError: 
 org/apache/hadoop/hdfs/protocol/FSConstants$SafeModeAction
   at org.apache.hadoop.hbase.util.FSUtils.waitOnSafeMode(FSUtils.java:524)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:324)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:127)
   at 
 org.apache.hadoop.hbase.master.MasterFileSystem.init(MasterFileSystem.java:112)
   at 
 org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:496)
   at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:363)
   at java.lang.Thread.run(Thread.java:662)
 Caused by: java.lang.ClassNotFoundException: 
 org.apache.hadoop.hdfs.protocol.FSConstants$SafeModeAction
   at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
   ... 7 more
 There is a change in the FSConstants

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5213) hbase master stop does not bring down backup masters

2012-03-31 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13243638#comment-13243638
 ] 

Jonathan Hsieh commented on HBASE-5213:
---

Committed to 0.92 and 0.90.  Thanks Greg.

 hbase master stop does not bring down backup masters
 --

 Key: HBASE-5213
 URL: https://issues.apache.org/jira/browse/HBASE-5213
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.5, 0.92.0, 0.94.0, 0.96.0
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Priority: Minor
 Fix For: 0.94.0, 0.96.0

 Attachments: HBASE-5213-v0-trunk.patch, HBASE-5213-v1-trunk.patch, 
 HBASE-5213-v2-90.patch, HBASE-5213-v2-92.patch, HBASE-5213-v2-trunk.patch


 Typing hbase master stop produces the following message:
 stop   Start cluster shutdown; Master signals RegionServer shutdown
 It seems like backup masters should be considered part of the cluster, but 
 they are not brought down by hbase master stop.
 stop-hbase.sh does correctly bring down the backup masters.
 The same behavior is observed when a client app makes use of the client API 
 HBaseAdmin.shutdown() 
 http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html#shutdown()
  -- this isn't too surprising since I think hbase master stop just calls 
 this API.
 It seems like HBASE-1448 address this; perhaps there was a regression?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5213) hbase master stop does not bring down backup masters

2012-03-31 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13243639#comment-13243639
 ] 

Jonathan Hsieh commented on HBASE-5213:
---

Committed to 0.92 and 0.90.  Thanks Greg.

 hbase master stop does not bring down backup masters
 --

 Key: HBASE-5213
 URL: https://issues.apache.org/jira/browse/HBASE-5213
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.5, 0.92.0, 0.94.0, 0.96.0
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Priority: Minor
 Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0

 Attachments: HBASE-5213-v0-trunk.patch, HBASE-5213-v1-trunk.patch, 
 HBASE-5213-v2-90.patch, HBASE-5213-v2-92.patch, HBASE-5213-v2-trunk.patch


 Typing hbase master stop produces the following message:
 stop   Start cluster shutdown; Master signals RegionServer shutdown
 It seems like backup masters should be considered part of the cluster, but 
 they are not brought down by hbase master stop.
 stop-hbase.sh does correctly bring down the backup masters.
 The same behavior is observed when a client app makes use of the client API 
 HBaseAdmin.shutdown() 
 http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html#shutdown()
  -- this isn't too surprising since I think hbase master stop just calls 
 this API.
 It seems like HBASE-1448 address this; perhaps there was a regression?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5535) Make the functions in task monitor synchronized

2012-03-30 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242591#comment-13242591
 ] 

Jonathan Hsieh commented on HBASE-5535:
---

I looked at the code and this may have been an alternate solution to the also 
applied HBASE-4386.  This doesn't really hurt the code (synchronized is 
slightly larger now) so we can leave it as is.

 Make the functions in task monitor synchronized
 ---

 Key: HBASE-5535
 URL: https://issues.apache.org/jira/browse/HBASE-5535
 Project: HBase
  Issue Type: Bug
Reporter: Liyin Tang
Assignee: Liyin Tang
 Fix For: 0.92.2, 0.94.0

 Attachments: 
 HBASE-5535-Make-the-functions-in-task-monitor-synchr-2012-03-08_16_33_42.patch


 There are some potential race condition in the task monitor. So update the 
 functions in task monitor to be synchronized.
 The example of the problem caused by the race condition:
 ERROR org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flush 
 failed for region 
 java.lang.IndexOutOfBoundsException: Index: 1745, Size: 1744
 at java.util.ArrayList.add(ArrayList.java:367)
 at java.util.SubList.add(AbstractList.java:633)
 at java.util.SubList.add(AbstractList.java:633)
 at java.util.SubList.add(AbstractList.java:633)
 at java.util.SubList.add(AbstractList.java:633)
 at java.util.SubList.add(AbstractList.java:633)
 at java.util.AbstractList.add(AbstractList.java:91)
 at 
 org.apache.hadoop.hbase.monitoring.TaskMonitor.createStatus(TaskMonitor.java:74)
 at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1139)
 at 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:260)
 at 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:234)
 at 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:146)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5685) [findbugs] Exclude Protobuf warnings from wire compat patches.

2012-03-30 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242863#comment-13242863
 ] 

Jonathan Hsieh commented on HBASE-5685:
---

precommit test is happy, since trivial, committing.

 [findbugs] Exclude Protobuf warnings from wire compat patches.
 --

 Key: HBASE-5685
 URL: https://issues.apache.org/jira/browse/HBASE-5685
 Project: HBase
  Issue Type: Sub-task
  Components: scripts
Affects Versions: 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Fix For: 0.96.0

 Attachments: hbase-5598.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5638) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase

2012-03-29 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241077#comment-13241077
 ] 

Jonathan Hsieh commented on HBASE-5638:
---

Matteo, 

We should probably use the built-in boolean parser in configuration instead of 
doing a the string compare -- something like this:

{code}
if (conf.getBoolean(HConstants.CLUSTER_DISTRIBUTED, 
HConstants.DEFAULT_CLUSTER_DISTRIBUTED))
{code}

see: 
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/conf/Configuration.html#getBoolean(java.lang.String,
 boolean)

We'd also end up changing HConstants.DEFAULT_CLUSTER_DISTRBUTED to be a boolean 
type instead of a string as well.



 Backport to 0.90 and 0.92 - NPE reading ZK config in HBase
 --

 Key: HBASE-5638
 URL: https://issues.apache.org/jira/browse/HBASE-5638
 Project: HBase
  Issue Type: Sub-task
  Components: zookeeper
Affects Versions: 0.90.6, 0.92.1
Reporter: Matteo Bertozzi
Priority: Minor
 Fix For: 0.90.7, 0.92.2

 Attachments: HBASE-5633-0.90.patch, HBASE-5633-0.92.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5564) Bulkload is discarding duplicate records

2012-03-29 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241341#comment-13241341
 ] 

Jonathan Hsieh commented on HBASE-5564:
---

maybe not worry about find bugs for normal patches? (ideally it does go up 
though) the find bugs number isn't the focus of this patch.

Sent from my iPhone




 Bulkload is discarding duplicate records
 

 Key: HBASE-5564
 URL: https://issues.apache.org/jira/browse/HBASE-5564
 Project: HBase
  Issue Type: Bug
  Components: mapreduce
Affects Versions: 0.96.0
 Environment: HBase 0.92
Reporter: Laxman
Assignee: Laxman
  Labels: bulkloader
 Fix For: 0.96.0

 Attachments: 5564.lint, 5564v5.txt, HBASE-5564_trunk.1.patch, 
 HBASE-5564_trunk.1.patch, HBASE-5564_trunk.2.patch, HBASE-5564_trunk.3.patch, 
 HBASE-5564_trunk.4_final.patch, HBASE-5564_trunk.patch


 Duplicate records are getting discarded when duplicate records exists in same 
 input file and more specifically if they exists in same split.
 Duplicate records are considered if the records are from diffrent different 
 splits.
 Version under test: HBase 0.92

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5599) The hbck tool can not fix the six scenarios, it is NO_VERSION_FILE, NOT_IN_META_OR_DEPLOYED, NOT_IN_META, SHOULD_NOT_BE_DEPLOYED, FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_

2012-03-29 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241452#comment-13241452
 ] 

Jonathan Hsieh commented on HBASE-5599:
---

Thanks for updating the patches.

Instead of relying on shouldFixAssignments, could you add a new config flag for 
fixing the NO_VERSION_FILE? The fix isn't really a region assignment fix!  
Maybe call it -fixVersionFile, and have it on with the combo '-repair*' 
options? 

Can you add a test case for SHOULD_NOT_BE_DEPLOYED?  You should be able to hack 
one of the test cases from TestHBaseFsck.  I've had problems just using assign 
before so if tests come back failing, you may want to try 
HBaseFsckRepair.fixMultiAssignemnt or tryAssignemntRepair.

The merge addition isn't really related -- was its inclusion intended?

 The hbck tool can not fix the six scenarios, it is NO_VERSION_FILE, 
 NOT_IN_META_OR_DEPLOYED, NOT_IN_META, SHOULD_NOT_BE_DEPLOYED, 
 FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_IN_REGION_CHAIN.
 

 Key: HBASE-5599
 URL: https://issues.apache.org/jira/browse/HBASE-5599
 Project: HBase
  Issue Type: New Feature
  Components: hbck
Affects Versions: 0.90.6
Reporter: fulin wang
 Fix For: 0.90.6

 Attachments: hbase-5599-0.90.patch, hbase-5599-0.90_v2.patch, 
 hbase-5599-0.90_v3.patch, hbase-5599-0.90_v5.patch, hbase-5599-0.92_v5.patch, 
 hbase-5599-0.94_v5.patch, hbase-5599-trunk_v5.patch


 The hbck tool can not fix the six scenarios.
 1. Version file does not exist in root dir.
Fix: I try to create a version file by 'FSUtils.setVersion' method.

 2. [REGIONNAME][KEY] on HDFS, but not listed in META or deployed on any 
 region server.
Fix: I get region info form the hdfs file, this region info write to 
 '.META.' table.

 3. [REGIONNAME][KEY] not in META, but deployed on [SERVERNAME]
Fix: I get region info form the hdfs file, this region info write to 
 '.META.' table.

 4. [REGIONNAME] should not be deployed according to META, but is deployed on 
 [SERVERNAME]
Fix: Close this region.

 5. First region should start with an empty key.  You need to  create a new 
 region and regioninfo in HDFS to plug the hole.
Fix: The region info is not in hdfs and .META., so it create a empty 
 region for this error.
 6. There is a hole in the region chain between [KEY] and [KEY]. You need to 
 create a new regioninfo and region dir in hdfs to plug the hole.
   Fix: The region info is not in hdfs and .META., so it create a empty region 
 for this hole.
   

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5564) Bulkload is discarding duplicate records

2012-03-29 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241485#comment-13241485
 ] 

Jonathan Hsieh commented on HBASE-5564:
---

meant to say ideally it does not go up.  I think stack's action (he didn't 
lower findbugs number on normal patch) captured the same idea.

 Bulkload is discarding duplicate records
 

 Key: HBASE-5564
 URL: https://issues.apache.org/jira/browse/HBASE-5564
 Project: HBase
  Issue Type: Bug
  Components: mapreduce
Affects Versions: 0.96.0
 Environment: HBase 0.92
Reporter: Laxman
Assignee: Laxman
  Labels: bulkloader
 Fix For: 0.96.0

 Attachments: 5564.lint, 5564v5.txt, HBASE-5564_trunk.1.patch, 
 HBASE-5564_trunk.1.patch, HBASE-5564_trunk.2.patch, HBASE-5564_trunk.3.patch, 
 HBASE-5564_trunk.4_final.patch, HBASE-5564_trunk.patch


 Duplicate records are getting discarded when duplicate records exists in same 
 input file and more specifically if they exists in same split.
 Duplicate records are considered if the records are from diffrent different 
 splits.
 Version under test: HBase 0.92

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5638) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase

2012-03-29 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241565#comment-13241565
 ] 

Jonathan Hsieh commented on HBASE-5638:
---

Lgtm.  I'll do a quick test and commit to 0.90/0.92/0.94/trunk branches.

 Backport to 0.90 and 0.92 - NPE reading ZK config in HBase
 --

 Key: HBASE-5638
 URL: https://issues.apache.org/jira/browse/HBASE-5638
 Project: HBase
  Issue Type: Sub-task
  Components: zookeeper
Affects Versions: 0.90.6, 0.92.1
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
 Fix For: 0.90.7, 0.92.2

 Attachments: HBASE-5633-0.90.patch, HBASE-5633-0.92.patch, 
 HBASE-5638-0.90-v1.patch, HBASE-5638-0.90-v2.patch, HBASE-5638-0.92-v1.patch, 
 HBASE-5638-0.92-v2.patch, HBASE-5638-trunk-v1.patch, HBASE-5638-trunk-v2.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4348) Add metrics for regions in transition

2012-03-29 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241761#comment-13241761
 ] 

Jonathan Hsieh commented on HBASE-4348:
---

Himanshu, for us to commit patches, we need you to attach the patch to the JIRA 
and the grant permission.  Currently,  review board is optional, but makes it 
way easier for us to review.  

Please do so after you address the latest comments.  Thanks!

 Add metrics for regions in transition
 -

 Key: HBASE-4348
 URL: https://issues.apache.org/jira/browse/HBASE-4348
 Project: HBase
  Issue Type: Improvement
  Components: metrics
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Assignee: Himanshu Vashishtha
Priority: Minor
  Labels: noob
 Attachments: 4348-metrics-v3.patch, 4348-v1.patch, 4348-v2.patch, 
 RITs.png, RegionInTransitions2.png, metrics-v2.patch


 The following metrics would be useful for monitoring the master:
 - the number of regions in transition
 - the number of regions in transition that have been in transition for more 
 than a minute
 - how many seconds has the oldest region-in-transition been in transition

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5675) Create table fails if we keep refreshing master's UI for task monitor status

2012-03-29 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241866#comment-13241866
 ] 

Jonathan Hsieh commented on HBASE-5675:
---

The fix was in HBASE-5535 and in the 0.94 branch and above.  I'll mark this 
issue as duplicate, and get it added into the issues to fix in CDH. 

 Create table fails if we keep refreshing master's UI for task monitor status
 

 Key: HBASE-5675
 URL: https://issues.apache.org/jira/browse/HBASE-5675
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.4, 0.92.0, 0.94.0
Reporter: Mubarak Seyed
Assignee: Mubarak Seyed
  Labels: noob

 I tried to create a table with 2K pre-split regions, region assignment was in 
 middle and i was keep refreshing master's web UI to find the status of the 
 task using task monitor, table creation was failed and {{META}} was showing 
 2K regions with server location value is {{null}} and regions weren't 
 deployed onto region-servers.
 {code}
 table_ACreating table table_A
 java.io.IOException: java.io.IOException: 
 java.util.ConcurrentModificationException
   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
   at 
 sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
   at 
 sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
   at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
   at 
 org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
   at 
 org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
   at 
 org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:384)
   at 
 org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:294)
   at com.test.tools.hbase.schema.createIfNotExists(schema.java:520)
   at com.test.tools.hbase.schema.main(schema.java:627)
 Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: 
 java.util.ConcurrentModificationException
   at java.util.SubList.checkForComodification(AbstractList.java:752)
   at java.util.SubList.add(AbstractList.java:632)
   at java.util.SubList.add(AbstractList.java:633)
   at java.util.SubList.add(AbstractList.java:633)
   ..
   ..
   at java.util.SubList.add(AbstractList.java:633)
   at java.util.AbstractList.add(AbstractList.java:91)
   at 
 org.apache.hadoop.hbase.monitoring.TaskMonitor.createStatus(TaskMonitor.java:76)
   at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:510)
   at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:490)
   at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:853)
   at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:813)
   at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:780)
   at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
   at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
   at 
 org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
   at $Proxy5.createTable(Unknown Source)
   at 
 org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:382)
   
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5535) Make the functions in task monitor synchronized

2012-03-29 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241873#comment-13241873
 ] 

Jonathan Hsieh commented on HBASE-5535:
---

Stack, looks like this hasn't made it into the 0.92.2/0.92 branch.  Shall we 
just commit from this jira and add fix version?

 Make the functions in task monitor synchronized
 ---

 Key: HBASE-5535
 URL: https://issues.apache.org/jira/browse/HBASE-5535
 Project: HBase
  Issue Type: Bug
Reporter: Liyin Tang
Assignee: Liyin Tang
 Fix For: 0.94.0

 Attachments: 
 HBASE-5535-Make-the-functions-in-task-monitor-synchr-2012-03-08_16_33_42.patch


 There are some potential race condition in the task monitor. So update the 
 functions in task monitor to be synchronized.
 The example of the problem caused by the race condition:
 ERROR org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flush 
 failed for region 
 java.lang.IndexOutOfBoundsException: Index: 1745, Size: 1744
 at java.util.ArrayList.add(ArrayList.java:367)
 at java.util.SubList.add(AbstractList.java:633)
 at java.util.SubList.add(AbstractList.java:633)
 at java.util.SubList.add(AbstractList.java:633)
 at java.util.SubList.add(AbstractList.java:633)
 at java.util.SubList.add(AbstractList.java:633)
 at java.util.AbstractList.add(AbstractList.java:91)
 at 
 org.apache.hadoop.hbase.monitoring.TaskMonitor.createStatus(TaskMonitor.java:74)
 at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1139)
 at 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:260)
 at 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:234)
 at 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:146)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5675) Create table fails if we keep refreshing master's UI for task monitor status

2012-03-29 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241938#comment-13241938
 ] 

Jonathan Hsieh commented on HBASE-5675:
---

I took a closer look and HBASE-5535 doesn't seem to really add any new 
synchronization help on the tasks list.  Another related patch is a part of 
HBASE-4057; investigating further.

 Create table fails if we keep refreshing master's UI for task monitor status
 

 Key: HBASE-5675
 URL: https://issues.apache.org/jira/browse/HBASE-5675
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.4, 0.92.0
Reporter: Mubarak Seyed
Assignee: Mubarak Seyed
  Labels: noob

 I tried to create a table with 2K pre-split regions, region assignment was in 
 middle and i was keep refreshing master's web UI to find the status of the 
 task using task monitor, table creation was failed and {{META}} was showing 
 2K regions with server location value is {{null}} and regions weren't 
 deployed onto region-servers.
 {code}
 table_ACreating table table_A
 java.io.IOException: java.io.IOException: 
 java.util.ConcurrentModificationException
   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
   at 
 sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
   at 
 sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
   at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
   at 
 org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
   at 
 org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
   at 
 org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:384)
   at 
 org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:294)
   at com.test.tools.hbase.schema.createIfNotExists(schema.java:520)
   at com.test.tools.hbase.schema.main(schema.java:627)
 Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: 
 java.util.ConcurrentModificationException
   at java.util.SubList.checkForComodification(AbstractList.java:752)
   at java.util.SubList.add(AbstractList.java:632)
   at java.util.SubList.add(AbstractList.java:633)
   at java.util.SubList.add(AbstractList.java:633)
   ..
   ..
   at java.util.SubList.add(AbstractList.java:633)
   at java.util.AbstractList.add(AbstractList.java:91)
   at 
 org.apache.hadoop.hbase.monitoring.TaskMonitor.createStatus(TaskMonitor.java:76)
   at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:510)
   at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:490)
   at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:853)
   at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:813)
   at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:780)
   at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
   at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
   at 
 org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
   at $Proxy5.createTable(Unknown Source)
   at 
 org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:382)
   
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5675) Create table fails if we keep refreshing master's UI for task monitor status

2012-03-29 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241955#comment-13241955
 ] 

Jonathan Hsieh commented on HBASE-5675:
---

Mubarak, found it -- the fix is actually HBASE-4386, fixed in 0.92/0.94/trunk 
(and CDH3u3).

 Create table fails if we keep refreshing master's UI for task monitor status
 

 Key: HBASE-5675
 URL: https://issues.apache.org/jira/browse/HBASE-5675
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.4, 0.92.0
Reporter: Mubarak Seyed
Assignee: Mubarak Seyed
  Labels: noob

 I tried to create a table with 2K pre-split regions, region assignment was in 
 middle and i was keep refreshing master's web UI to find the status of the 
 task using task monitor, table creation was failed and {{META}} was showing 
 2K regions with server location value is {{null}} and regions weren't 
 deployed onto region-servers.
 {code}
 table_ACreating table table_A
 java.io.IOException: java.io.IOException: 
 java.util.ConcurrentModificationException
   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
   at 
 sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
   at 
 sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
   at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
   at 
 org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
   at 
 org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
   at 
 org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:384)
   at 
 org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:294)
   at com.test.tools.hbase.schema.createIfNotExists(schema.java:520)
   at com.test.tools.hbase.schema.main(schema.java:627)
 Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: 
 java.util.ConcurrentModificationException
   at java.util.SubList.checkForComodification(AbstractList.java:752)
   at java.util.SubList.add(AbstractList.java:632)
   at java.util.SubList.add(AbstractList.java:633)
   at java.util.SubList.add(AbstractList.java:633)
   ..
   ..
   at java.util.SubList.add(AbstractList.java:633)
   at java.util.AbstractList.add(AbstractList.java:91)
   at 
 org.apache.hadoop.hbase.monitoring.TaskMonitor.createStatus(TaskMonitor.java:76)
   at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:510)
   at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:490)
   at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:853)
   at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:813)
   at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:780)
   at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
   at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
   at 
 org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
   at $Proxy5.createTable(Unknown Source)
   at 
 org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:382)
   
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5642) [findbugs] Exclude Thrift and Protobuf warnings

2012-03-28 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13240467#comment-13240467
 ] 

Jonathan Hsieh commented on HBASE-5642:
---

The o.a.h.h.generated.master code looks to be generated from jsp's/jamon code 
and seems more potentially serious.  From the warning there, that the string 
needs to be sanitized before being re-displayed (as opposed to being ignored).



 [findbugs] Exclude Thrift and Protobuf warnings
 ---

 Key: HBASE-5642
 URL: https://issues.apache.org/jira/browse/HBASE-5642
 Project: HBase
  Issue Type: Sub-task
  Components: build
Affects Versions: 0.96.0
Reporter: Jonathan Hsieh
Assignee: Uma Maheswara Rao G
 Attachments: HBASE-5642.patch, HBASE-5642.patch


 Exclude thrift and protobuf warnings since these are machine generated.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5642) [findbugs] Exclude Thrift and Protobuf warnings

2012-03-28 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13240474#comment-13240474
 ] 

Jonathan Hsieh commented on HBASE-5642:
---

Yup, I'm in the process of doing a quick test and committing.  Thanks Uma!


 [findbugs] Exclude Thrift and Protobuf warnings
 ---

 Key: HBASE-5642
 URL: https://issues.apache.org/jira/browse/HBASE-5642
 Project: HBase
  Issue Type: Sub-task
  Components: build
Affects Versions: 0.96.0
Reporter: Jonathan Hsieh
Assignee: Uma Maheswara Rao G
 Attachments: HBASE-5642.patch, HBASE-5642.patch


 Exclude thrift and protobuf warnings since these are machine generated.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5598) Analyse and fix the findbugs reporting by QA and add invalid bugs into findbugs-excludeFilter file

2012-03-27 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239490#comment-13239490
 ] 

Jonathan Hsieh commented on HBASE-5598:
---

Uma, good stuff.  Please take a look at this thread from the mailing list if 
you haven't seen it yet.

http://mail-archives.apache.org/mod_mbox/hbase-dev/201203.mbox/browser

I'd suggest updating the findbugs warning count in 
dev-support/test-patch.properties.  Mind moving it over to HBASE-5642?  We can 
keep this on as the umbrella issue.

For the other folks interested, here's some link on how to add 
findbugs/findbugs excludes for those who want to jump on some issues.

http://findbugs.sourceforge.net/manual/filter.html
http://mojo.codehaus.org/findbugs-maven-plugin-2.4.0/

You can run findbugs locally via 'mvn findbugs:findbugs'.  I thought we auto 
gen'ed the file for the website -- making findbugs run and generate html when 
'mvn site' is run might be another subtask to add.


 Analyse and fix the findbugs reporting by QA and add invalid bugs into 
 findbugs-excludeFilter file
 --

 Key: HBASE-5598
 URL: https://issues.apache.org/jira/browse/HBASE-5598
 Project: HBase
  Issue Type: Bug
  Components: scripts
Affects Versions: 0.92.1, 0.94.0, 0.96.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
Priority: Minor
 Attachments: HBASE-5598.patch, findbugs-gui-report.jpg


 There are many findbugs errors reporting by HbaseQA. HBASE-5597 is going to 
 up the OK count.
 This may lead to other issues when we re-factor the code, if we induce new 
 valid ones and remove invalid bugs also can not be reported by QA.
 So, I would propose to add the exclude filter file for findbugs(for the 
 invalid bugs). If we find any valid ones, we can fix under this JIRA.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5598) Analyse and fix the findbugs reporting by QA and add invalid bugs into findbugs-excludeFilter file

2012-03-27 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239639#comment-13239639
 ] 

Jonathan Hsieh commented on HBASE-5598:
---


bq. Sure, I will update the patch in HBASE-5642. But regarding updating the 
count, I am not agreeing. I am planning to use exclude filter file to exclude 
the invalid bugs. Please see the discussion in HBASE-5597 about updating the 
count. Please correct me if i understood wrongly your question. Thanks a lot.

Here's what I'm suggesting.  Eventually we set the number to 0 due to the 
excludes files.  Meanwhile, as we commit exclude/fix these warnings patches, we 
update the count to the number you expect to see after the patch.  Patches in 
flight would just see the same number of new warnings it introduced instead of 
warnings from disappearing as you mentioned in HBASE-5597.  


 Analyse and fix the findbugs reporting by QA and add invalid bugs into 
 findbugs-excludeFilter file
 --

 Key: HBASE-5598
 URL: https://issues.apache.org/jira/browse/HBASE-5598
 Project: HBase
  Issue Type: Bug
  Components: scripts
Affects Versions: 0.92.1, 0.94.0, 0.96.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
Priority: Minor
 Attachments: HBASE-5598.patch, findbugs-gui-report.jpg


 There are many findbugs errors reporting by HbaseQA. HBASE-5597 is going to 
 up the OK count.
 This may lead to other issues when we re-factor the code, if we induce new 
 valid ones and remove invalid bugs also can not be reported by QA.
 So, I would propose to add the exclude filter file for findbugs(for the 
 invalid bugs). If we find any valid ones, we can fix under this JIRA.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5128) [uber hbck] Online automated repair of table integrity and region consistency problems

2012-03-27 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239959#comment-13239959
 ] 

Jonathan Hsieh commented on HBASE-5128:
---

Docs jira is here: HBASE-5634.  


 [uber hbck] Online automated repair of table integrity and region consistency 
 problems
 --

 Key: HBASE-5128
 URL: https://issues.apache.org/jira/browse/HBASE-5128
 Project: HBase
  Issue Type: New Feature
  Components: hbck
Affects Versions: 0.90.5, 0.92.0, 0.94.0, 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0

 Attachments: 5128-trunk.addendum, hbase-5128-0.90-v2.patch, 
 hbase-5128-0.90-v2b.patch, hbase-5128-0.90-v4.patch, 
 hbase-5128-0.92-v2.patch, hbase-5128-0.92-v4.patch, hbase-5128-0.94-v2.patch, 
 hbase-5128-0.94-v4.patch, hbase-5128-trunk-v2.patch, hbase-5128-trunk.patch, 
 hbase-5128-v3.patch, hbase-5128-v4.patch


 The current (0.90.5, 0.92.0rc2) versions of hbck detects most of region 
 consistency and table integrity invariant violations.  However with '-fix' it 
 can only automatically repair region consistency cases having to do with 
 deployment problems.  This updated version should be able to handle all cases 
 (including a new orphan regiondir case).  When complete will likely deprecate 
 the OfflineMetaRepair tool and subsume several open META-hole related issue.
 Here's the approach (from the comment of at the top of the new version of the 
 file).
 {code}
 /**
  * HBaseFsck (hbck) is a tool for checking and repairing region consistency 
 and
  * table integrity.  
  * 
  * Region consistency checks verify that META, region deployment on
  * region servers and the state of data in HDFS (.regioninfo files) all are in
  * accordance. 
  * 
  * Table integrity checks verify that that all possible row keys can resolve 
 to
  * exactly one region of a table.  This means there are no individual 
 degenerate
  * or backwards regions; no holes between regions; and that there no 
 overlapping
  * regions. 
  * 
  * The general repair strategy works in these steps.
  * 1) Repair Table Integrity on HDFS. (merge or fabricate regions)
  * 2) Repair Region Consistency with META and assignments
  * 
  * For table integrity repairs, the tables their region directories are 
 scanned
  * for .regioninfo files.  Each table's integrity is then verified.  If there 
  * are any orphan regions (regions with no .regioninfo files), or holes, new 
  * regions are fabricated.  Backwards regions are sidelined as well as empty
  * degenerate (endkey==startkey) regions.  If there are any overlapping 
 regions,
  * a new region is created and all data is merged into the new region.  
  * 
  * Table integrity repairs deal solely with HDFS and can be done offline -- 
 the
  * hbase region servers or master do not need to be running.  These phase can 
 be
  * use to completely reconstruct the META table in an offline fashion. 
  * 
  * Region consistency requires three conditions -- 1) valid .regioninfo file 
  * present in an hdfs region dir,  2) valid row with .regioninfo data in META,
  * and 3) a region is deployed only at the regionserver that is was assigned 
 to.
  * 
  * Region consistency requires hbck to contact the HBase master and region
  * servers, so the connect() must first be called successfully.  Much of the
  * region consistency information is transient and less risky to repair.
  */
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5577) improve 'patch submission' section in HBase book

2012-03-27 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13240129#comment-13240129
 ] 

Jonathan Hsieh commented on HBASE-5577:
---

Sujee -- ideally you'd modify the docs source code -- I believe the files is  
src/docbkx/developer.xml -- and submit a patch. :)

 improve 'patch submission' section in HBase book
 

 Key: HBASE-5577
 URL: https://issues.apache.org/jira/browse/HBASE-5577
 Project: HBase
  Issue Type: Improvement
  Components: documentation
Reporter: Sujee Maniyam
Assignee: Sujee Maniyam
  Labels: documentation
 Attachments: HBASE-5577-v1.txt, HBASE-5577-v2.txt


 Improve patch section in the book 
 http://hbase.apache.org/book/submitting.patches.html

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5637) Fix failing 0.90 TestHMsg testcase introduced by HBASE-5563.

2012-03-26 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238565#comment-13238565
 ] 

Jonathan Hsieh commented on HBASE-5637:
---

This test case does not exist in 0.92/0.94/trunk and is not relevent to it.

 Fix failing 0.90 TestHMsg testcase introduced by HBASE-5563.
 

 Key: HBASE-5637
 URL: https://issues.apache.org/jira/browse/HBASE-5637
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.7
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Attachments: hbase-5637.patch


 After committing HBASE-5128 and HBASE-5563 to the 0.90 branch, Ted noticed 
 that TestHMsg#getList started to fail consistently.  This updates the test to 
 deal with the updated equality semantics in HBASE-5563.  This fixes that.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5596) Few minor bugs from HBASE-5209

2012-03-26 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238591#comment-13238591
 ] 

Jonathan Hsieh commented on HBASE-5596:
---

+1, lgtm.  Since this is an addendum to HBASE-5209, I've tweaked it to work on 
0.94 and 0.92.  I'll run them through test suites and if they pass tests I'll 
commit.  



 Few minor bugs from HBASE-5209
 --

 Key: HBASE-5596
 URL: https://issues.apache.org/jira/browse/HBASE-5596
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.92.1, 0.94.0
Reporter: David S. Wang
Assignee: David S. Wang
Priority: Minor
 Attachments: HBASE-5596.patch, hbase-5596-0.94.patch


 A few leftover bugs from HBASE-5209.  Comments are documented here:
 https://reviews.apache.org/r/3892/

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5637) Fix failing 0.90 TestHMsg testcase introduced by HBASE-5563.

2012-03-26 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238609#comment-13238609
 ] 

Jonathan Hsieh commented on HBASE-5637:
---

Committed.  Dave and Stack, thanks for review.

 Fix failing 0.90 TestHMsg testcase introduced by HBASE-5563.
 

 Key: HBASE-5637
 URL: https://issues.apache.org/jira/browse/HBASE-5637
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.7
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Fix For: 0.90.7

 Attachments: hbase-5637.patch


 After committing HBASE-5128 and HBASE-5563 to the 0.90 branch, Ted noticed 
 that TestHMsg#getList started to fail consistently.  This updates the test to 
 deal with the updated equality semantics in HBASE-5563.  This fixes that.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5628) Improve performance of uberhbck

2012-03-26 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238721#comment-13238721
 ] 

Jonathan Hsieh commented on HBASE-5628:
---

Minor optimization: 

in HBaseFsck#loadHdfsRegionInfos we could consolidate.

{code}
if (modTInfo = null) {
 ...
}
..
tablesInfo.put(tableNAme, modTInfo);
{code}

to be 


{code}
if (modTInfo = null) {
  ...
  tablesInfo.put(tableNAme, modTInfo);
}
..
{code}

 Improve performance of uberhbck
 ---

 Key: HBASE-5628
 URL: https://issues.apache.org/jira/browse/HBASE-5628
 Project: HBase
  Issue Type: Improvement
  Components: hbck
Affects Versions: 0.90.7, 0.92.2, 0.94.0, 0.96.0
Reporter: Jonathan Hsieh

 During reviews of HBASE-5128 there are several opportunities investigate for 
 improving the performance of the tool.
 - Change regionInfoMap and tablesInfo from TreeMap to HashMap.
 - Change some full region set reloads to be incremental to require fewer 
 passes.
 - Cache meta for subsequent calls of closeRegionSileneglyAndWait

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5128) [uber hbck] Online automated repair of table integrity and region consistency problems

2012-03-24 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237474#comment-13237474
 ] 

Jonathan Hsieh commented on HBASE-5128:
---

Thanks Ted.  I've updated the rest.  Will do better next time. :)

 [uber hbck] Online automated repair of table integrity and region consistency 
 problems
 --

 Key: HBASE-5128
 URL: https://issues.apache.org/jira/browse/HBASE-5128
 Project: HBase
  Issue Type: New Feature
  Components: hbck
Affects Versions: 0.90.5, 0.92.0, 0.94.0, 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0

 Attachments: 5128-trunk.addendum, hbase-5128-0.90-v2.patch, 
 hbase-5128-0.90-v2b.patch, hbase-5128-0.90-v4.patch, 
 hbase-5128-0.92-v2.patch, hbase-5128-0.92-v4.patch, hbase-5128-0.94-v2.patch, 
 hbase-5128-0.94-v4.patch, hbase-5128-trunk-v2.patch, hbase-5128-trunk.patch, 
 hbase-5128-v3.patch, hbase-5128-v4.patch


 The current (0.90.5, 0.92.0rc2) versions of hbck detects most of region 
 consistency and table integrity invariant violations.  However with '-fix' it 
 can only automatically repair region consistency cases having to do with 
 deployment problems.  This updated version should be able to handle all cases 
 (including a new orphan regiondir case).  When complete will likely deprecate 
 the OfflineMetaRepair tool and subsume several open META-hole related issue.
 Here's the approach (from the comment of at the top of the new version of the 
 file).
 {code}
 /**
  * HBaseFsck (hbck) is a tool for checking and repairing region consistency 
 and
  * table integrity.  
  * 
  * Region consistency checks verify that META, region deployment on
  * region servers and the state of data in HDFS (.regioninfo files) all are in
  * accordance. 
  * 
  * Table integrity checks verify that that all possible row keys can resolve 
 to
  * exactly one region of a table.  This means there are no individual 
 degenerate
  * or backwards regions; no holes between regions; and that there no 
 overlapping
  * regions. 
  * 
  * The general repair strategy works in these steps.
  * 1) Repair Table Integrity on HDFS. (merge or fabricate regions)
  * 2) Repair Region Consistency with META and assignments
  * 
  * For table integrity repairs, the tables their region directories are 
 scanned
  * for .regioninfo files.  Each table's integrity is then verified.  If there 
  * are any orphan regions (regions with no .regioninfo files), or holes, new 
  * regions are fabricated.  Backwards regions are sidelined as well as empty
  * degenerate (endkey==startkey) regions.  If there are any overlapping 
 regions,
  * a new region is created and all data is merged into the new region.  
  * 
  * Table integrity repairs deal solely with HDFS and can be done offline -- 
 the
  * hbase region servers or master do not need to be running.  These phase can 
 be
  * use to completely reconstruct the META table in an offline fashion. 
  * 
  * Region consistency requires three conditions -- 1) valid .regioninfo file 
  * present in an hdfs region dir,  2) valid row with .regioninfo data in META,
  * and 3) a region is deployed only at the regionserver that is was assigned 
 to.
  * 
  * Region consistency requires hbck to contact the HBase master and region
  * servers, so the connect() must first be called successfully.  Much of the
  * region consistency information is transient and less risky to repair.
  */
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5604) HLog replay tool that generates HFiles for use by LoadIncrementalHFiles.

2012-03-24 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237543#comment-13237543
 ] 

Jonathan Hsieh commented on HBASE-5604:
---

Hm, so this is similar to this 
http://www.postgresql.org/docs/8.2/static/continuous-archiving.html? For hbase, 
this would to enable a consistent warm backup (though no strong guarantees 
across regions) that would be cheaper than the full replication mechanism?

 HLog replay tool that generates HFiles for use by LoadIncrementalHFiles.
 

 Key: HBASE-5604
 URL: https://issues.apache.org/jira/browse/HBASE-5604
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl

 Just an idea I had. Might be useful for restore of a backup using the HLogs.
 This could an M/R (with a mapper per HLog file).
 The tool would get a timerange and a (set of) table(s). We'd pick the right 
 HLogs based on time before the M/R job is started and then have a mapper per 
 HLog file.
 The mapper would then go through the HLog, filter all WALEdits that didn't 
 fit into the time range or are not any of the tables and then uses 
 HFileOutputFormat to generate HFiles.
 Would need to indicate the splits we want, probably from a live table.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5599) The hbkc tool can not fix the six scenarios, it is NO_VERSION_FILE, NOT_IN_META_OR_DEPLOYED, NOT_IN_META, SHOULD_NOT_BE_DEPLOYED, FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_

2012-03-24 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237554#comment-13237554
 ] 

Jonathan Hsieh commented on HBASE-5599:
---

Hi Fulin,

I've committed HBASE-5128 so let's work together to port the functionality 
you've implemented into it as well. 

* Some of the methods you've added in HBaseFsckRepair are similar in 
HBASE-5128.  Let's consolidate.
* It seems that you are focused on the 0.90 versions -- it is important that 
whatever changes we make also make it into the newer 0.92/0.94/trunk versions.  
I can definitely help out there.
* It is important to add some testing for the new scenarios as well -- checkout 
TestHbckFsck for some example to emulate.  I've basically started to have have 
one test for each error condition.




 The hbkc tool can not fix the six scenarios, it is NO_VERSION_FILE, 
 NOT_IN_META_OR_DEPLOYED, NOT_IN_META, SHOULD_NOT_BE_DEPLOYED, 
 FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_IN_REGION_CHAIN.
 

 Key: HBASE-5599
 URL: https://issues.apache.org/jira/browse/HBASE-5599
 Project: HBase
  Issue Type: New Feature
  Components: hbck
Affects Versions: 0.90.6
Reporter: fulin wang
 Fix For: 0.90.6

 Attachments: hbase-5599-0.90.patch, hbase-5599-0.90_v2.patch, 
 hbase-5599-0.90_v3.patch


 The hbck tool can not fix the six scenarios.
 1. Version file does not exist in root dir.
Fix: I try to create a version file by 'FSUtils.setVersion' method.

 2. [REGIONNAME][KEY] on HDFS, but not listed in META or deployed on any 
 region server.
Fix: I get region info form the hdfs file, this region info write to 
 '.META.' table.

 3. [REGIONNAME][KEY] not in META, but deployed on [SERVERNAME]
Fix: I get region info form the hdfs file, this region info write to 
 '.META.' table.

 4. [REGIONNAME] should not be deployed according to META, but is deployed on 
 [SERVERNAME]
Fix: Close this region.

 5. First region should start with an empty key.  You need to  create a new 
 region and regioninfo in HDFS to plug the hole.
Fix: The region info is not in hdfs and .META., so it create a empty 
 region for this error.
 6. There is a hole in the region chain between [KEY] and [KEY]. You need to 
 create a new regioninfo and region dir in hdfs to plug the hole.
   Fix: The region info is not in hdfs and .META., so it create a empty region 
 for this hole.
   

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5598) Analyse and fix the findbugs reporting by QA and add invalid bugs into findbugs-excludeFilter file

2012-03-24 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237582#comment-13237582
 ] 

Jonathan Hsieh commented on HBASE-5598:
---

Currently we are somewhere around the 770 warnings/errors mark.  We should chop 
this into subtasks to break down the work and knock out related issues.

For this to last, once this we get the findbugs warnings to 0, committers need 
to enforce a no-new-findbugs errors policy on reviews.  Agreed?

 Analyse and fix the findbugs reporting by QA and add invalid bugs into 
 findbugs-excludeFilter file
 --

 Key: HBASE-5598
 URL: https://issues.apache.org/jira/browse/HBASE-5598
 Project: HBase
  Issue Type: Bug
  Components: scripts
Affects Versions: 0.92.1, 0.94.0, 0.96.0
Reporter: Uma Maheswara Rao G
Priority: Minor

 There are many findbugs errors reporting by HbaseQA. HBASE-5597 is going to 
 up the OK count.
 This may lead to other issues when we re-factor the code, if we induce new 
 valid ones and remove invalid bugs also can not be reported by QA.
 So, I would propose to add the exclude filter file for findbugs(for the 
 invalid bugs). If we find any valid ones, we can fix under this JIRA.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5128) [uber hbck] Enable hbck to automatically repair table integrity problems as well as region consistency problems while online.

2012-03-23 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236704#comment-13236704
 ] 

Jonathan Hsieh commented on HBASE-5128:
---

Porting to changes to 0.90 is causing some test flakyness in that version.  My 
plan is to work these out (there are more constraints there -- need to figure 
out why they flake, need to avoid a master-side HBASE-5563 change, and figure 
out the ramifications.  I plan on opening a new issue to back port this patch 
to 0.90.  While trunk/0.94/0.92 versions are very similar, 0.90 has several 
differences.

 [uber hbck] Enable hbck to automatically repair table integrity problems as 
 well as region consistency problems while online.
 -

 Key: HBASE-5128
 URL: https://issues.apache.org/jira/browse/HBASE-5128
 Project: HBase
  Issue Type: New Feature
  Components: hbck
Affects Versions: 0.90.5, 0.92.0, 0.94.0, 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0

 Attachments: hbase-5128-0.90-v2.patch, hbase-5128-0.90-v2b.patch, 
 hbase-5128-0.92-v2.patch, hbase-5128-0.92-v4.patch, hbase-5128-0.94-v2.patch, 
 hbase-5128-0.94-v4.patch, hbase-5128-trunk-v2.patch, hbase-5128-trunk.patch, 
 hbase-5128-v3.patch, hbase-5128-v4.patch


 The current (0.90.5, 0.92.0rc2) versions of hbck detects most of region 
 consistency and table integrity invariant violations.  However with '-fix' it 
 can only automatically repair region consistency cases having to do with 
 deployment problems.  This updated version should be able to handle all cases 
 (including a new orphan regiondir case).  When complete will likely deprecate 
 the OfflineMetaRepair tool and subsume several open META-hole related issue.
 Here's the approach (from the comment of at the top of the new version of the 
 file).
 {code}
 /**
  * HBaseFsck (hbck) is a tool for checking and repairing region consistency 
 and
  * table integrity.  
  * 
  * Region consistency checks verify that META, region deployment on
  * region servers and the state of data in HDFS (.regioninfo files) all are in
  * accordance. 
  * 
  * Table integrity checks verify that that all possible row keys can resolve 
 to
  * exactly one region of a table.  This means there are no individual 
 degenerate
  * or backwards regions; no holes between regions; and that there no 
 overlapping
  * regions. 
  * 
  * The general repair strategy works in these steps.
  * 1) Repair Table Integrity on HDFS. (merge or fabricate regions)
  * 2) Repair Region Consistency with META and assignments
  * 
  * For table integrity repairs, the tables their region directories are 
 scanned
  * for .regioninfo files.  Each table's integrity is then verified.  If there 
  * are any orphan regions (regions with no .regioninfo files), or holes, new 
  * regions are fabricated.  Backwards regions are sidelined as well as empty
  * degenerate (endkey==startkey) regions.  If there are any overlapping 
 regions,
  * a new region is created and all data is merged into the new region.  
  * 
  * Table integrity repairs deal solely with HDFS and can be done offline -- 
 the
  * hbase region servers or master do not need to be running.  These phase can 
 be
  * use to completely reconstruct the META table in an offline fashion. 
  * 
  * Region consistency requires three conditions -- 1) valid .regioninfo file 
  * present in an hdfs region dir,  2) valid row with .regioninfo data in META,
  * and 3) a region is deployed only at the regionserver that is was assigned 
 to.
  * 
  * Region consistency requires hbck to contact the HBase master and region
  * servers, so the connect() must first be called successfully.  Much of the
  * region consistency information is transient and less risky to repair.
  */
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5608) MR testcases are failing in QA builds

2012-03-23 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237133#comment-13237133
 ] 

Jonathan Hsieh commented on HBASE-5608:
---

@Greg

I cannot access the physical machines to get the actual logs, but we should be 
able to configure jenkins to save particular files after a build.

Looks like trunk saves all build and test artifacts from the most recent run 
here 
https://builds.apache.org/view/G-L/view/HBase/job/PreCommit-HBASE-Build/ws/.  
Can you take a look and identify the file you need to help debug?

We can enable it for a little while until we get it resolved.

 MR testcases are failing in QA builds
 -

 Key: HBASE-5608
 URL: https://issues.apache.org/jira/browse/HBASE-5608
 Project: HBase
  Issue Type: Bug
  Components: build, mapreduce, test
Affects Versions: 0.92.2
 Environment: Hadoop QA - precommit builds
Reporter: Laxman
Priority: Blocker
  Labels: build-failure, mapreduce, test-fail

 Many of the MR testcases are failing in PreCommit builds (triggered by Hadoop 
 QA).
 Failing testcases are
 a) TestImportTsv
 b) TestHFileOutputFormat
 c) TestTableMapReduce

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5128) [uber hbck] Enable hbck to automatically repair table integrity problems as well as region consistency problems while online.

2012-03-23 Thread Jonathan Hsieh (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237241#comment-13237241
 ] 

Jonathan Hsieh commented on HBASE-5128:
---

Full suites of 0.92/0.94/trunk versions pass.

Looks like the 0.90 version has always had flakey tests for the same reason the 
0.92/0.94/trunk versions.  It is related to assignment and HBASE-5563, but just 
didn't  happen as often as in the 0.92/0.94/trunk version (2/10 runs vs 5/10 
runs).  HBASE-5563 would not be available on older clusters but won't cause 
permanent problems if this updated hbck was used against a version that did not 
have the improvement.

Let's say using this hbck against an older 0.90-based cluster that didn't have  
HBASE-5563 or HBASE-5589.  The side effect is that you may have to run 'hbck 
-fixAssignments' an extra time to fix region assignment/deployment problems 
after disabling and deleting a table that has been fixed, or alternately, you 
may need to bounce the HMaster or affected RegionServer to clean up this 
transient state.

I currently have a 0.90 version of HBASE-5563 (attached there), and an updated 
HBASE-5128 for 0.90 that is as close as possible to the 0.92/0.94/trunk 
versions as possible.



 [uber hbck] Enable hbck to automatically repair table integrity problems as 
 well as region consistency problems while online.
 -

 Key: HBASE-5128
 URL: https://issues.apache.org/jira/browse/HBASE-5128
 Project: HBase
  Issue Type: New Feature
  Components: hbck
Affects Versions: 0.90.5, 0.92.0, 0.94.0, 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0

 Attachments: hbase-5128-0.90-v2.patch, hbase-5128-0.90-v2b.patch, 
 hbase-5128-0.92-v2.patch, hbase-5128-0.92-v4.patch, hbase-5128-0.94-v2.patch, 
 hbase-5128-0.94-v4.patch, hbase-5128-trunk-v2.patch, hbase-5128-trunk.patch, 
 hbase-5128-v3.patch, hbase-5128-v4.patch


 The current (0.90.5, 0.92.0rc2) versions of hbck detects most of region 
 consistency and table integrity invariant violations.  However with '-fix' it 
 can only automatically repair region consistency cases having to do with 
 deployment problems.  This updated version should be able to handle all cases 
 (including a new orphan regiondir case).  When complete will likely deprecate 
 the OfflineMetaRepair tool and subsume several open META-hole related issue.
 Here's the approach (from the comment of at the top of the new version of the 
 file).
 {code}
 /**
  * HBaseFsck (hbck) is a tool for checking and repairing region consistency 
 and
  * table integrity.  
  * 
  * Region consistency checks verify that META, region deployment on
  * region servers and the state of data in HDFS (.regioninfo files) all are in
  * accordance. 
  * 
  * Table integrity checks verify that that all possible row keys can resolve 
 to
  * exactly one region of a table.  This means there are no individual 
 degenerate
  * or backwards regions; no holes between regions; and that there no 
 overlapping
  * regions. 
  * 
  * The general repair strategy works in these steps.
  * 1) Repair Table Integrity on HDFS. (merge or fabricate regions)
  * 2) Repair Region Consistency with META and assignments
  * 
  * For table integrity repairs, the tables their region directories are 
 scanned
  * for .regioninfo files.  Each table's integrity is then verified.  If there 
  * are any orphan regions (regions with no .regioninfo files), or holes, new 
  * regions are fabricated.  Backwards regions are sidelined as well as empty
  * degenerate (endkey==startkey) regions.  If there are any overlapping 
 regions,
  * a new region is created and all data is merged into the new region.  
  * 
  * Table integrity repairs deal solely with HDFS and can be done offline -- 
 the
  * hbase region servers or master do not need to be running.  These phase can 
 be
  * use to completely reconstruct the META table in an offline fashion. 
  * 
  * Region consistency requires three conditions -- 1) valid .regioninfo file 
  * present in an hdfs region dir,  2) valid row with .regioninfo data in META,
  * and 3) a region is deployed only at the regionserver that is was assigned 
 to.
  * 
  * Region consistency requires hbck to contact the HBase master and region
  * servers, so the connect() must first be called successfully.  Much of the
  * region consistency information is transient and less risky to repair.
  */
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: 

  1   2   3   4   >