[jira] [Updated] (HBASE-5887) Make TestAcidGuarantees usable for system testing.

2012-04-27 Thread Jonathan Hsieh (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-5887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-5887:
--

Comment: was deleted

(was: -1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12524790/hbase-5887-92.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 3 new or modified tests.

-1 patch.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1662//console

This message is automatically generated.)

 Make TestAcidGuarantees usable for system testing.
 --

 Key: HBASE-5887
 URL: https://issues.apache.org/jira/browse/HBASE-5887
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.92.0, 0.92.1, 0.94.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Attachments: hbase-5887-92.patch, hbase-5887.patch


 Currently, the TestAcidGuarantees run via main() will always abort with an 
 NPE because it digs into a non-existant HBaseTestingUtility for a flusher 
 thread.  We should tool this up so that it works properly from the command 
 line.  This would be a very useful long running test when used in conjunction 
 with fault injections to verify row acid properties.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-5872) Improve hadoopqa script to include checks for hadoop 0.23 build

2012-04-27 Thread Jonathan Hsieh (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-5872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-5872:
--

Attachment: hbase-5872-part2.patch

An addendum that makes the success/failure message explicit and more 
informative.

 Improve hadoopqa script to include checks for hadoop 0.23 build
 ---

 Key: HBASE-5872
 URL: https://issues.apache.org/jira/browse/HBASE-5872
 Project: HBase
  Issue Type: New Feature
Affects Versions: 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Fix For: 0.96.0

 Attachments: hbase-5872-part2.patch, hbase-5872.patch


 There have been a few patches that have made it into hbase trunk that have 
 broken the compile of hbase against hadoop 0.23.x, without being known for a 
 few days.
 We could have the bot do a few things:
 1) verify that patch compiles against hadoop 23
 2) verify that unit tests pass against hadoop 23

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5872) Improve hadoopqa script to include checks for hadoop 0.23 build

2012-04-27 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263483#comment-13263483
 ] 

Hudson commented on HBASE-5872:
---

Integrated in HBase-TRUNK #2821 (See 
[https://builds.apache.org/job/HBase-TRUNK/2821/])
HBASE-5872 ADDENDUM - improve hadoop 23 compile sucess/fail message 
(Revision 1331248)

 Result = FAILURE
jmhsieh : 
Files : 
* /hbase/trunk/dev-support/test-patch.sh


 Improve hadoopqa script to include checks for hadoop 0.23 build
 ---

 Key: HBASE-5872
 URL: https://issues.apache.org/jira/browse/HBASE-5872
 Project: HBase
  Issue Type: New Feature
Affects Versions: 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Fix For: 0.96.0

 Attachments: hbase-5872-part2.patch, hbase-5872.patch


 There have been a few patches that have made it into hbase trunk that have 
 broken the compile of hbase against hadoop 0.23.x, without being known for a 
 few days.
 We could have the bot do a few things:
 1) verify that patch compiles against hadoop 23
 2) verify that unit tests pass against hadoop 23

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-5886) Add new metric for possible data loss due to puts without WAL

2012-04-27 Thread Matteo Bertozzi (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Bertozzi updated HBASE-5886:
---

Attachment: HBASE-5886-v1.patch

Renamed metrics and function, removed unused  DEFAULT_ constant and the log on 
flush.

I'm not sure about how to get the client ip from the region, since the region 
shouldn't be aware of the client. Maybe moving something in Region Server?

heapSize() reflect the in memory state, so you've pointer size and so on... 
maybe using kv.getValueLength() + kv.getKeyLength() or just valueLength()?

I've to look at OperationMetrics, but seems that updatePutMetrics() is not 
called for doMiniBatchPut()... (I've to take a look at that)

 Add new metric for possible data loss due to puts without WAL 
 --

 Key: HBASE-5886
 URL: https://issues.apache.org/jira/browse/HBASE-5886
 Project: HBase
  Issue Type: New Feature
  Components: metrics, regionserver
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
  Labels: metrics
 Attachments: HBASE-5886-v0.patch, HBASE-5886-v1.patch


 Add a metrics to keep track of puts without WAL and possible data loss size.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-5886) Add new metric for possible data loss due to puts without WAL

2012-04-27 Thread Matteo Bertozzi (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Bertozzi updated HBASE-5886:
---

Attachment: HBASE-5886-v1.patch

 Add new metric for possible data loss due to puts without WAL 
 --

 Key: HBASE-5886
 URL: https://issues.apache.org/jira/browse/HBASE-5886
 Project: HBase
  Issue Type: New Feature
  Components: metrics, regionserver
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
  Labels: metrics
 Attachments: HBASE-5886-v0.patch, HBASE-5886-v1.patch


 Add a metrics to keep track of puts without WAL and possible data loss size.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-5886) Add new metric for possible data loss due to puts without WAL

2012-04-27 Thread Matteo Bertozzi (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Bertozzi updated HBASE-5886:
---

Attachment: (was: HBASE-5886-v1.patch)

 Add new metric for possible data loss due to puts without WAL 
 --

 Key: HBASE-5886
 URL: https://issues.apache.org/jira/browse/HBASE-5886
 Project: HBase
  Issue Type: New Feature
  Components: metrics, regionserver
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
  Labels: metrics
 Attachments: HBASE-5886-v0.patch, HBASE-5886-v1.patch


 Add a metrics to keep track of puts without WAL and possible data loss size.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5886) Add new metric for possible data loss due to puts without WAL

2012-04-27 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263575#comment-13263575
 ] 

Hadoop QA commented on HBASE-5886:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12524847/HBASE-5886-v1.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

+1 hadoop23.  The patch compiles against the hadoop 0.23.x profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 2 new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.io.TestHeapSize

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1668//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1668//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1668//console

This message is automatically generated.

 Add new metric for possible data loss due to puts without WAL 
 --

 Key: HBASE-5886
 URL: https://issues.apache.org/jira/browse/HBASE-5886
 Project: HBase
  Issue Type: New Feature
  Components: metrics, regionserver
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
  Labels: metrics
 Attachments: HBASE-5886-v0.patch, HBASE-5886-v1.patch


 Add a metrics to keep track of puts without WAL and possible data loss size.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-5856) byte - String is not consistent between HBaseAdmin and HRegionInfo

2012-04-27 Thread binlijin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-5856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

binlijin updated HBASE-5856:


Affects Version/s: (was: 0.90.6)

 byte - String is not consistent between HBaseAdmin and HRegionInfo
 

 Key: HBASE-5856
 URL: https://issues.apache.org/jira/browse/HBASE-5856
 Project: HBase
  Issue Type: Bug
  Components: client
Affects Versions: 0.92.1
Reporter: binlijin
 Attachments: HBASE-5856-0.92.patch


 In HBaseAdmin 
   public void split(final String tableNameOrRegionName)
   throws IOException, InterruptedException {
 split(Bytes.toBytes(tableNameOrRegionName));  // string - byte 
   }
 In HRegionInfo
   this.regionNameStr = Bytes.toStringBinary(this.regionName);  // byte - 
 string
 Should we use Bytes.toBytesBinary in HBaseAdmin ?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5856) byte - String is not consistent between HBaseAdmin and HRegionInfo

2012-04-27 Thread binlijin (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263582#comment-13263582
 ] 

binlijin commented on HBASE-5856:
-

@stack
Yes, I find the single quotes will fail and double quotes success. Thanks stack.

 byte - String is not consistent between HBaseAdmin and HRegionInfo
 

 Key: HBASE-5856
 URL: https://issues.apache.org/jira/browse/HBASE-5856
 Project: HBase
  Issue Type: Bug
  Components: client
Affects Versions: 0.92.1
Reporter: binlijin
 Attachments: HBASE-5856-0.92.patch


 In HBaseAdmin 
   public void split(final String tableNameOrRegionName)
   throws IOException, InterruptedException {
 split(Bytes.toBytes(tableNameOrRegionName));  // string - byte 
   }
 In HRegionInfo
   this.regionNameStr = Bytes.toStringBinary(this.regionName);  // byte - 
 string
 Should we use Bytes.toBytesBinary in HBaseAdmin ?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5874) The HBase do not configure the 'fs.default.name' attribute, the hbck tool and Merge tool throw IllegalArgumentException.

2012-04-27 Thread Jonathan Hsieh (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263681#comment-13263681
 ] 

Jonathan Hsieh commented on HBASE-5874:
---

Fulin:

Have you tried using the tool after the patch?  

Do you have a dir with hdfs-site.xml in it in your 'hbase classpath'?

Actually looking at it, I don't think the original hbck line that potentially 
sets fs.defaultFS to the HBASE root dir makes sense at all.  Setting the 
fs.default.name also seems incorrect.  I'm actually more inclined to just 
remove the line.  Maybe it would be better to allow the user to override the 
value by converting hbck  into subclass of Tool?

In the case of Merge, it is a Tool already so if you don't have your 
hdfs-site.xml files in the hbase classpath you could run it with an extra flag 
to override particular properties such as fs.default.name.

{code}
hbase org.apache.hadoop.hbase.util.Merge -Dfs.default.name=hdfs://nn:port 
normal args
{code}


 The HBase do not configure the 'fs.default.name' attribute, the hbck tool and 
 Merge tool throw IllegalArgumentException.
 

 Key: HBASE-5874
 URL: https://issues.apache.org/jira/browse/HBASE-5874
 Project: HBase
  Issue Type: Bug
  Components: hbck
Affects Versions: 0.90.6
Reporter: fulin wang
 Attachments: HBASE-5874-0.90.patch, HBASE-5874-trunk.patch


 The HBase do not configure the 'fs.default.name' attribute, the hbck tool and 
 Merge tool throw IllegalArgumentException.
 the hbck tool and Merge tool, we should add 'fs.default.name' attriubte to 
 the code.
 hbck exception:
 Exception in thread main java.lang.IllegalArgumentException: Wrong FS: 
 hdfs://160.176.0.101:9000/hbase/.META./1028785192/.regioninfo, expected: 
 file:///
   at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:412)
   at 
 org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:59)
   at 
 org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:382)
   at 
 org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:285)
   at 
 org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.init(ChecksumFileSystem.java:128)
   at 
 org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:301)
   at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:489)
   at 
 org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegioninfo(HBaseFsck.java:565)
   at 
 org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionInfos(HBaseFsck.java:596)
   at 
 org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:332)
   at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:360)
   at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:2907)
 
 Merge exception:  
 [2012-05-05 10:48:24,830] [ERROR] [main] [org.apache.hadoop.hbase.util.Merge 
 381] exiting due to error
 java.lang.IllegalArgumentException: Wrong FS: 
 hdfs://160.176.0.101:9000/hbase/.META./1028785192/.regioninfo, expected: 
 file:///
   at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:412)
   at 
 org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:59)
   at 
 org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:382)
   at 
 org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:285)
   at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:823)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkRegioninfoOnFilesystem(HRegion.java:415)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:340)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2679)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2665)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2634)
   at 
 org.apache.hadoop.hbase.util.MetaUtils.openMetaRegion(MetaUtils.java:276)
   at 
 org.apache.hadoop.hbase.util.MetaUtils.scanMetaRegion(MetaUtils.java:261)
   at org.apache.hadoop.hbase.util.Merge.run(Merge.java:115)
   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
   at org.apache.hadoop.hbase.util.Merge.main(Merge.java:379)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5864) Error while reading from hfile in 0.94

2012-04-27 Thread Jonathan Hsieh (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263684#comment-13263684
 ] 

Jonathan Hsieh commented on HBASE-5864:
---

I haven't looked carefully but do you all think this would affect the 0.92 line 
as well?

 Error while reading from hfile in 0.94
 --

 Key: HBASE-5864
 URL: https://issues.apache.org/jira/browse/HBASE-5864
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
 Fix For: 0.94.0

 Attachments: HBASE-5864_1.patch, HBASE-5864_2.patch, 
 HBASE-5864_3.patch, HBASE-5864_test.patch


 Got the following stacktrace during region split.
 {noformat}
 2012-04-24 16:05:42,168 WARN org.apache.hadoop.hbase.regionserver.Store: 
 Failed getting store size for value
 java.io.IOException: Requested block is out of range: 2906737606134037404, 
 lastDataBlockOffset: 84764558
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:278)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:285)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:402)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1638)
   at 
 org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1943)
   at 
 org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:77)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:4921)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:2901)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Assigned] (HBASE-5874) The HBase do not configure the 'fs.default.name' attribute, the hbck tool and Merge tool throw IllegalArgumentException.

2012-04-27 Thread Jonathan Hsieh (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh reassigned HBASE-5874:
-

Assignee: fulin wang

Hey fulin, in the future you can set yourself as the assignee to an issue if 
you are working on it.

 The HBase do not configure the 'fs.default.name' attribute, the hbck tool and 
 Merge tool throw IllegalArgumentException.
 

 Key: HBASE-5874
 URL: https://issues.apache.org/jira/browse/HBASE-5874
 Project: HBase
  Issue Type: Bug
  Components: hbck
Affects Versions: 0.90.6
Reporter: fulin wang
Assignee: fulin wang
 Attachments: HBASE-5874-0.90.patch, HBASE-5874-trunk.patch


 The HBase do not configure the 'fs.default.name' attribute, the hbck tool and 
 Merge tool throw IllegalArgumentException.
 the hbck tool and Merge tool, we should add 'fs.default.name' attriubte to 
 the code.
 hbck exception:
 Exception in thread main java.lang.IllegalArgumentException: Wrong FS: 
 hdfs://160.176.0.101:9000/hbase/.META./1028785192/.regioninfo, expected: 
 file:///
   at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:412)
   at 
 org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:59)
   at 
 org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:382)
   at 
 org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:285)
   at 
 org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.init(ChecksumFileSystem.java:128)
   at 
 org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:301)
   at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:489)
   at 
 org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegioninfo(HBaseFsck.java:565)
   at 
 org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionInfos(HBaseFsck.java:596)
   at 
 org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:332)
   at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:360)
   at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:2907)
 
 Merge exception:  
 [2012-05-05 10:48:24,830] [ERROR] [main] [org.apache.hadoop.hbase.util.Merge 
 381] exiting due to error
 java.lang.IllegalArgumentException: Wrong FS: 
 hdfs://160.176.0.101:9000/hbase/.META./1028785192/.regioninfo, expected: 
 file:///
   at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:412)
   at 
 org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:59)
   at 
 org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:382)
   at 
 org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:285)
   at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:823)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkRegioninfoOnFilesystem(HRegion.java:415)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:340)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2679)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2665)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2634)
   at 
 org.apache.hadoop.hbase.util.MetaUtils.openMetaRegion(MetaUtils.java:276)
   at 
 org.apache.hadoop.hbase.util.MetaUtils.scanMetaRegion(MetaUtils.java:261)
   at org.apache.hadoop.hbase.util.Merge.run(Merge.java:115)
   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
   at org.apache.hadoop.hbase.util.Merge.main(Merge.java:379)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5620) Convert the client protocol of HRegionInterface to PB

2012-04-27 Thread Jimmy Xiang (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263742#comment-13263742
 ] 

Jimmy Xiang commented on HBASE-5620:


Yes, I was thinking to eliminate the conversion layer if possible, probably 
starting from the server side.
As to I performance test I did, I think the 27us overhead is more meaningful 
than the percentage number.

 Convert the client protocol of HRegionInterface to PB
 -

 Key: HBASE-5620
 URL: https://issues.apache.org/jira/browse/HBASE-5620
 Project: HBase
  Issue Type: Sub-task
  Components: ipc, master, migration, regionserver
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0

 Attachments: hbase-5620-sec.patch, hbase-5620_v3.patch, 
 hbase-5620_v4.patch, hbase-5620_v4.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5885) Invalid HFile block magic on Local file System

2012-04-27 Thread stack (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263751#comment-13263751
 ] 

stack commented on HBASE-5885:
--

With one region with one file only in it, I run into the issue.  I can read the 
file outside of hbase using HFile tool fine.  When I scan, the problem happens 
at a random seeming point each time; i.e. run it once and its at key 162, run 
it again its at key 246, etc.  There does not seem to be correlation with index 
boundaries if I dump them out with:

{code}
pynchon:0.94 stack$ ./bin/hbase org.apache.hadoop.hbase.io.hfile.HFile -m -s -b 
-f 
/tmp/hbase-stack/hbase/TestTable/75c7e8e6612fdf1fa308f9a34104e847/info/2830d6bcf8e34767af9baaf94228559a
 
{code}

 Invalid HFile block magic on Local file System
 --

 Key: HBASE-5885
 URL: https://issues.apache.org/jira/browse/HBASE-5885
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.0, 0.96.0
Reporter: Elliott Clark
Priority: Blocker
 Fix For: 0.94.0


 ERROR: java.lang.RuntimeException: 
 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after 
 attempts=7, exceptions:
 Thu Apr 26 11:19:18 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: Could not iterate StoreFileScanner[HFileScanner for 
 reader 
 reader=file:/tmp/hbase-eclark/hbase/TestTable/e2d1c846363c75262cbfd85ea278b342/info/bae2681d63734066957b58fe791a0268,
  compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], 
 firstKey=01/info:data/1335463981520/Put, 
 lastKey=0002588100/info:data/1335463902296/Put, avgKeyLen=30, 
 avgValueLen=1000, entries=1215085, length=1264354417, 
 cur=000248/info:data/1335463994457/Put/vlen=1000/ts=0]
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:135)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95)
   at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:368)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.io.IOException: Invalid HFile block magic: 
 \xEC\xD5\x9D\xB4\xC2bfo
   at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153)
   at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock.init(HFileBlock.java:254)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1779)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:327)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:555)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:651)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130)
   ... 12 more
 Thu Apr 26 11:19:19 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: java.lang.IllegalArgumentException
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1132)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1121)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2420)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 

[jira] [Commented] (HBASE-5847) Support createTable(splitKeys) in Thrift

2012-04-27 Thread Nicolas Spiegelberg (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263756#comment-13263756
 ] 

Nicolas Spiegelberg commented on HBASE-5847:


@stack: I assume you're talking about HBASE-1744?  thrift2 does not have this 
API either.  I take it I should put the new API in thrift/Hbase.thrift?

 Support createTable(splitKeys) in Thrift
 

 Key: HBASE-5847
 URL: https://issues.apache.org/jira/browse/HBASE-5847
 Project: HBase
  Issue Type: Improvement
Reporter: Nicolas Spiegelberg
Assignee: Nicolas Spiegelberg
Priority: Trivial

 The Thrift API does not allow a user to create a table with multiple split 
 keys.  This is needed for a handful of new internal projects that are written 
 in PHP/C++.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-5785) Adding unit tests for protbuf utils introduced for HRegionInterface pb conversion

2012-04-27 Thread Jimmy Xiang (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang updated HBASE-5785:
---

Attachment: hbase-5785.patch

 Adding unit tests for protbuf utils introduced for HRegionInterface pb 
 conversion
 -

 Key: HBASE-5785
 URL: https://issues.apache.org/jira/browse/HBASE-5785
 Project: HBase
  Issue Type: Sub-task
  Components: ipc, master, migration, regionserver
Affects Versions: 0.96.0
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Critical
  Labels: noob
 Fix For: 0.96.0

 Attachments: hbase-5785.patch


 We need to add some unit tests for the probuf utilities to catch issues 
 earlier.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-5785) Adding unit tests for protbuf utils introduced for HRegionInterface pb conversion

2012-04-27 Thread Jimmy Xiang (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang updated HBASE-5785:
---

Status: Patch Available  (was: Open)

I added some tests.  I indeed found an issue.  It is helpful.
We can add more later on.

 Adding unit tests for protbuf utils introduced for HRegionInterface pb 
 conversion
 -

 Key: HBASE-5785
 URL: https://issues.apache.org/jira/browse/HBASE-5785
 Project: HBase
  Issue Type: Sub-task
  Components: ipc, master, migration, regionserver
Affects Versions: 0.96.0
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Critical
  Labels: noob
 Fix For: 0.96.0

 Attachments: hbase-5785.patch


 We need to add some unit tests for the probuf utilities to catch issues 
 earlier.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5885) Invalid HFile block magic on Local file System

2012-04-27 Thread Lars Hofhansl (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263768#comment-13263768
 ] 

Lars Hofhansl commented on HBASE-5885:
--

Scary. Just to confirm, you still only see this in local mode *and* with HBase 
checksums enabled, right?


 Invalid HFile block magic on Local file System
 --

 Key: HBASE-5885
 URL: https://issues.apache.org/jira/browse/HBASE-5885
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.0, 0.96.0
Reporter: Elliott Clark
Priority: Blocker
 Fix For: 0.94.0


 ERROR: java.lang.RuntimeException: 
 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after 
 attempts=7, exceptions:
 Thu Apr 26 11:19:18 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: Could not iterate StoreFileScanner[HFileScanner for 
 reader 
 reader=file:/tmp/hbase-eclark/hbase/TestTable/e2d1c846363c75262cbfd85ea278b342/info/bae2681d63734066957b58fe791a0268,
  compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], 
 firstKey=01/info:data/1335463981520/Put, 
 lastKey=0002588100/info:data/1335463902296/Put, avgKeyLen=30, 
 avgValueLen=1000, entries=1215085, length=1264354417, 
 cur=000248/info:data/1335463994457/Put/vlen=1000/ts=0]
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:135)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95)
   at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:368)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.io.IOException: Invalid HFile block magic: 
 \xEC\xD5\x9D\xB4\xC2bfo
   at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153)
   at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock.init(HFileBlock.java:254)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1779)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:327)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:555)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:651)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130)
   ... 12 more
 Thu Apr 26 11:19:19 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: java.lang.IllegalArgumentException
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1132)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1121)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2420)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.lang.IllegalArgumentException
   at java.nio.Buffer.position(Buffer.java:216)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:630)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130)
   

[jira] [Commented] (HBASE-5844) Delete the region servers znode after a regions server crash

2012-04-27 Thread nkeywal (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263778#comment-13263778
 ] 

nkeywal commented on HBASE-5844:


I found the issue, and (hopefully) a fix. I will have a new patch middle of 
next week, I will include the master znode in this one...

 Delete the region servers znode after a regions server crash
 

 Key: HBASE-5844
 URL: https://issues.apache.org/jira/browse/HBASE-5844
 Project: HBase
  Issue Type: Improvement
  Components: regionserver, scripts
Affects Versions: 0.96.0
Reporter: nkeywal
Assignee: nkeywal
 Fix For: 0.96.0

 Attachments: 5844.v1.patch, 5844.v2.patch, 5844.v3.patch


 today, if the regions server crashes, its znode is not deleted in ZooKeeper. 
 So the recovery process will stop only after a timeout, usually 30s.
 By deleting the znode in start script, we remove this delay and the recovery 
 starts immediately.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Assigned] (HBASE-5876) TestImportExport has been failing against hadoop 0.23 profile

2012-04-27 Thread Jonathan Hsieh (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-5876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh reassigned HBASE-5876:
-

Assignee: Jonathan Hsieh  (was: Uma Maheswara Rao G)

 TestImportExport has been failing against hadoop 0.23 profile
 -

 Key: HBASE-5876
 URL: https://issues.apache.org/jira/browse/HBASE-5876
 Project: HBase
  Issue Type: Bug
Reporter: Zhihong Yu
Assignee: Jonathan Hsieh

 TestImportExport has been failing against hadoop 0.23 profile

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Created] (HBASE-5889) Remove HRegionInterface

2012-04-27 Thread Jimmy Xiang (JIRA)
Jimmy Xiang created HBASE-5889:
--

 Summary: Remove HRegionInterface
 Key: HBASE-5889
 URL: https://issues.apache.org/jira/browse/HBASE-5889
 Project: HBase
  Issue Type: Improvement
  Components: client, ipc, regionserver
Affects Versions: 0.96.0
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0


As a step to move internals to PB, so as to avoid the conversion for 
performance reason, we should remove the HRegionInterface. 
Therefore region server only supports ClientProtocol and AdminProtocol.  Later 
on, HRegion can work with PB messages directly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5889) Remove HRegionInterface

2012-04-27 Thread Todd Lipcon (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263805#comment-13263805
 ] 

Todd Lipcon commented on HBASE-5889:


Can you explain which layer of conversion you're trying to remove? I think we 
still need our own interface for clients -- clients shouldn't directly speak 
protobuf.

Rather than doing a big change like this, can we get some good profiling on a 
benchmark and see where our time is going? We can probably make up the lost 
time in other ways.

 Remove HRegionInterface
 ---

 Key: HBASE-5889
 URL: https://issues.apache.org/jira/browse/HBASE-5889
 Project: HBase
  Issue Type: Improvement
  Components: client, ipc, regionserver
Affects Versions: 0.96.0
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0


 As a step to move internals to PB, so as to avoid the conversion for 
 performance reason, we should remove the HRegionInterface. 
 Therefore region server only supports ClientProtocol and AdminProtocol.  
 Later on, HRegion can work with PB messages directly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5785) Adding unit tests for protbuf utils introduced for HRegionInterface pb conversion

2012-04-27 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263812#comment-13263812
 ] 

Hadoop QA commented on HBASE-5785:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12524879/hbase-5785.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 2 new or modified tests.

+1 hadoop23.  The patch compiles against the hadoop 0.23.x profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 2 new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.replication.TestReplication

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1669//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1669//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1669//console

This message is automatically generated.

 Adding unit tests for protbuf utils introduced for HRegionInterface pb 
 conversion
 -

 Key: HBASE-5785
 URL: https://issues.apache.org/jira/browse/HBASE-5785
 Project: HBase
  Issue Type: Sub-task
  Components: ipc, master, migration, regionserver
Affects Versions: 0.96.0
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Critical
  Labels: noob
 Fix For: 0.96.0

 Attachments: hbase-5785.patch


 We need to add some unit tests for the probuf utilities to catch issues 
 earlier.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5889) Remove HRegionInterface

2012-04-27 Thread Jimmy Xiang (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263818#comment-13263818
 ] 

Jimmy Xiang commented on HBASE-5889:


The existing functions in the client side will not be touched for now.  
Probably we will add new methods to use the new types later.

I was thinking to remove the conversion in server side, for example, HRegion.  
But for now, I'd like to remove HRegionInterface at first so
as to clean up the references to it.  This is necessary since I found some test 
codes are still using the old interface. It will be
good for us to test the new interface.

 Remove HRegionInterface
 ---

 Key: HBASE-5889
 URL: https://issues.apache.org/jira/browse/HBASE-5889
 Project: HBase
  Issue Type: Improvement
  Components: client, ipc, regionserver
Affects Versions: 0.96.0
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0


 As a step to move internals to PB, so as to avoid the conversion for 
 performance reason, we should remove the HRegionInterface. 
 Therefore region server only supports ClientProtocol and AdminProtocol.  
 Later on, HRegion can work with PB messages directly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5860) splitlogmanager should not unnecessarily resubmit tasks when zk unavailable

2012-04-27 Thread Nicolas Spiegelberg (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263851#comment-13263851
 ] 

Nicolas Spiegelberg commented on HBASE-5860:


+1

 splitlogmanager should not unnecessarily resubmit tasks when zk unavailable
 ---

 Key: HBASE-5860
 URL: https://issues.apache.org/jira/browse/HBASE-5860
 Project: HBase
  Issue Type: Improvement
Reporter: Prakash Khemani
Assignee: Prakash Khemani
 Attachments: 
 0001-HBASE-5860-splitlogmanager-should-not-unnecessarily-.patch


 (Doesn't really impact the run time or correctness of log splitting)
 say the master has lost connection to zk. splitlogmanager's timeoutmanager 
 will realize that all the tasks that were submitted are still unassigned. It 
 will resubmit those tasks (i.e. create dummy znodes)
 splitlogmanager should realze that the tasks are unassigned but their znodes 
 have not been created.
 012-04-20 13:11:20,516 INFO org.apache.hadoop.hbase.master.SplitLogManager: 
 dead splitlog worker msgstore295.snc4.facebook.com,60020,1334948757026
 2012-04-20 13:11:20,517 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: 
 Scheduling batch of logs to split
 2012-04-20 13:11:20,517 INFO org.apache.hadoop.hbase.master.SplitLogManager: 
 started splitting logs in 
 [hdfs://msgstore215.snc4.facebook.com:9000/MSGSTORE215-SNC4-HBASE/.logs/msgstore295.snc4.facebook.com,60020,1334948757026-splitting]
 2012-04-20 13:11:20,565 INFO org.apache.zookeeper.ClientCnxn: Opening socket 
 connection to server msgstore235.snc4.facebook.com/10.30.222.186:2181
 2012-04-20 13:11:20,566 INFO org.apache.zookeeper.ClientCnxn: Socket 
 connection established to msgstore235.snc4.facebook.com/10.30.222.186:2181, 
 initiating session
 2012-04-20 13:11:20,575 INFO org.apache.hadoop.hbase.master.SplitLogManager: 
 total tasks = 4 unassigned = 4
 2012-04-20 13:11:20,576 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: 
 resubmitting unassigned task(s) after timeout
 2012-04-20 13:11:21,577 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: 
 resubmitting unassigned task(s) after timeout
 2012-04-20 13:11:21,683 INFO org.apache.zookeeper.ClientCnxn: Unable to read 
 additional data from server sessionid 0x36ccb0f8010002, likely server has 
 closed socket, closing socket connection and attempting reconnect
 2012-04-20 13:11:21,683 INFO org.apache.zookeeper.ClientCnxn: Unable to read 
 additional data from server sessionid 0x136ccb0f489, likely server has 
 closed socket, closing socket connection and attempting reconnect
 2012-04-20 13:11:21,786 WARN 
 org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback: create rc 
 =CONNECTIONLOSS for 
 /hbase/splitlog/hdfs%3A%2F%2Fmsgstore215.snc4.facebook.com%3A9000%2FMSGSTORE215-SNC4-HBASE%2F.logs%2Fmsgstore295.snc4.facebook.com%2C60020%2C1334948757026-splitting%2F10.30.251.186%253A60020.1334951586677
  retry=3
 2012-04-20 13:11:21,786 WARN 
 org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback: create rc 
 =CONNECTIONLOSS for 
 /hbase/splitlog/hdfs%3A%2F%2Fmsgstore215.snc4.facebook.com%3A9000%2FMSGSTORE215-SNC4-HBASE%2F.logs%2Fmsgstore295.snc4.facebook.com%2C60020%2C1334948757026-splitting%2F10.30.251.186%253A60020.1334951920332
  retry=3

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Assigned] (HBASE-5342) Grant/Revoke global permissions

2012-04-27 Thread Matteo Bertozzi (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Bertozzi reassigned HBASE-5342:
--

Assignee: Matteo Bertozzi  (was: Enis Soztutar)

 Grant/Revoke global permissions
 ---

 Key: HBASE-5342
 URL: https://issues.apache.org/jira/browse/HBASE-5342
 Project: HBase
  Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Matteo Bertozzi

 HBASE-3025 introduced simple ACLs based on coprocessors. It defines 
 global/table/cf/cq level permissions. However, there is no way to 
 grant/revoke global level permissions, other than the hbase.superuser conf 
 setting. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5860) splitlogmanager should not unnecessarily resubmit tasks when zk unavailable

2012-04-27 Thread Jimmy Xiang (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263871#comment-13263871
 ] 

Jimmy Xiang commented on HBASE-5860:


Looks good to me.

 splitlogmanager should not unnecessarily resubmit tasks when zk unavailable
 ---

 Key: HBASE-5860
 URL: https://issues.apache.org/jira/browse/HBASE-5860
 Project: HBase
  Issue Type: Improvement
Reporter: Prakash Khemani
Assignee: Prakash Khemani
 Attachments: 
 0001-HBASE-5860-splitlogmanager-should-not-unnecessarily-.patch


 (Doesn't really impact the run time or correctness of log splitting)
 say the master has lost connection to zk. splitlogmanager's timeoutmanager 
 will realize that all the tasks that were submitted are still unassigned. It 
 will resubmit those tasks (i.e. create dummy znodes)
 splitlogmanager should realze that the tasks are unassigned but their znodes 
 have not been created.
 012-04-20 13:11:20,516 INFO org.apache.hadoop.hbase.master.SplitLogManager: 
 dead splitlog worker msgstore295.snc4.facebook.com,60020,1334948757026
 2012-04-20 13:11:20,517 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: 
 Scheduling batch of logs to split
 2012-04-20 13:11:20,517 INFO org.apache.hadoop.hbase.master.SplitLogManager: 
 started splitting logs in 
 [hdfs://msgstore215.snc4.facebook.com:9000/MSGSTORE215-SNC4-HBASE/.logs/msgstore295.snc4.facebook.com,60020,1334948757026-splitting]
 2012-04-20 13:11:20,565 INFO org.apache.zookeeper.ClientCnxn: Opening socket 
 connection to server msgstore235.snc4.facebook.com/10.30.222.186:2181
 2012-04-20 13:11:20,566 INFO org.apache.zookeeper.ClientCnxn: Socket 
 connection established to msgstore235.snc4.facebook.com/10.30.222.186:2181, 
 initiating session
 2012-04-20 13:11:20,575 INFO org.apache.hadoop.hbase.master.SplitLogManager: 
 total tasks = 4 unassigned = 4
 2012-04-20 13:11:20,576 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: 
 resubmitting unassigned task(s) after timeout
 2012-04-20 13:11:21,577 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: 
 resubmitting unassigned task(s) after timeout
 2012-04-20 13:11:21,683 INFO org.apache.zookeeper.ClientCnxn: Unable to read 
 additional data from server sessionid 0x36ccb0f8010002, likely server has 
 closed socket, closing socket connection and attempting reconnect
 2012-04-20 13:11:21,683 INFO org.apache.zookeeper.ClientCnxn: Unable to read 
 additional data from server sessionid 0x136ccb0f489, likely server has 
 closed socket, closing socket connection and attempting reconnect
 2012-04-20 13:11:21,786 WARN 
 org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback: create rc 
 =CONNECTIONLOSS for 
 /hbase/splitlog/hdfs%3A%2F%2Fmsgstore215.snc4.facebook.com%3A9000%2FMSGSTORE215-SNC4-HBASE%2F.logs%2Fmsgstore295.snc4.facebook.com%2C60020%2C1334948757026-splitting%2F10.30.251.186%253A60020.1334951586677
  retry=3
 2012-04-20 13:11:21,786 WARN 
 org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback: create rc 
 =CONNECTIONLOSS for 
 /hbase/splitlog/hdfs%3A%2F%2Fmsgstore215.snc4.facebook.com%3A9000%2FMSGSTORE215-SNC4-HBASE%2F.logs%2Fmsgstore295.snc4.facebook.com%2C60020%2C1334948757026-splitting%2F10.30.251.186%253A60020.1334951920332
  retry=3

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5876) TestImportExport has been failing against hadoop 0.23 profile

2012-04-27 Thread Uma Maheswara Rao G (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263875#comment-13263875
 ] 

Uma Maheswara Rao G commented on HBASE-5876:


Thanks a lot, Jon for signing up this. Actually I thought to take a look on 
this weekend. Unfortunately I may not be able to look on this weekend as came 
out of station. Thanks for taking this:-)

 TestImportExport has been failing against hadoop 0.23 profile
 -

 Key: HBASE-5876
 URL: https://issues.apache.org/jira/browse/HBASE-5876
 Project: HBase
  Issue Type: Bug
Reporter: Zhihong Yu
Assignee: Jonathan Hsieh

 TestImportExport has been failing against hadoop 0.23 profile

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Created] (HBASE-5890) SplitLog Rescan BusyWaits upon Zk.CONNECTIONLOSS

2012-04-27 Thread Nicolas Spiegelberg (JIRA)
Nicolas Spiegelberg created HBASE-5890:
--

 Summary: SplitLog Rescan BusyWaits upon Zk.CONNECTIONLOSS
 Key: HBASE-5890
 URL: https://issues.apache.org/jira/browse/HBASE-5890
 Project: HBase
  Issue Type: Bug
Reporter: Nicolas Spiegelberg
Priority: Minor


We ran into a production issue yesterday where the SplitLogManager tried to 
create a Rescan node in ZK.  The createAsync() generated a 
KeeperException.CONNECTIONLOSS that was immedately sent to processResult(), 
createRescan node with --retry_count was called, and this created a CPU 
busywait that also clogged up the logs.  We should handle this better.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5890) SplitLog Rescan BusyWaits upon Zk.CONNECTIONLOSS

2012-04-27 Thread Nicolas Spiegelberg (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263880#comment-13263880
 ] 

Nicolas Spiegelberg commented on HBASE-5890:


The original idea is to have a timeout when we encounter this error.  Since we 
have a recoverable ZK, it seems okay to retry after connection loss; but we 
should have some sort of dampening so that this isn't a CPU  log hog.

 SplitLog Rescan BusyWaits upon Zk.CONNECTIONLOSS
 

 Key: HBASE-5890
 URL: https://issues.apache.org/jira/browse/HBASE-5890
 Project: HBase
  Issue Type: Bug
Reporter: Nicolas Spiegelberg
Priority: Minor

 We ran into a production issue yesterday where the SplitLogManager tried to 
 create a Rescan node in ZK.  The createAsync() generated a 
 KeeperException.CONNECTIONLOSS that was immedately sent to processResult(), 
 createRescan node with --retry_count was called, and this created a CPU 
 busywait that also clogged up the logs.  We should handle this better.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-5890) SplitLog Rescan BusyWaits upon Zk.CONNECTIONLOSS

2012-04-27 Thread Nicolas Spiegelberg (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-5890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nicolas Spiegelberg updated HBASE-5890:
---

Attachment: HBASE-5890.patch

patch should work for 89fb, 94, and trunk

 SplitLog Rescan BusyWaits upon Zk.CONNECTIONLOSS
 

 Key: HBASE-5890
 URL: https://issues.apache.org/jira/browse/HBASE-5890
 Project: HBase
  Issue Type: Bug
Reporter: Nicolas Spiegelberg
Priority: Minor
 Fix For: 0.94.0, 0.96.0, 0.89-fb

 Attachments: HBASE-5890.patch


 We ran into a production issue yesterday where the SplitLogManager tried to 
 create a Rescan node in ZK.  The createAsync() generated a 
 KeeperException.CONNECTIONLOSS that was immedately sent to processResult(), 
 createRescan node with --retry_count was called, and this created a CPU 
 busywait that also clogged up the logs.  We should handle this better.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-5890) SplitLog Rescan BusyWaits upon Zk.CONNECTIONLOSS

2012-04-27 Thread Nicolas Spiegelberg (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-5890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nicolas Spiegelberg updated HBASE-5890:
---

Fix Version/s: 0.89-fb
   0.96.0
   0.94.0
   Status: Patch Available  (was: Open)

 SplitLog Rescan BusyWaits upon Zk.CONNECTIONLOSS
 

 Key: HBASE-5890
 URL: https://issues.apache.org/jira/browse/HBASE-5890
 Project: HBase
  Issue Type: Bug
Reporter: Nicolas Spiegelberg
Priority: Minor
 Fix For: 0.94.0, 0.96.0, 0.89-fb

 Attachments: HBASE-5890.patch


 We ran into a production issue yesterday where the SplitLogManager tried to 
 create a Rescan node in ZK.  The createAsync() generated a 
 KeeperException.CONNECTIONLOSS that was immedately sent to processResult(), 
 createRescan node with --retry_count was called, and this created a CPU 
 busywait that also clogged up the logs.  We should handle this better.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5611) Replayed edits from regions that failed to open during recovery aren't removed from the global MemStore size

2012-04-27 Thread Ted Yu (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263904#comment-13263904
 ] 

Ted Yu commented on HBASE-5611:
---

I ran 0.92 test suite over 0.92 patch and didn't find regression.

Will integrate tomorrow if there is no objection.

 Replayed edits from regions that failed to open during recovery aren't 
 removed from the global MemStore size
 

 Key: HBASE-5611
 URL: https://issues.apache.org/jira/browse/HBASE-5611
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.6
Reporter: Jean-Daniel Cryans
Assignee: Jieshan Bean
Priority: Critical
 Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1

 Attachments: HBASE-5611-92.patch, HBASE-5611-94-minorchange.patch, 
 HBASE-5611-trunk-v2-minorchange.patch


 This bug is rather easy to get if the {{TimeoutMonitor}} is on, else I think 
 it's still possible to hit it if a region fails to open for more obscure 
 reasons like HDFS errors.
 Consider a region that just went through distributed splitting and that's now 
 being opened by a new RS. The first thing it does is to read the recovery 
 files and put the edits in the {{MemStores}}. If this process takes a long 
 time, the master will move that region away. At that point the edits are 
 still accounted for in the global {{MemStore}} size but they are dropped when 
 the {{HRegion}} gets cleaned up. It's completely invisible until the 
 {{MemStoreFlusher}} needs to force flush a region and that none of them have 
 edits:
 {noformat}
 2012-03-21 00:33:39,303 DEBUG 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush thread woke up 
 because memory above low water=5.9g
 2012-03-21 00:33:39,303 ERROR 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flusher failed 
 for entry null
 java.lang.IllegalStateException
 at 
 com.google.common.base.Preconditions.checkState(Preconditions.java:129)
 at 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:199)
 at 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:223)
 at java.lang.Thread.run(Thread.java:662)
 {noformat}
 The {{null}} here is a region. In my case I had so many edits in the 
 {{MemStore}} during recovery that I'm over the low barrier although in fact 
 I'm at 0. It happened yesterday and it still printing this out.
 To fix this we need to be able to decrease the global {{MemStore}} size when 
 the region can't open.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

2012-04-27 Thread jirapos...@reviews.apache.org (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263906#comment-13263906
 ] 

jirapos...@reviews.apache.org commented on HBASE-5547:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4633/#review7318
---



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
https://reviews.apache.org/r/4633/#comment16155

'for which' - 'the'



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
https://reviews.apache.org/r/4633/#comment16156

Insert 'whose' between 'tables' and 'HFiles'



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
https://reviews.apache.org/r/4633/#comment16157

The condition for calling addAndReWatchTable() is when table exists.



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
https://reviews.apache.org/r/4633/#comment16159

The method name is confusing.



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
https://reviews.apache.org/r/4633/#comment16158

When would this be true, considering that we have come through line 106 ?



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
https://reviews.apache.org/r/4633/#comment16160

'reup' - 'renew' ?



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
https://reviews.apache.org/r/4633/#comment16161

getTracker() and direct read of tracker appear on the same line ?



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
https://reviews.apache.org/r/4633/#comment16162

Can you use archivedTables.addAll() ?



src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
https://reviews.apache.org/r/4633/#comment16163

Remove the 's' following 'Region'


- Ted


On 2012-04-26 00:24:36, Jesse Yates wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4633/
bq.  ---
bq.  
bq.  (Updated 2012-04-26 00:24:36)
bq.  
bq.  
bq.  Review request for hbase, Michael Stack and Lars Hofhansl.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Essentially, whenever an hfile would be deleted, it is instead moved to 
the archive directory. In this impl, the archive directory is on a per table 
basis, but defaults to '.archive'. Removing hfiles occurs in three places - 
compaction, merge and catalog janitor. The former and two latter are distinctly 
different code paths, but but did pull out some similarities. The latter two 
end up calling the same method, so there should be a reasonable amount of 
overlap.
bq.  
bq.  Implementation wise: 
bq.  Updated the HMasterInterface to pass the calls onto the zookeeper.
bq.  Added a zk listener to handle updates from the master to the RS to 
backup.
bq.  Added a utility for removing files and finding archive directories
bq.  Added tests for the regionserver and catalogjanitor approaches.
bq.  Added creation of manager in regionserver.
bq.  
bq.  
bq.  This addresses bug HBASE-5547.
bq.  https://issues.apache.org/jira/browse/HBASE-5547
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0 
bq.src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java ee16e72 
bq.src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java d47b83a 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7858846 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
61a5988 
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerHFileTableArchiveTracker.java
 PRE-CREATION 
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 
6884d53 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 
ea12da4 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/Store.java bf1618e 
bq.src/main/java/org/apache/hadoop/hbase/util/HFileArchiveCleanup.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 
4fc105f 
bq.src/main/resources/hbase-default.xml f54b345 
bq.

[jira] [Commented] (HBASE-5890) SplitLog Rescan BusyWaits upon Zk.CONNECTIONLOSS

2012-04-27 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263919#comment-13263919
 ] 

Hadoop QA commented on HBASE-5890:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12524903/HBASE-5890.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

+1 hadoop23.  The patch compiles against the hadoop 0.23.x profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 2 new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

+1 core tests.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1670//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1670//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1670//console

This message is automatically generated.

 SplitLog Rescan BusyWaits upon Zk.CONNECTIONLOSS
 

 Key: HBASE-5890
 URL: https://issues.apache.org/jira/browse/HBASE-5890
 Project: HBase
  Issue Type: Bug
Reporter: Nicolas Spiegelberg
Priority: Minor
 Fix For: 0.94.0, 0.96.0, 0.89-fb

 Attachments: HBASE-5890.patch


 We ran into a production issue yesterday where the SplitLogManager tried to 
 create a Rescan node in ZK.  The createAsync() generated a 
 KeeperException.CONNECTIONLOSS that was immedately sent to processResult(), 
 createRescan node with --retry_count was called, and this created a CPU 
 busywait that also clogged up the logs.  We should handle this better.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5890) SplitLog Rescan BusyWaits upon Zk.CONNECTIONLOSS

2012-04-27 Thread Ted Yu (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263930#comment-13263930
 ] 

Ted Yu commented on HBASE-5890:
---

Patch looks good.
In the catch clause, the following statement should be added:
{code}
  Thread.currentThread().interrupt();
{code}

 SplitLog Rescan BusyWaits upon Zk.CONNECTIONLOSS
 

 Key: HBASE-5890
 URL: https://issues.apache.org/jira/browse/HBASE-5890
 Project: HBase
  Issue Type: Bug
Reporter: Nicolas Spiegelberg
Priority: Minor
 Fix For: 0.94.0, 0.96.0, 0.89-fb

 Attachments: HBASE-5890.patch


 We ran into a production issue yesterday where the SplitLogManager tried to 
 create a Rescan node in ZK.  The createAsync() generated a 
 KeeperException.CONNECTIONLOSS that was immedately sent to processResult(), 
 createRescan node with --retry_count was called, and this created a CPU 
 busywait that also clogged up the logs.  We should handle this better.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-5886) Add new metric for possible data loss due to puts without WAL

2012-04-27 Thread Matteo Bertozzi (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Bertozzi updated HBASE-5886:
---

Attachment: HBASE-5886-v2.patch

 Add new metric for possible data loss due to puts without WAL 
 --

 Key: HBASE-5886
 URL: https://issues.apache.org/jira/browse/HBASE-5886
 Project: HBase
  Issue Type: New Feature
  Components: metrics, regionserver
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
  Labels: metrics
 Attachments: HBASE-5886-v0.patch, HBASE-5886-v1.patch, 
 HBASE-5886-v2.patch


 Add a metrics to keep track of puts without WAL and possible data loss size.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5886) Add new metric for possible data loss due to puts without WAL

2012-04-27 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263939#comment-13263939
 ] 

Hadoop QA commented on HBASE-5886:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12524910/HBASE-5886-v2.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

+1 hadoop23.  The patch compiles against the hadoop 0.23.x profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 1 new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.io.TestHeapSize

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1671//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1671//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1671//console

This message is automatically generated.

 Add new metric for possible data loss due to puts without WAL 
 --

 Key: HBASE-5886
 URL: https://issues.apache.org/jira/browse/HBASE-5886
 Project: HBase
  Issue Type: New Feature
  Components: metrics, regionserver
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
  Labels: metrics
 Attachments: HBASE-5886-v0.patch, HBASE-5886-v1.patch, 
 HBASE-5886-v2.patch


 Add a metrics to keep track of puts without WAL and possible data loss size.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Created] (HBASE-5891) Change Compression Based on Type of Compaction

2012-04-27 Thread Nicolas Spiegelberg (JIRA)
Nicolas Spiegelberg created HBASE-5891:
--

 Summary: Change Compression Based on Type of Compaction
 Key: HBASE-5891
 URL: https://issues.apache.org/jira/browse/HBASE-5891
 Project: HBase
  Issue Type: New Feature
Reporter: Nicolas Spiegelberg
Priority: Minor


We currently use LZO on our production systems because the on-demand 
decompression speed of GZ is too slow.  That said, many of our major-compacted 
StoreFiles are infrequently read because of lazy seek optimizations, but they 
occupy the majority of our disk space.  One idea is to change the type of 
compression depending upon compaction characteristics (input size or major 
compaction flag).  This would allow us to have our largest and least-read files 
be GZ compressed and save space.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-5886) Add new metric for possible data loss due to puts without WAL

2012-04-27 Thread Matteo Bertozzi (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Bertozzi updated HBASE-5886:
---

Attachment: HBASE-5886-v2.patch

 Add new metric for possible data loss due to puts without WAL 
 --

 Key: HBASE-5886
 URL: https://issues.apache.org/jira/browse/HBASE-5886
 Project: HBase
  Issue Type: New Feature
  Components: metrics, regionserver
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
  Labels: metrics
 Attachments: HBASE-5886-v0.patch, HBASE-5886-v1.patch, 
 HBASE-5886-v2.patch


 Add a metrics to keep track of puts without WAL and possible data loss size.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-5886) Add new metric for possible data loss due to puts without WAL

2012-04-27 Thread Matteo Bertozzi (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Bertozzi updated HBASE-5886:
---

Attachment: (was: HBASE-5886-v2.patch)

 Add new metric for possible data loss due to puts without WAL 
 --

 Key: HBASE-5886
 URL: https://issues.apache.org/jira/browse/HBASE-5886
 Project: HBase
  Issue Type: New Feature
  Components: metrics, regionserver
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
  Labels: metrics
 Attachments: HBASE-5886-v0.patch, HBASE-5886-v1.patch, 
 HBASE-5886-v2.patch


 Add a metrics to keep track of puts without WAL and possible data loss size.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-5826) Improve sync of HLog edits

2012-04-27 Thread Ted Yu (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-5826:
--

Attachment: (was: 5826.txt)

 Improve sync of HLog edits
 --

 Key: HBASE-5826
 URL: https://issues.apache.org/jira/browse/HBASE-5826
 Project: HBase
  Issue Type: Improvement
Reporter: Zhihong Yu
 Attachments: 5826.txt


 HBASE-5782 solved the correctness issue for the sync of HLog edits.
 Todd provided a patch that would achieve higher throughput.
 This JIRA is a continuation of Todd's work submitted there.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-5826) Improve sync of HLog edits

2012-04-27 Thread Ted Yu (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-5826:
--

Attachment: 5826.txt

 Improve sync of HLog edits
 --

 Key: HBASE-5826
 URL: https://issues.apache.org/jira/browse/HBASE-5826
 Project: HBase
  Issue Type: Improvement
Reporter: Zhihong Yu
 Attachments: 5826.txt


 HBASE-5782 solved the correctness issue for the sync of HLog edits.
 Todd provided a patch that would achieve higher throughput.
 This JIRA is a continuation of Todd's work submitted there.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5712) Parallelize load of .regioninfo files in diagnostic/repair portion of hbck.

2012-04-27 Thread jirapos...@reviews.apache.org (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263972#comment-13263972
 ] 

jirapos...@reviews.apache.org commented on HBASE-5712:
--



bq.  On 2012-04-26 02:25:57, Jimmy Xiang wrote:
bq.   src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 644
bq.   https://reviews.apache.org/r/4883/diff/1/?file=104442#file104442line644
bq.  
bq.   This is ok. However, I think it is better to use a timed wait in 
case it hangs.

There is another section of code that uses the same logic and has been around 
for a while.  I'll file a follow on issue to convert this homebrew fork join 
stuff in to a Futures.


- jmhsieh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4883/#review7248
---


On 2012-04-26 01:42:01, jmhsieh wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4883/
bq.  ---
bq.  
bq.  (Updated 2012-04-26 01:42:01)
bq.  
bq.  
bq.  Review request for hbase, Ted Yu and Jimmy Xiang.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  * Parallelized load of .regioninfo files
bq.  * changed TreeMap to SortedMap in method signatures
bq.  * renamed a test's name.
bq.  
bq.  
bq.  This addresses bug HBASE-5712.
bq.  https://issues.apache.org/jira/browse/HBASE-5712
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 66156c2 
bq.src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java 6b64f10 
bq.  
bq.  Diff: https://reviews.apache.org/r/4883/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Ran patch 10x on trunk, passes.  Ran 1x on 0.92 and 0.94.
bq.  
bq.  Ther 0.90 version that is nearly identical except for ignoring changes 
near lines HBaseFsck lines 671-680.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  jmhsieh
bq.  
bq.



 Parallelize load of .regioninfo files in diagnostic/repair portion of hbck.
 ---

 Key: HBASE-5712
 URL: https://issues.apache.org/jira/browse/HBASE-5712
 Project: HBase
  Issue Type: Sub-task
  Components: hbck
Affects Versions: 0.90.7, 0.92.2, 0.94.0, 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Attachments: hbase-5712-90.patch, hbase-5712.patch


 On heavily loaded hdfs's some dfs nodes may not respond quickly and backs off 
 for 60s before attempting to read data from another datanode.  Portions of 
 the information gathered from hdfs (.regioninfo files) are loaded serially.  
 With HBase with clusters with 100's, or 1000's, or 1's regions 
 encountering these 60s delay blocks progress and can be very painful.  
 There is already some parallelization of portions of the hdfs information 
 load operations and the goal here is move the reading of .regioninfos into 
 the parallelized sections..

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Created] (HBASE-5892) [hbck] Refactor parallel WorkItem* to Futures.

2012-04-27 Thread Jonathan Hsieh (JIRA)
Jonathan Hsieh created HBASE-5892:
-

 Summary: [hbck] Refactor parallel WorkItem* to Futures.
 Key: HBASE-5892
 URL: https://issues.apache.org/jira/browse/HBASE-5892
 Project: HBase
  Issue Type: Improvement
Reporter: Jonathan Hsieh


This would convert WorkItem* logic (with low level notifies, and rough 
exception handling)  into a more canonical Futures pattern.

Currently there are two instances of this pattern (for loading hdfs dirs, for 
contacting regionservers for assignments, and soon -- for loading hdfs 
.regioninfo files).



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5712) Parallelize load of .regioninfo files in diagnostic/repair portion of hbck.

2012-04-27 Thread jirapos...@reviews.apache.org (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263992#comment-13263992
 ] 

jirapos...@reviews.apache.org commented on HBASE-5712:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4883/#review7325
---



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4883/#comment16179

What does 'Adopting' mean ?
The grammar doesn't seem to be right.


- Ted


On 2012-04-26 01:42:01, jmhsieh wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4883/
bq.  ---
bq.  
bq.  (Updated 2012-04-26 01:42:01)
bq.  
bq.  
bq.  Review request for hbase, Ted Yu and Jimmy Xiang.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  * Parallelized load of .regioninfo files
bq.  * changed TreeMap to SortedMap in method signatures
bq.  * renamed a test's name.
bq.  
bq.  
bq.  This addresses bug HBASE-5712.
bq.  https://issues.apache.org/jira/browse/HBASE-5712
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 66156c2 
bq.src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java 6b64f10 
bq.  
bq.  Diff: https://reviews.apache.org/r/4883/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Ran patch 10x on trunk, passes.  Ran 1x on 0.92 and 0.94.
bq.  
bq.  Ther 0.90 version that is nearly identical except for ignoring changes 
near lines HBaseFsck lines 671-680.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  jmhsieh
bq.  
bq.



 Parallelize load of .regioninfo files in diagnostic/repair portion of hbck.
 ---

 Key: HBASE-5712
 URL: https://issues.apache.org/jira/browse/HBASE-5712
 Project: HBase
  Issue Type: Sub-task
  Components: hbck
Affects Versions: 0.90.7, 0.92.2, 0.94.0, 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Attachments: hbase-5712-90.patch, hbase-5712.patch


 On heavily loaded hdfs's some dfs nodes may not respond quickly and backs off 
 for 60s before attempting to read data from another datanode.  Portions of 
 the information gathered from hdfs (.regioninfo files) are loaded serially.  
 With HBase with clusters with 100's, or 1000's, or 1's regions 
 encountering these 60s delay blocks progress and can be very painful.  
 There is already some parallelization of portions of the hdfs information 
 load operations and the goal here is move the reading of .regioninfos into 
 the parallelized sections..

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Created] (HBASE-5893) Allow spaces in coprocessor conf (aka trim() className)

2012-04-27 Thread Matteo Bertozzi (JIRA)
Matteo Bertozzi created HBASE-5893:
--

 Summary: Allow spaces in coprocessor conf (aka trim() className)
 Key: HBASE-5893
 URL: https://issues.apache.org/jira/browse/HBASE-5893
 Project: HBase
  Issue Type: Bug
  Components: coprocessors
Affects Versions: 0.92.1, 0.94.0, 0.96.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor


This is annoying especially for coprocessors where you've long class name.
but maybe is a bug of Configuration.getStrings() that doesn't trim each string.

When you've comma separated values like in the coprocessors case, you've to 
pack together your values without spaces (v1,v2,v3,...) otherwise the 
coprocessor is not loaded because the class name with spaces is not found.
{code}
property
   namehbase.coprocessor.master.classes/name
   value
org.apache.hadoop.hbase.security.token.TokenProvider,
org.apache.hadoop.hbase.security.access.AccessController
   /value
/property
{code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-5893) Allow spaces in coprocessor conf (aka trim() className)

2012-04-27 Thread Matteo Bertozzi (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-5893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Bertozzi updated HBASE-5893:
---

Attachment: HBASE-5893-v0.patch

 Allow spaces in coprocessor conf (aka trim() className)
 ---

 Key: HBASE-5893
 URL: https://issues.apache.org/jira/browse/HBASE-5893
 Project: HBase
  Issue Type: Bug
  Components: coprocessors
Affects Versions: 0.92.1, 0.94.0, 0.96.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
  Labels: conf, coprocessors
 Attachments: HBASE-5893-v0.patch


 This is annoying especially for coprocessors where you've long class name.
 but maybe is a bug of Configuration.getStrings() that doesn't trim each 
 string.
 When you've comma separated values like in the coprocessors case, you've to 
 pack together your values without spaces (v1,v2,v3,...) otherwise the 
 coprocessor is not loaded because the class name with spaces is not found.
 {code}
 property
namehbase.coprocessor.master.classes/name
value
 org.apache.hadoop.hbase.security.token.TokenProvider,
 org.apache.hadoop.hbase.security.access.AccessController
/value
 /property
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-5893) Allow spaces in coprocessor conf (aka trim() className)

2012-04-27 Thread Matteo Bertozzi (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-5893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Bertozzi updated HBASE-5893:
---

Status: Patch Available  (was: Open)

 Allow spaces in coprocessor conf (aka trim() className)
 ---

 Key: HBASE-5893
 URL: https://issues.apache.org/jira/browse/HBASE-5893
 Project: HBase
  Issue Type: Bug
  Components: coprocessors
Affects Versions: 0.92.1, 0.94.0, 0.96.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
  Labels: conf, coprocessors
 Attachments: HBASE-5893-v0.patch


 This is annoying especially for coprocessors where you've long class name.
 but maybe is a bug of Configuration.getStrings() that doesn't trim each 
 string.
 When you've comma separated values like in the coprocessors case, you've to 
 pack together your values without spaces (v1,v2,v3,...) otherwise the 
 coprocessor is not loaded because the class name with spaces is not found.
 {code}
 property
namehbase.coprocessor.master.classes/name
value
 org.apache.hadoop.hbase.security.token.TokenProvider,
 org.apache.hadoop.hbase.security.access.AccessController
/value
 /property
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5712) Parallelize load of .regioninfo files in diagnostic/repair portion of hbck.

2012-04-27 Thread jirapos...@reviews.apache.org (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264029#comment-13264029
 ] 

jirapos...@reviews.apache.org commented on HBASE-5712:
--



bq.  On 2012-04-27 21:33:19, Ted Yu wrote:
bq.   src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 418
bq.   https://reviews.apache.org/r/4883/diff/1/?file=104442#file104442line418
bq.  
bq.   What does 'Adopting' mean ?
bq.   The grammar doesn't seem to be right.

Adopting is a term that I use for taking an orphaned hdfs region dir (a 
region in hdfs missing .regioninfo file, and likely not in META/deployed) and 
adding it to meta, adding a .regioninfo file, and potentially deploying the 
region.  

How about I change it to this?
Attempt to adopt orphan region skipped because no files present in  ... 


- jmhsieh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4883/#review7325
---


On 2012-04-26 01:42:01, jmhsieh wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4883/
bq.  ---
bq.  
bq.  (Updated 2012-04-26 01:42:01)
bq.  
bq.  
bq.  Review request for hbase, Ted Yu and Jimmy Xiang.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  * Parallelized load of .regioninfo files
bq.  * changed TreeMap to SortedMap in method signatures
bq.  * renamed a test's name.
bq.  
bq.  
bq.  This addresses bug HBASE-5712.
bq.  https://issues.apache.org/jira/browse/HBASE-5712
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 66156c2 
bq.src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java 6b64f10 
bq.  
bq.  Diff: https://reviews.apache.org/r/4883/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Ran patch 10x on trunk, passes.  Ran 1x on 0.92 and 0.94.
bq.  
bq.  Ther 0.90 version that is nearly identical except for ignoring changes 
near lines HBaseFsck lines 671-680.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  jmhsieh
bq.  
bq.



 Parallelize load of .regioninfo files in diagnostic/repair portion of hbck.
 ---

 Key: HBASE-5712
 URL: https://issues.apache.org/jira/browse/HBASE-5712
 Project: HBase
  Issue Type: Sub-task
  Components: hbck
Affects Versions: 0.90.7, 0.92.2, 0.94.0, 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Attachments: hbase-5712-90.patch, hbase-5712.patch


 On heavily loaded hdfs's some dfs nodes may not respond quickly and backs off 
 for 60s before attempting to read data from another datanode.  Portions of 
 the information gathered from hdfs (.regioninfo files) are loaded serially.  
 With HBase with clusters with 100's, or 1000's, or 1's regions 
 encountering these 60s delay blocks progress and can be very painful.  
 There is already some parallelization of portions of the hdfs information 
 load operations and the goal here is move the reading of .regioninfos into 
 the parallelized sections..

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-5611) Replayed edits from regions that failed to open during recovery aren't removed from the global MemStore size

2012-04-27 Thread Lars Hofhansl (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-5611:
-

Fix Version/s: (was: 0.94.1)
   0.94.0

Still waiting for a 0.94.0 blocker. +1 on 0.94 commit as well.

 Replayed edits from regions that failed to open during recovery aren't 
 removed from the global MemStore size
 

 Key: HBASE-5611
 URL: https://issues.apache.org/jira/browse/HBASE-5611
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.6
Reporter: Jean-Daniel Cryans
Assignee: Jieshan Bean
Priority: Critical
 Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0

 Attachments: HBASE-5611-92.patch, HBASE-5611-94-minorchange.patch, 
 HBASE-5611-trunk-v2-minorchange.patch


 This bug is rather easy to get if the {{TimeoutMonitor}} is on, else I think 
 it's still possible to hit it if a region fails to open for more obscure 
 reasons like HDFS errors.
 Consider a region that just went through distributed splitting and that's now 
 being opened by a new RS. The first thing it does is to read the recovery 
 files and put the edits in the {{MemStores}}. If this process takes a long 
 time, the master will move that region away. At that point the edits are 
 still accounted for in the global {{MemStore}} size but they are dropped when 
 the {{HRegion}} gets cleaned up. It's completely invisible until the 
 {{MemStoreFlusher}} needs to force flush a region and that none of them have 
 edits:
 {noformat}
 2012-03-21 00:33:39,303 DEBUG 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush thread woke up 
 because memory above low water=5.9g
 2012-03-21 00:33:39,303 ERROR 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flusher failed 
 for entry null
 java.lang.IllegalStateException
 at 
 com.google.common.base.Preconditions.checkState(Preconditions.java:129)
 at 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:199)
 at 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:223)
 at java.lang.Thread.run(Thread.java:662)
 {noformat}
 The {{null}} here is a region. In my case I had so many edits in the 
 {{MemStore}} during recovery that I'm over the low barrier although in fact 
 I'm at 0. It happened yesterday and it still printing this out.
 To fix this we need to be able to decrease the global {{MemStore}} size when 
 the region can't open.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5826) Improve sync of HLog edits

2012-04-27 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264036#comment-13264036
 ] 

Hadoop QA commented on HBASE-5826:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12524921/5826.txt
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 3 new or modified tests.

+1 hadoop23.  The patch compiles against the hadoop 0.23.x profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 2 new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   
org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1673//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1673//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1673//console

This message is automatically generated.

 Improve sync of HLog edits
 --

 Key: HBASE-5826
 URL: https://issues.apache.org/jira/browse/HBASE-5826
 Project: HBase
  Issue Type: Improvement
Reporter: Zhihong Yu
 Attachments: 5826.txt


 HBASE-5782 solved the correctness issue for the sync of HLog edits.
 Todd provided a patch that would achieve higher throughput.
 This JIRA is a continuation of Todd's work submitted there.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5890) SplitLog Rescan BusyWaits upon Zk.CONNECTIONLOSS

2012-04-27 Thread Prakash Khemani (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264046#comment-13264046
 ] 

Prakash Khemani commented on HBASE-5890:


Most likely, it isn't a good idea to sleep in the zookeeper callback thread. 
(isn't the zk client single threaded?)

Can these be queued in a DelayedQueue(socket-timeout) and retried from 
SplitLogManager.TimeoutMonitor.chore()

 SplitLog Rescan BusyWaits upon Zk.CONNECTIONLOSS
 

 Key: HBASE-5890
 URL: https://issues.apache.org/jira/browse/HBASE-5890
 Project: HBase
  Issue Type: Bug
Reporter: Nicolas Spiegelberg
Priority: Minor
 Fix For: 0.94.0, 0.96.0, 0.89-fb

 Attachments: HBASE-5890.patch


 We ran into a production issue yesterday where the SplitLogManager tried to 
 create a Rescan node in ZK.  The createAsync() generated a 
 KeeperException.CONNECTIONLOSS that was immedately sent to processResult(), 
 createRescan node with --retry_count was called, and this created a CPU 
 busywait that also clogged up the logs.  We should handle this better.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5712) Parallelize load of .regioninfo files in diagnostic/repair portion of hbck.

2012-04-27 Thread Ted Yu (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264048#comment-13264048
 ] 

Ted Yu commented on HBASE-5712:
---

The new sentence is better.
+1.

 Parallelize load of .regioninfo files in diagnostic/repair portion of hbck.
 ---

 Key: HBASE-5712
 URL: https://issues.apache.org/jira/browse/HBASE-5712
 Project: HBase
  Issue Type: Sub-task
  Components: hbck
Affects Versions: 0.90.7, 0.92.2, 0.94.0, 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Attachments: hbase-5712-90.patch, hbase-5712.patch


 On heavily loaded hdfs's some dfs nodes may not respond quickly and backs off 
 for 60s before attempting to read data from another datanode.  Portions of 
 the information gathered from hdfs (.regioninfo files) are loaded serially.  
 With HBase with clusters with 100's, or 1000's, or 1's regions 
 encountering these 60s delay blocks progress and can be very painful.  
 There is already some parallelization of portions of the hdfs information 
 load operations and the goal here is move the reading of .regioninfos into 
 the parallelized sections..

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Assigned] (HBASE-5885) Invalid HFile block magic on Local file System

2012-04-27 Thread Elliott Clark (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark reassigned HBASE-5885:


Assignee: Elliott Clark

 Invalid HFile block magic on Local file System
 --

 Key: HBASE-5885
 URL: https://issues.apache.org/jira/browse/HBASE-5885
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.0, 0.96.0
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Blocker
 Fix For: 0.94.0

 Attachments: HBASE-5885-94-0.patch


 ERROR: java.lang.RuntimeException: 
 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after 
 attempts=7, exceptions:
 Thu Apr 26 11:19:18 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: Could not iterate StoreFileScanner[HFileScanner for 
 reader 
 reader=file:/tmp/hbase-eclark/hbase/TestTable/e2d1c846363c75262cbfd85ea278b342/info/bae2681d63734066957b58fe791a0268,
  compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], 
 firstKey=01/info:data/1335463981520/Put, 
 lastKey=0002588100/info:data/1335463902296/Put, avgKeyLen=30, 
 avgValueLen=1000, entries=1215085, length=1264354417, 
 cur=000248/info:data/1335463994457/Put/vlen=1000/ts=0]
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:135)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95)
   at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:368)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.io.IOException: Invalid HFile block magic: 
 \xEC\xD5\x9D\xB4\xC2bfo
   at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153)
   at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock.init(HFileBlock.java:254)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1779)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:327)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:555)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:651)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130)
   ... 12 more
 Thu Apr 26 11:19:19 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: java.lang.IllegalArgumentException
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1132)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1121)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2420)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.lang.IllegalArgumentException
   at java.nio.Buffer.position(Buffer.java:216)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:630)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130)
   at 
 

[jira] [Updated] (HBASE-5885) Invalid HFile block magic on Local file System

2012-04-27 Thread Elliott Clark (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-5885:
-

Attachment: HBASE-5885-94-0.patch

94 patch.

 Invalid HFile block magic on Local file System
 --

 Key: HBASE-5885
 URL: https://issues.apache.org/jira/browse/HBASE-5885
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.0, 0.96.0
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Blocker
 Fix For: 0.94.0

 Attachments: HBASE-5885-94-0.patch


 ERROR: java.lang.RuntimeException: 
 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after 
 attempts=7, exceptions:
 Thu Apr 26 11:19:18 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: Could not iterate StoreFileScanner[HFileScanner for 
 reader 
 reader=file:/tmp/hbase-eclark/hbase/TestTable/e2d1c846363c75262cbfd85ea278b342/info/bae2681d63734066957b58fe791a0268,
  compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], 
 firstKey=01/info:data/1335463981520/Put, 
 lastKey=0002588100/info:data/1335463902296/Put, avgKeyLen=30, 
 avgValueLen=1000, entries=1215085, length=1264354417, 
 cur=000248/info:data/1335463994457/Put/vlen=1000/ts=0]
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:135)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95)
   at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:368)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.io.IOException: Invalid HFile block magic: 
 \xEC\xD5\x9D\xB4\xC2bfo
   at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153)
   at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock.init(HFileBlock.java:254)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1779)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:327)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:555)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:651)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130)
   ... 12 more
 Thu Apr 26 11:19:19 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: java.lang.IllegalArgumentException
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1132)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1121)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2420)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.lang.IllegalArgumentException
   at java.nio.Buffer.position(Buffer.java:216)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:630)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130)
   at 
 

[jira] [Updated] (HBASE-5885) Invalid HFile block magic on Local file System

2012-04-27 Thread Elliott Clark (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-5885:
-

Attachment: HBASE-5885-trunk-0.patch

Trunk patch.
org.apache.hadoop.LocalFileSystem's seek does not work as expected when 
setVerifyChecksum is set to false.

This patch means that on the local machine all reads will use the underlying 
checksum.

 Invalid HFile block magic on Local file System
 --

 Key: HBASE-5885
 URL: https://issues.apache.org/jira/browse/HBASE-5885
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.0, 0.96.0
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Blocker
 Fix For: 0.94.0

 Attachments: HBASE-5885-94-0.patch, HBASE-5885-trunk-0.patch


 ERROR: java.lang.RuntimeException: 
 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after 
 attempts=7, exceptions:
 Thu Apr 26 11:19:18 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: Could not iterate StoreFileScanner[HFileScanner for 
 reader 
 reader=file:/tmp/hbase-eclark/hbase/TestTable/e2d1c846363c75262cbfd85ea278b342/info/bae2681d63734066957b58fe791a0268,
  compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], 
 firstKey=01/info:data/1335463981520/Put, 
 lastKey=0002588100/info:data/1335463902296/Put, avgKeyLen=30, 
 avgValueLen=1000, entries=1215085, length=1264354417, 
 cur=000248/info:data/1335463994457/Put/vlen=1000/ts=0]
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:135)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95)
   at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:368)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.io.IOException: Invalid HFile block magic: 
 \xEC\xD5\x9D\xB4\xC2bfo
   at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153)
   at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock.init(HFileBlock.java:254)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1779)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:327)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:555)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:651)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130)
   ... 12 more
 Thu Apr 26 11:19:19 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: java.lang.IllegalArgumentException
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1132)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1121)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2420)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.lang.IllegalArgumentException
   at java.nio.Buffer.position(Buffer.java:216)
   

[jira] [Updated] (HBASE-5885) Invalid HFile block magic on Local file System

2012-04-27 Thread Elliott Clark (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-5885:
-

Status: Patch Available  (was: Open)

 Invalid HFile block magic on Local file System
 --

 Key: HBASE-5885
 URL: https://issues.apache.org/jira/browse/HBASE-5885
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.0, 0.96.0
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Blocker
 Fix For: 0.94.0

 Attachments: HBASE-5885-94-0.patch, HBASE-5885-trunk-0.patch


 ERROR: java.lang.RuntimeException: 
 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after 
 attempts=7, exceptions:
 Thu Apr 26 11:19:18 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: Could not iterate StoreFileScanner[HFileScanner for 
 reader 
 reader=file:/tmp/hbase-eclark/hbase/TestTable/e2d1c846363c75262cbfd85ea278b342/info/bae2681d63734066957b58fe791a0268,
  compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], 
 firstKey=01/info:data/1335463981520/Put, 
 lastKey=0002588100/info:data/1335463902296/Put, avgKeyLen=30, 
 avgValueLen=1000, entries=1215085, length=1264354417, 
 cur=000248/info:data/1335463994457/Put/vlen=1000/ts=0]
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:135)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95)
   at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:368)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.io.IOException: Invalid HFile block magic: 
 \xEC\xD5\x9D\xB4\xC2bfo
   at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153)
   at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock.init(HFileBlock.java:254)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1779)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:327)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:555)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:651)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130)
   ... 12 more
 Thu Apr 26 11:19:19 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: java.lang.IllegalArgumentException
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1132)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1121)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2420)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.lang.IllegalArgumentException
   at java.nio.Buffer.position(Buffer.java:216)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:630)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130)
   at 
 

[jira] [Commented] (HBASE-5893) Allow spaces in coprocessor conf (aka trim() className)

2012-04-27 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264058#comment-13264058
 ] 

Hadoop QA commented on HBASE-5893:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12524934/HBASE-5893-v0.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

+1 hadoop23.  The patch compiles against the hadoop 0.23.x profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 2 new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

+1 core tests.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1674//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1674//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1674//console

This message is automatically generated.

 Allow spaces in coprocessor conf (aka trim() className)
 ---

 Key: HBASE-5893
 URL: https://issues.apache.org/jira/browse/HBASE-5893
 Project: HBase
  Issue Type: Bug
  Components: coprocessors
Affects Versions: 0.92.1, 0.94.0, 0.96.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
  Labels: conf, coprocessors
 Attachments: HBASE-5893-v0.patch


 This is annoying especially for coprocessors where you've long class name.
 but maybe is a bug of Configuration.getStrings() that doesn't trim each 
 string.
 When you've comma separated values like in the coprocessors case, you've to 
 pack together your values without spaces (v1,v2,v3,...) otherwise the 
 coprocessor is not loaded because the class name with spaces is not found.
 {code}
 property
namehbase.coprocessor.master.classes/name
value
 org.apache.hadoop.hbase.security.token.TokenProvider,
 org.apache.hadoop.hbase.security.access.AccessController
/value
 /property
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5885) Invalid HFile block magic on Local file System

2012-04-27 Thread Lars Hofhansl (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264103#comment-13264103
 ] 

Lars Hofhansl commented on HBASE-5885:
--

Do you envision this as the final fix, or as a tempory solutions for now?
Might still be nice to know what exactly causes this for the local fs.

 Invalid HFile block magic on Local file System
 --

 Key: HBASE-5885
 URL: https://issues.apache.org/jira/browse/HBASE-5885
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.0, 0.96.0
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Blocker
 Fix For: 0.94.0

 Attachments: HBASE-5885-94-0.patch, HBASE-5885-trunk-0.patch


 ERROR: java.lang.RuntimeException: 
 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after 
 attempts=7, exceptions:
 Thu Apr 26 11:19:18 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: Could not iterate StoreFileScanner[HFileScanner for 
 reader 
 reader=file:/tmp/hbase-eclark/hbase/TestTable/e2d1c846363c75262cbfd85ea278b342/info/bae2681d63734066957b58fe791a0268,
  compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], 
 firstKey=01/info:data/1335463981520/Put, 
 lastKey=0002588100/info:data/1335463902296/Put, avgKeyLen=30, 
 avgValueLen=1000, entries=1215085, length=1264354417, 
 cur=000248/info:data/1335463994457/Put/vlen=1000/ts=0]
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:135)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95)
   at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:368)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.io.IOException: Invalid HFile block magic: 
 \xEC\xD5\x9D\xB4\xC2bfo
   at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153)
   at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock.init(HFileBlock.java:254)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1779)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:327)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:555)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:651)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130)
   ... 12 more
 Thu Apr 26 11:19:19 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: java.lang.IllegalArgumentException
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1132)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1121)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2420)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.lang.IllegalArgumentException
   at java.nio.Buffer.position(Buffer.java:216)
   at 
 

[jira] [Commented] (HBASE-5712) Parallelize load of .regioninfo files in diagnostic/repair portion of hbck.

2012-04-27 Thread jirapos...@reviews.apache.org (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264107#comment-13264107
 ] 

jirapos...@reviews.apache.org commented on HBASE-5712:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4883/#review7337
---

Ship it!


+1  Minor comments below if interested


src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4883/#comment16190

This'll work but why not ConcurrentSkipListMap?



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4883/#comment16191

+1 on suggested change



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4883/#comment16192

Is this flag needed?  Why not just check thread is alive?  I see we can 
return with an error.  What happens if the return on 2816 happens?  Will the 
wait at #643 above be for ever?


- Michael


On 2012-04-26 01:42:01, jmhsieh wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4883/
bq.  ---
bq.  
bq.  (Updated 2012-04-26 01:42:01)
bq.  
bq.  
bq.  Review request for hbase, Ted Yu and Jimmy Xiang.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  * Parallelized load of .regioninfo files
bq.  * changed TreeMap to SortedMap in method signatures
bq.  * renamed a test's name.
bq.  
bq.  
bq.  This addresses bug HBASE-5712.
bq.  https://issues.apache.org/jira/browse/HBASE-5712
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 66156c2 
bq.src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java 6b64f10 
bq.  
bq.  Diff: https://reviews.apache.org/r/4883/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Ran patch 10x on trunk, passes.  Ran 1x on 0.92 and 0.94.
bq.  
bq.  Ther 0.90 version that is nearly identical except for ignoring changes 
near lines HBaseFsck lines 671-680.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  jmhsieh
bq.  
bq.



 Parallelize load of .regioninfo files in diagnostic/repair portion of hbck.
 ---

 Key: HBASE-5712
 URL: https://issues.apache.org/jira/browse/HBASE-5712
 Project: HBase
  Issue Type: Sub-task
  Components: hbck
Affects Versions: 0.90.7, 0.92.2, 0.94.0, 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Attachments: hbase-5712-90.patch, hbase-5712.patch


 On heavily loaded hdfs's some dfs nodes may not respond quickly and backs off 
 for 60s before attempting to read data from another datanode.  Portions of 
 the information gathered from hdfs (.regioninfo files) are loaded serially.  
 With HBase with clusters with 100's, or 1000's, or 1's regions 
 encountering these 60s delay blocks progress and can be very painful.  
 There is already some parallelization of portions of the hdfs information 
 load operations and the goal here is move the reading of .regioninfos into 
 the parallelized sections..

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5885) Invalid HFile block magic on Local file System

2012-04-27 Thread Lars Hofhansl (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264105#comment-13264105
 ] 

Lars Hofhansl commented on HBASE-5885:
--

+1 on patch

 Invalid HFile block magic on Local file System
 --

 Key: HBASE-5885
 URL: https://issues.apache.org/jira/browse/HBASE-5885
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.0, 0.96.0
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Blocker
 Fix For: 0.94.0

 Attachments: HBASE-5885-94-0.patch, HBASE-5885-trunk-0.patch


 ERROR: java.lang.RuntimeException: 
 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after 
 attempts=7, exceptions:
 Thu Apr 26 11:19:18 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: Could not iterate StoreFileScanner[HFileScanner for 
 reader 
 reader=file:/tmp/hbase-eclark/hbase/TestTable/e2d1c846363c75262cbfd85ea278b342/info/bae2681d63734066957b58fe791a0268,
  compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], 
 firstKey=01/info:data/1335463981520/Put, 
 lastKey=0002588100/info:data/1335463902296/Put, avgKeyLen=30, 
 avgValueLen=1000, entries=1215085, length=1264354417, 
 cur=000248/info:data/1335463994457/Put/vlen=1000/ts=0]
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:135)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95)
   at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:368)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.io.IOException: Invalid HFile block magic: 
 \xEC\xD5\x9D\xB4\xC2bfo
   at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153)
   at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock.init(HFileBlock.java:254)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1779)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:327)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:555)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:651)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130)
   ... 12 more
 Thu Apr 26 11:19:19 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: java.lang.IllegalArgumentException
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1132)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1121)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2420)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.lang.IllegalArgumentException
   at java.nio.Buffer.position(Buffer.java:216)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:630)
   at 
 

[jira] [Commented] (HBASE-5885) Invalid HFile block magic on Local file System

2012-04-27 Thread stack (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264114#comment-13264114
 ] 

stack commented on HBASE-5885:
--

+1 on patch.

 Invalid HFile block magic on Local file System
 --

 Key: HBASE-5885
 URL: https://issues.apache.org/jira/browse/HBASE-5885
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.0, 0.96.0
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Blocker
 Fix For: 0.94.0

 Attachments: HBASE-5885-94-0.patch, HBASE-5885-trunk-0.patch


 ERROR: java.lang.RuntimeException: 
 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after 
 attempts=7, exceptions:
 Thu Apr 26 11:19:18 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: Could not iterate StoreFileScanner[HFileScanner for 
 reader 
 reader=file:/tmp/hbase-eclark/hbase/TestTable/e2d1c846363c75262cbfd85ea278b342/info/bae2681d63734066957b58fe791a0268,
  compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], 
 firstKey=01/info:data/1335463981520/Put, 
 lastKey=0002588100/info:data/1335463902296/Put, avgKeyLen=30, 
 avgValueLen=1000, entries=1215085, length=1264354417, 
 cur=000248/info:data/1335463994457/Put/vlen=1000/ts=0]
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:135)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95)
   at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:368)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.io.IOException: Invalid HFile block magic: 
 \xEC\xD5\x9D\xB4\xC2bfo
   at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153)
   at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock.init(HFileBlock.java:254)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1779)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:327)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:555)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:651)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130)
   ... 12 more
 Thu Apr 26 11:19:19 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: java.lang.IllegalArgumentException
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1132)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1121)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2420)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.lang.IllegalArgumentException
   at java.nio.Buffer.position(Buffer.java:216)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:630)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130)
   at 
 

[jira] [Commented] (HBASE-5840) Open Region FAILED_OPEN doesn't clear the TaskMonitor Status, keeps showing the old status

2012-04-27 Thread stack (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264120#comment-13264120
 ] 

stack commented on HBASE-5840:
--

@Rajeshbabu I think you are right on the latter wal==null issue for newHRegion 
case.  +1 on patch.  Running by hadoopqa.

 Open Region FAILED_OPEN doesn't clear the TaskMonitor Status, keeps showing 
 the old status
 --

 Key: HBASE-5840
 URL: https://issues.apache.org/jira/browse/HBASE-5840
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: rajeshbabu
 Fix For: 0.96.0, 0.94.1

 Attachments: HBASE-5840.patch, HBASE-5840_v2.patch


 TaskMonitor Status will not be cleared in case Regions FAILED_OPEN. This will 
 keeps showing old status.
 This will miss leads the user.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-5840) Open Region FAILED_OPEN doesn't clear the TaskMonitor Status, keeps showing the old status

2012-04-27 Thread stack (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-5840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-5840:
-

Status: Patch Available  (was: Open)

 Open Region FAILED_OPEN doesn't clear the TaskMonitor Status, keeps showing 
 the old status
 --

 Key: HBASE-5840
 URL: https://issues.apache.org/jira/browse/HBASE-5840
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: rajeshbabu
 Fix For: 0.96.0, 0.94.1

 Attachments: HBASE-5840.patch, HBASE-5840_v2.patch


 TaskMonitor Status will not be cleared in case Regions FAILED_OPEN. This will 
 keeps showing old status.
 This will miss leads the user.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5885) Invalid HFile block magic on Local file System

2012-04-27 Thread Elliott Clark (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264121#comment-13264121
 ] 

Elliott Clark commented on HBASE-5885:
--

I would see this as the final solution.  Anything else would either be much 
bigger and more invasive or would rely on hadoop changes which is unlikely.
Inside local FS they seek to the next boundary of checksums.  However when not 
reading checksums it gives the wrong data when a read is issued.

 Invalid HFile block magic on Local file System
 --

 Key: HBASE-5885
 URL: https://issues.apache.org/jira/browse/HBASE-5885
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.0, 0.96.0
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Blocker
 Fix For: 0.94.0

 Attachments: HBASE-5885-94-0.patch, HBASE-5885-trunk-0.patch


 ERROR: java.lang.RuntimeException: 
 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after 
 attempts=7, exceptions:
 Thu Apr 26 11:19:18 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: Could not iterate StoreFileScanner[HFileScanner for 
 reader 
 reader=file:/tmp/hbase-eclark/hbase/TestTable/e2d1c846363c75262cbfd85ea278b342/info/bae2681d63734066957b58fe791a0268,
  compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], 
 firstKey=01/info:data/1335463981520/Put, 
 lastKey=0002588100/info:data/1335463902296/Put, avgKeyLen=30, 
 avgValueLen=1000, entries=1215085, length=1264354417, 
 cur=000248/info:data/1335463994457/Put/vlen=1000/ts=0]
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:135)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95)
   at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:368)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.io.IOException: Invalid HFile block magic: 
 \xEC\xD5\x9D\xB4\xC2bfo
   at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153)
   at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock.init(HFileBlock.java:254)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1779)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:327)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:555)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:651)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130)
   ... 12 more
 Thu Apr 26 11:19:19 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: java.lang.IllegalArgumentException
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1132)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1121)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2420)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 

[jira] [Commented] (HBASE-5867) Improve Compaction Throttle Default

2012-04-27 Thread Phabricator (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264123#comment-13264123
 ] 

Phabricator commented on HBASE-5867:


stack has commented on the revision [jira] [HBASE-5867] [89-fb] Improve 
Compaction Throttle Default.

  Should we pull this into trunk Nicolas?  Would it serve as good default for 
out-of-the-box hbase?

REVISION DETAIL
  https://reviews.facebook.net/D2943

BRANCH
  (no branch)


 Improve Compaction Throttle Default
 ---

 Key: HBASE-5867
 URL: https://issues.apache.org/jira/browse/HBASE-5867
 Project: HBase
  Issue Type: Improvement
Reporter: Nicolas Spiegelberg
Assignee: Nicolas Spiegelberg
Priority: Minor
 Attachments: D2943.1.patch


 We recently had a production issue where our compactions fell behind because 
 our compaction throttle was improperly tuned and accidentally upgraded all 
 compactions to the large pool.  The default from HBASE-3877 makes 1 bad 
 assumption: the default number of flushed files in a compaction.  Currently 
 the algorithm is:
 throttleSize ~= flushSize * 2
 This assumes that the basic compaction utilizes 3 files and that all 3 files 
 are compressed.  In this case, hbase.hstore.compaction.min == 6  the 
 values were not very compressible.  Both conditions should be taken into 
 consideration.  As a default, it is less damaging for the large thread to be 
 slightly higher than it needs to be versus having everything accidentally 
 promoted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-5893) Allow spaces in coprocessor conf (aka trim() className)

2012-04-27 Thread stack (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-5893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-5893:
-

   Resolution: Fixed
Fix Version/s: 0.94.0
   0.92.2
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Committed to 0.92, 0.94, and trunk (Usability).  Thanks for the patch Matteo.

 Allow spaces in coprocessor conf (aka trim() className)
 ---

 Key: HBASE-5893
 URL: https://issues.apache.org/jira/browse/HBASE-5893
 Project: HBase
  Issue Type: Bug
  Components: coprocessors
Affects Versions: 0.92.1, 0.94.0, 0.96.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
  Labels: conf, coprocessors
 Fix For: 0.92.2, 0.94.0

 Attachments: HBASE-5893-v0.patch


 This is annoying especially for coprocessors where you've long class name.
 but maybe is a bug of Configuration.getStrings() that doesn't trim each 
 string.
 When you've comma separated values like in the coprocessors case, you've to 
 pack together your values without spaces (v1,v2,v3,...) otherwise the 
 coprocessor is not loaded because the class name with spaces is not found.
 {code}
 property
namehbase.coprocessor.master.classes/name
value
 org.apache.hadoop.hbase.security.token.TokenProvider,
 org.apache.hadoop.hbase.security.access.AccessController
/value
 /property
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5885) Invalid HFile block magic on Local file System

2012-04-27 Thread Lars Hofhansl (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264132#comment-13264132
 ] 

Lars Hofhansl commented on HBASE-5885:
--

@Elliot: fair enough 




 Invalid HFile block magic on Local file System
 --

 Key: HBASE-5885
 URL: https://issues.apache.org/jira/browse/HBASE-5885
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.0, 0.96.0
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Blocker
 Fix For: 0.94.0

 Attachments: HBASE-5885-94-0.patch, HBASE-5885-trunk-0.patch


 ERROR: java.lang.RuntimeException: 
 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after 
 attempts=7, exceptions:
 Thu Apr 26 11:19:18 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: Could not iterate StoreFileScanner[HFileScanner for 
 reader 
 reader=file:/tmp/hbase-eclark/hbase/TestTable/e2d1c846363c75262cbfd85ea278b342/info/bae2681d63734066957b58fe791a0268,
  compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], 
 firstKey=01/info:data/1335463981520/Put, 
 lastKey=0002588100/info:data/1335463902296/Put, avgKeyLen=30, 
 avgValueLen=1000, entries=1215085, length=1264354417, 
 cur=000248/info:data/1335463994457/Put/vlen=1000/ts=0]
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:135)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95)
   at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:368)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.io.IOException: Invalid HFile block magic: 
 \xEC\xD5\x9D\xB4\xC2bfo
   at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153)
   at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock.init(HFileBlock.java:254)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1779)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:327)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:555)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:651)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130)
   ... 12 more
 Thu Apr 26 11:19:19 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: java.lang.IllegalArgumentException
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1132)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1121)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2420)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.lang.IllegalArgumentException
   at java.nio.Buffer.position(Buffer.java:216)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:630)
   at 
 

[jira] [Commented] (HBASE-5885) Invalid HFile block magic on Local file System

2012-04-27 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264134#comment-13264134
 ] 

Hadoop QA commented on HBASE-5885:
--

-1 overall.  Here are the results of testing the latest attachment 
  
http://issues.apache.org/jira/secure/attachment/12524942/HBASE-5885-trunk-0.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

+1 hadoop23.  The patch compiles against the hadoop 0.23.x profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 2 new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.replication.TestReplication
  org.apache.hadoop.hbase.io.hfile.TestChecksum
  org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1675//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1675//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1675//console

This message is automatically generated.

 Invalid HFile block magic on Local file System
 --

 Key: HBASE-5885
 URL: https://issues.apache.org/jira/browse/HBASE-5885
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.0, 0.96.0
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Blocker
 Fix For: 0.94.0

 Attachments: HBASE-5885-94-0.patch, HBASE-5885-trunk-0.patch


 ERROR: java.lang.RuntimeException: 
 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after 
 attempts=7, exceptions:
 Thu Apr 26 11:19:18 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: Could not iterate StoreFileScanner[HFileScanner for 
 reader 
 reader=file:/tmp/hbase-eclark/hbase/TestTable/e2d1c846363c75262cbfd85ea278b342/info/bae2681d63734066957b58fe791a0268,
  compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], 
 firstKey=01/info:data/1335463981520/Put, 
 lastKey=0002588100/info:data/1335463902296/Put, avgKeyLen=30, 
 avgValueLen=1000, entries=1215085, length=1264354417, 
 cur=000248/info:data/1335463994457/Put/vlen=1000/ts=0]
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:135)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95)
   at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:368)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.io.IOException: Invalid HFile block magic: 
 \xEC\xD5\x9D\xB4\xC2bfo
   at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153)
   at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock.init(HFileBlock.java:254)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1779)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637)
   at 
 

[jira] [Commented] (HBASE-5738) Using HBase with HA HDFS requires bogus hardcoded port value

2012-04-27 Thread stack (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264135#comment-13264135
 ] 

stack commented on HBASE-5738:
--

Shaneal, shall we close this (as per Eli's suggestion above)?

 Using HBase with HA HDFS requires bogus hardcoded port value
 

 Key: HBASE-5738
 URL: https://issues.apache.org/jira/browse/HBASE-5738
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.92.1, 0.94.0
Reporter: Shaneal Manek
Assignee: Shaneal Manek
 Attachments: HBASE-5738.patch


 When configuring HBase with HDFS HA, we currently have to have the 8020 port 
 (regardless of what port HDFS is using for the namenode rpc address) in the 
 following property in hbase-site.xml:
 {noformat}
   property
 namehbase.rootdir/name
 valuehdfs://ha-nn-uri:8020/hbase/value
   /property
 {noformat}
 Otherwise the master and regionservers will not start.
 The value in the above property should really just be 
 hdfs://ha-nn-uri/hbase (replace ha-nn-uri with your uri and hbase with 
 the name of the hbase directory in HDFS that you are using, as appropriate).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5876) TestImportExport has been failing against hadoop 0.23 profile

2012-04-27 Thread Jonathan Hsieh (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264138#comment-13264138
 ] 

Jonathan Hsieh commented on HBASE-5876:
---

I started going down the path and chatting with alejandro -- one of the guys 
involved with MAPREDUCE-3169 (new MiniMRCluster for 23).  It is likely a 
problem with job submission and will take a little digging.

 TestImportExport has been failing against hadoop 0.23 profile
 -

 Key: HBASE-5876
 URL: https://issues.apache.org/jira/browse/HBASE-5876
 Project: HBase
  Issue Type: Bug
Reporter: Zhihong Yu
Assignee: Jonathan Hsieh

 TestImportExport has been failing against hadoop 0.23 profile

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-5885) Invalid HFile block magic on Local file System

2012-04-27 Thread Elliott Clark (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-5885:
-

Status: Open  (was: Patch Available)

 Invalid HFile block magic on Local file System
 --

 Key: HBASE-5885
 URL: https://issues.apache.org/jira/browse/HBASE-5885
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.0, 0.96.0
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Blocker
 Fix For: 0.94.0

 Attachments: HBASE-5885-94-0.patch, HBASE-5885-trunk-0.patch


 ERROR: java.lang.RuntimeException: 
 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after 
 attempts=7, exceptions:
 Thu Apr 26 11:19:18 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: Could not iterate StoreFileScanner[HFileScanner for 
 reader 
 reader=file:/tmp/hbase-eclark/hbase/TestTable/e2d1c846363c75262cbfd85ea278b342/info/bae2681d63734066957b58fe791a0268,
  compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], 
 firstKey=01/info:data/1335463981520/Put, 
 lastKey=0002588100/info:data/1335463902296/Put, avgKeyLen=30, 
 avgValueLen=1000, entries=1215085, length=1264354417, 
 cur=000248/info:data/1335463994457/Put/vlen=1000/ts=0]
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:135)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95)
   at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:368)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.io.IOException: Invalid HFile block magic: 
 \xEC\xD5\x9D\xB4\xC2bfo
   at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153)
   at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock.init(HFileBlock.java:254)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1779)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:327)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:555)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:651)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130)
   ... 12 more
 Thu Apr 26 11:19:19 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: java.lang.IllegalArgumentException
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1132)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1121)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2420)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.lang.IllegalArgumentException
   at java.nio.Buffer.position(Buffer.java:216)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:630)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130)
   at 
 

[jira] [Updated] (HBASE-5885) Invalid HFile block magic on Local file System

2012-04-27 Thread Elliott Clark (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-5885:
-

Attachment: HBASE-5885-94-1.patch

94 missed a couple asserts

 Invalid HFile block magic on Local file System
 --

 Key: HBASE-5885
 URL: https://issues.apache.org/jira/browse/HBASE-5885
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.0, 0.96.0
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Blocker
 Fix For: 0.94.0

 Attachments: HBASE-5885-94-0.patch, HBASE-5885-94-1.patch, 
 HBASE-5885-trunk-0.patch, HBASE-5885-trunk-1.patch


 ERROR: java.lang.RuntimeException: 
 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after 
 attempts=7, exceptions:
 Thu Apr 26 11:19:18 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: Could not iterate StoreFileScanner[HFileScanner for 
 reader 
 reader=file:/tmp/hbase-eclark/hbase/TestTable/e2d1c846363c75262cbfd85ea278b342/info/bae2681d63734066957b58fe791a0268,
  compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], 
 firstKey=01/info:data/1335463981520/Put, 
 lastKey=0002588100/info:data/1335463902296/Put, avgKeyLen=30, 
 avgValueLen=1000, entries=1215085, length=1264354417, 
 cur=000248/info:data/1335463994457/Put/vlen=1000/ts=0]
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:135)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95)
   at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:368)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.io.IOException: Invalid HFile block magic: 
 \xEC\xD5\x9D\xB4\xC2bfo
   at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153)
   at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock.init(HFileBlock.java:254)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1779)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:327)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:555)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:651)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130)
   ... 12 more
 Thu Apr 26 11:19:19 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: java.lang.IllegalArgumentException
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1132)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1121)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2420)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.lang.IllegalArgumentException
   at java.nio.Buffer.position(Buffer.java:216)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:630)
   at 
 

[jira] [Updated] (HBASE-5885) Invalid HFile block magic on Local file System

2012-04-27 Thread Elliott Clark (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-5885:
-

Attachment: HBASE-5885-trunk-1.patch

trunk

 Invalid HFile block magic on Local file System
 --

 Key: HBASE-5885
 URL: https://issues.apache.org/jira/browse/HBASE-5885
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.0, 0.96.0
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Blocker
 Fix For: 0.94.0

 Attachments: HBASE-5885-94-0.patch, HBASE-5885-94-1.patch, 
 HBASE-5885-trunk-0.patch, HBASE-5885-trunk-1.patch


 ERROR: java.lang.RuntimeException: 
 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after 
 attempts=7, exceptions:
 Thu Apr 26 11:19:18 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: Could not iterate StoreFileScanner[HFileScanner for 
 reader 
 reader=file:/tmp/hbase-eclark/hbase/TestTable/e2d1c846363c75262cbfd85ea278b342/info/bae2681d63734066957b58fe791a0268,
  compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], 
 firstKey=01/info:data/1335463981520/Put, 
 lastKey=0002588100/info:data/1335463902296/Put, avgKeyLen=30, 
 avgValueLen=1000, entries=1215085, length=1264354417, 
 cur=000248/info:data/1335463994457/Put/vlen=1000/ts=0]
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:135)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95)
   at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:368)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.io.IOException: Invalid HFile block magic: 
 \xEC\xD5\x9D\xB4\xC2bfo
   at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153)
   at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock.init(HFileBlock.java:254)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1779)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:327)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:555)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:651)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130)
   ... 12 more
 Thu Apr 26 11:19:19 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: java.lang.IllegalArgumentException
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1132)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1121)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2420)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.lang.IllegalArgumentException
   at java.nio.Buffer.position(Buffer.java:216)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:630)
   at 
 

[jira] [Commented] (HBASE-5885) Invalid HFile block magic on Local file System

2012-04-27 Thread Elliott Clark (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264150#comment-13264150
 ] 

Elliott Clark commented on HBASE-5885:
--

Missed a couple of asserts.  This should fix the TestChecksum test.  
TestForceCacheImportantBlocks passes for me and TestReplication doesn't run for 
me.

I've got to run to catch a plane but I'll try and check on this later.

 Invalid HFile block magic on Local file System
 --

 Key: HBASE-5885
 URL: https://issues.apache.org/jira/browse/HBASE-5885
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.0, 0.96.0
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Blocker
 Fix For: 0.94.0

 Attachments: HBASE-5885-94-0.patch, HBASE-5885-94-1.patch, 
 HBASE-5885-trunk-0.patch, HBASE-5885-trunk-1.patch


 ERROR: java.lang.RuntimeException: 
 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after 
 attempts=7, exceptions:
 Thu Apr 26 11:19:18 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: Could not iterate StoreFileScanner[HFileScanner for 
 reader 
 reader=file:/tmp/hbase-eclark/hbase/TestTable/e2d1c846363c75262cbfd85ea278b342/info/bae2681d63734066957b58fe791a0268,
  compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], 
 firstKey=01/info:data/1335463981520/Put, 
 lastKey=0002588100/info:data/1335463902296/Put, avgKeyLen=30, 
 avgValueLen=1000, entries=1215085, length=1264354417, 
 cur=000248/info:data/1335463994457/Put/vlen=1000/ts=0]
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:135)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95)
   at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:368)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.io.IOException: Invalid HFile block magic: 
 \xEC\xD5\x9D\xB4\xC2bfo
   at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153)
   at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock.init(HFileBlock.java:254)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1779)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:327)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:555)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:651)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130)
   ... 12 more
 Thu Apr 26 11:19:19 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: java.lang.IllegalArgumentException
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1132)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1121)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2420)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: 

[jira] [Updated] (HBASE-5885) Invalid HFile block magic on Local file System

2012-04-27 Thread Elliott Clark (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-5885:
-

Status: Patch Available  (was: Open)

 Invalid HFile block magic on Local file System
 --

 Key: HBASE-5885
 URL: https://issues.apache.org/jira/browse/HBASE-5885
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.0, 0.96.0
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Blocker
 Fix For: 0.94.0

 Attachments: HBASE-5885-94-0.patch, HBASE-5885-94-1.patch, 
 HBASE-5885-trunk-0.patch, HBASE-5885-trunk-1.patch


 ERROR: java.lang.RuntimeException: 
 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after 
 attempts=7, exceptions:
 Thu Apr 26 11:19:18 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: Could not iterate StoreFileScanner[HFileScanner for 
 reader 
 reader=file:/tmp/hbase-eclark/hbase/TestTable/e2d1c846363c75262cbfd85ea278b342/info/bae2681d63734066957b58fe791a0268,
  compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], 
 firstKey=01/info:data/1335463981520/Put, 
 lastKey=0002588100/info:data/1335463902296/Put, avgKeyLen=30, 
 avgValueLen=1000, entries=1215085, length=1264354417, 
 cur=000248/info:data/1335463994457/Put/vlen=1000/ts=0]
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:135)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95)
   at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:368)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.io.IOException: Invalid HFile block magic: 
 \xEC\xD5\x9D\xB4\xC2bfo
   at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153)
   at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock.init(HFileBlock.java:254)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1779)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:327)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:555)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:651)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130)
   ... 12 more
 Thu Apr 26 11:19:19 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: java.lang.IllegalArgumentException
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1132)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1121)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2420)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.lang.IllegalArgumentException
   at java.nio.Buffer.position(Buffer.java:216)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:630)
   at 
 

[jira] [Commented] (HBASE-5840) Open Region FAILED_OPEN doesn't clear the TaskMonitor Status, keeps showing the old status

2012-04-27 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264165#comment-13264165
 ] 

Hadoop QA commented on HBASE-5840:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12524288/HBASE-5840_v2.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 3 new or modified tests.

+1 hadoop23.  The patch compiles against the hadoop 0.23.x profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 2 new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

+1 core tests.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1676//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1676//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1676//console

This message is automatically generated.

 Open Region FAILED_OPEN doesn't clear the TaskMonitor Status, keeps showing 
 the old status
 --

 Key: HBASE-5840
 URL: https://issues.apache.org/jira/browse/HBASE-5840
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: rajeshbabu
 Fix For: 0.96.0, 0.94.1

 Attachments: HBASE-5840.patch, HBASE-5840_v2.patch


 TaskMonitor Status will not be cleared in case Regions FAILED_OPEN. This will 
 keeps showing old status.
 This will miss leads the user.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5885) Invalid HFile block magic on Local file System

2012-04-27 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264176#comment-13264176
 ] 

Hadoop QA commented on HBASE-5885:
--

-1 overall.  Here are the results of testing the latest attachment 
  
http://issues.apache.org/jira/secure/attachment/12524953/HBASE-5885-trunk-1.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 3 new or modified tests.

-1 patch.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1677//console

This message is automatically generated.

 Invalid HFile block magic on Local file System
 --

 Key: HBASE-5885
 URL: https://issues.apache.org/jira/browse/HBASE-5885
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.0, 0.96.0
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Blocker
 Fix For: 0.94.0

 Attachments: HBASE-5885-94-0.patch, HBASE-5885-94-1.patch, 
 HBASE-5885-trunk-0.patch, HBASE-5885-trunk-1.patch


 ERROR: java.lang.RuntimeException: 
 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after 
 attempts=7, exceptions:
 Thu Apr 26 11:19:18 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: Could not iterate StoreFileScanner[HFileScanner for 
 reader 
 reader=file:/tmp/hbase-eclark/hbase/TestTable/e2d1c846363c75262cbfd85ea278b342/info/bae2681d63734066957b58fe791a0268,
  compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], 
 firstKey=01/info:data/1335463981520/Put, 
 lastKey=0002588100/info:data/1335463902296/Put, avgKeyLen=30, 
 avgValueLen=1000, entries=1215085, length=1264354417, 
 cur=000248/info:data/1335463994457/Put/vlen=1000/ts=0]
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:135)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95)
   at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:368)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.io.IOException: Invalid HFile block magic: 
 \xEC\xD5\x9D\xB4\xC2bfo
   at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153)
   at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock.init(HFileBlock.java:254)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1779)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:327)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:555)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:651)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130)
   ... 12 more
 Thu Apr 26 11:19:19 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: java.lang.IllegalArgumentException
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1132)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1121)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2420)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 

[jira] [Commented] (HBASE-5885) Invalid HFile block magic on Local file System

2012-04-27 Thread Todd Lipcon (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264178#comment-13264178
 ] 

Todd Lipcon commented on HBASE-5885:


{quote}
I would see this as the final solution. Anything else would either be much 
bigger and more invasive or would rely on hadoop changes which is unlikely.
Inside local FS they seek to the next boundary of checksums. However when not 
reading checksums it gives the wrong data when a read is issued.
{quote}

Please file an upstream bug in the hadoop project for this as well - we should 
absolutely fix this. I agree a workaround in the near term makes sense, but if 
LocalFileSystem is broken, we should fix it!

 Invalid HFile block magic on Local file System
 --

 Key: HBASE-5885
 URL: https://issues.apache.org/jira/browse/HBASE-5885
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.0, 0.96.0
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Blocker
 Fix For: 0.94.0

 Attachments: HBASE-5885-94-0.patch, HBASE-5885-94-1.patch, 
 HBASE-5885-trunk-0.patch, HBASE-5885-trunk-1.patch


 ERROR: java.lang.RuntimeException: 
 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after 
 attempts=7, exceptions:
 Thu Apr 26 11:19:18 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: Could not iterate StoreFileScanner[HFileScanner for 
 reader 
 reader=file:/tmp/hbase-eclark/hbase/TestTable/e2d1c846363c75262cbfd85ea278b342/info/bae2681d63734066957b58fe791a0268,
  compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], 
 firstKey=01/info:data/1335463981520/Put, 
 lastKey=0002588100/info:data/1335463902296/Put, avgKeyLen=30, 
 avgValueLen=1000, entries=1215085, length=1264354417, 
 cur=000248/info:data/1335463994457/Put/vlen=1000/ts=0]
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:135)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95)
   at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:368)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.io.IOException: Invalid HFile block magic: 
 \xEC\xD5\x9D\xB4\xC2bfo
   at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153)
   at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock.init(HFileBlock.java:254)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1779)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:327)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:555)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:651)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130)
   ... 12 more
 Thu Apr 26 11:19:19 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: java.lang.IllegalArgumentException
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1132)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1121)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2420)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 

[jira] [Commented] (HBASE-5853) java.lang.RuntimeException: readObject can't find class org.apache.hadoop.hdfs.protocol.HdfsFileStatus

2012-04-27 Thread jiafeng.zhang (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264179#comment-13264179
 ] 

jiafeng.zhang commented on HBASE-5853:
--

@Lars Hofhansl : import the data into hbase

 java.lang.RuntimeException: readObject can't find class 
 org.apache.hadoop.hdfs.protocol.HdfsFileStatus
 --

 Key: HBASE-5853
 URL: https://issues.apache.org/jira/browse/HBASE-5853
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.92.1
 Environment: hadoop-0.23.1 hbase-0.92.1 
Reporter: jiafeng.zhang
 Fix For: 0.92.1, 0.94.1


 2012-04-23 12:51:07,474 WARN org.apache.hadoop.ipc.Client: Unexpected error 
 reading responses on connection Thread[IPC Client (1260987126) connection to 
 server121/172.16.40.121:9000 from smp,5,main]
 java.lang.RuntimeException: readObject can't find class 
 org.apache.hadoop.hdfs.protocol.HdfsFileStatus
   at 
 org.apache.hadoop.io.ObjectWritable.loadClass(ObjectWritable.java:372)
   at 
 org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:223)
   at 
 org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:75)
   at 
 org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:832)
   at org.apache.hadoop.ipc.Client$Connection.run(Client.java:756)
 Caused by: java.lang.ClassNotFoundException: Class 
 org.apache.hadoop.hdfs.protocol.HdfsFileStatus not found
   at 
 org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1151)
   at 
 org.apache.hadoop.io.ObjectWritable.loadClass(ObjectWritable.java:368)
   ... 4 more
 2012-04-23 12:51:07,797 FATAL 
 org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server 
 server124,60020,1335152900476: Replay of HLog required. Forcing server 
 shutdown
 org.apache.hadoop.hbase.DroppedSnapshotException: region: 
 hbase_cdr,e0072b2b-5e19-431f-bb69-a6427765eac4,1334902272934.8365a7cbf90dd558f297d70224113c8a.
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1278)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1162)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1104)
   at 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:400)
   at 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:202)
   at 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:223)
   at java.lang.Thread.run(Thread.java:662)
 Caused by: java.io.IOException: Failed on local exception: 
 java.io.IOException: Error reading responses; Host Details : local host is: 
 server124/172.16.40.124; destination host is: server121:9000; 
   at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:724)
   at org.apache.hadoop.ipc.Client.call(Client.java:1094)
   at 
 org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:193)
   at $Proxy10.getFileInfo(Unknown Source)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:100)
   at 
 org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:65)
   at $Proxy10.getFileInfo(Unknown Source)
   at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1172)
   at 
 org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:725)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile.computeHDFSBlockDistribution(StoreFile.java:449)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:473)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:548)
   at 
 org.apache.hadoop.hbase.regionserver.Store.internalFlushCache(Store.java:595)
   at org.apache.hadoop.hbase.regionserver.Store.flushCache(Store.java:506)
   at org.apache.hadoop.hbase.regionserver.Store.access$100(Store.java:89)
   at 
 org.apache.hadoop.hbase.regionserver.Store$StoreFlusherImpl.flushCache(Store.java:1905)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1254)
   ... 6 more
 Caused by: java.io.IOException: Error reading responses
   at org.apache.hadoop.ipc.Client$Connection.run(Client.java:763)
 Caused by: java.lang.RuntimeException: readObject can't find class 
 

[jira] [Commented] (HBASE-5853) java.lang.RuntimeException: readObject can't find class org.apache.hadoop.hdfs.protocol.HdfsFileStatus

2012-04-27 Thread jiafeng.zhang (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264180#comment-13264180
 ] 

jiafeng.zhang commented on HBASE-5853:
--

During operation exception

 java.lang.RuntimeException: readObject can't find class 
 org.apache.hadoop.hdfs.protocol.HdfsFileStatus
 --

 Key: HBASE-5853
 URL: https://issues.apache.org/jira/browse/HBASE-5853
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.92.1
 Environment: hadoop-0.23.1 hbase-0.92.1 
Reporter: jiafeng.zhang
 Fix For: 0.92.1, 0.94.1


 2012-04-23 12:51:07,474 WARN org.apache.hadoop.ipc.Client: Unexpected error 
 reading responses on connection Thread[IPC Client (1260987126) connection to 
 server121/172.16.40.121:9000 from smp,5,main]
 java.lang.RuntimeException: readObject can't find class 
 org.apache.hadoop.hdfs.protocol.HdfsFileStatus
   at 
 org.apache.hadoop.io.ObjectWritable.loadClass(ObjectWritable.java:372)
   at 
 org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:223)
   at 
 org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:75)
   at 
 org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:832)
   at org.apache.hadoop.ipc.Client$Connection.run(Client.java:756)
 Caused by: java.lang.ClassNotFoundException: Class 
 org.apache.hadoop.hdfs.protocol.HdfsFileStatus not found
   at 
 org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1151)
   at 
 org.apache.hadoop.io.ObjectWritable.loadClass(ObjectWritable.java:368)
   ... 4 more
 2012-04-23 12:51:07,797 FATAL 
 org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server 
 server124,60020,1335152900476: Replay of HLog required. Forcing server 
 shutdown
 org.apache.hadoop.hbase.DroppedSnapshotException: region: 
 hbase_cdr,e0072b2b-5e19-431f-bb69-a6427765eac4,1334902272934.8365a7cbf90dd558f297d70224113c8a.
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1278)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1162)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1104)
   at 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:400)
   at 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:202)
   at 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:223)
   at java.lang.Thread.run(Thread.java:662)
 Caused by: java.io.IOException: Failed on local exception: 
 java.io.IOException: Error reading responses; Host Details : local host is: 
 server124/172.16.40.124; destination host is: server121:9000; 
   at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:724)
   at org.apache.hadoop.ipc.Client.call(Client.java:1094)
   at 
 org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:193)
   at $Proxy10.getFileInfo(Unknown Source)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:100)
   at 
 org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:65)
   at $Proxy10.getFileInfo(Unknown Source)
   at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1172)
   at 
 org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:725)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile.computeHDFSBlockDistribution(StoreFile.java:449)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:473)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:548)
   at 
 org.apache.hadoop.hbase.regionserver.Store.internalFlushCache(Store.java:595)
   at org.apache.hadoop.hbase.regionserver.Store.flushCache(Store.java:506)
   at org.apache.hadoop.hbase.regionserver.Store.access$100(Store.java:89)
   at 
 org.apache.hadoop.hbase.regionserver.Store$StoreFlusherImpl.flushCache(Store.java:1905)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1254)
   ... 6 more
 Caused by: java.io.IOException: Error reading responses
   at org.apache.hadoop.ipc.Client$Connection.run(Client.java:763)
 Caused by: java.lang.RuntimeException: readObject can't find class 
 org.apache.hadoop.hdfs.protocol.HdfsFileStatus

[jira] [Updated] (HBASE-5885) Invalid HFile block magic on Local file System

2012-04-27 Thread Ted Yu (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-5885:
--

Attachment: 5885-trunk-v2.txt

Wrong patch was attached as trunk-1.

Re-attaching patch for trunk.

 Invalid HFile block magic on Local file System
 --

 Key: HBASE-5885
 URL: https://issues.apache.org/jira/browse/HBASE-5885
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.0, 0.96.0
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Blocker
 Fix For: 0.94.0

 Attachments: 5885-trunk-v2.txt, HBASE-5885-94-0.patch, 
 HBASE-5885-94-1.patch, HBASE-5885-trunk-0.patch, HBASE-5885-trunk-1.patch


 ERROR: java.lang.RuntimeException: 
 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after 
 attempts=7, exceptions:
 Thu Apr 26 11:19:18 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: Could not iterate StoreFileScanner[HFileScanner for 
 reader 
 reader=file:/tmp/hbase-eclark/hbase/TestTable/e2d1c846363c75262cbfd85ea278b342/info/bae2681d63734066957b58fe791a0268,
  compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], 
 firstKey=01/info:data/1335463981520/Put, 
 lastKey=0002588100/info:data/1335463902296/Put, avgKeyLen=30, 
 avgValueLen=1000, entries=1215085, length=1264354417, 
 cur=000248/info:data/1335463994457/Put/vlen=1000/ts=0]
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:135)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95)
   at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:368)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.io.IOException: Invalid HFile block magic: 
 \xEC\xD5\x9D\xB4\xC2bfo
   at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153)
   at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock.init(HFileBlock.java:254)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1779)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:327)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:555)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:651)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130)
   ... 12 more
 Thu Apr 26 11:19:19 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: java.lang.IllegalArgumentException
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1132)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1121)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2420)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.lang.IllegalArgumentException
   at java.nio.Buffer.position(Buffer.java:216)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:630)

[jira] [Updated] (HBASE-5885) Invalid HFile block magic on Local file System

2012-04-27 Thread Ted Yu (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-5885:
--

Fix Version/s: 0.96.0
 Hadoop Flags: Reviewed

 Invalid HFile block magic on Local file System
 --

 Key: HBASE-5885
 URL: https://issues.apache.org/jira/browse/HBASE-5885
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.0, 0.96.0
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Blocker
 Fix For: 0.94.0, 0.96.0

 Attachments: 5885-trunk-v2.txt, HBASE-5885-94-0.patch, 
 HBASE-5885-94-1.patch, HBASE-5885-trunk-0.patch, HBASE-5885-trunk-1.patch


 ERROR: java.lang.RuntimeException: 
 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after 
 attempts=7, exceptions:
 Thu Apr 26 11:19:18 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: Could not iterate StoreFileScanner[HFileScanner for 
 reader 
 reader=file:/tmp/hbase-eclark/hbase/TestTable/e2d1c846363c75262cbfd85ea278b342/info/bae2681d63734066957b58fe791a0268,
  compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], 
 firstKey=01/info:data/1335463981520/Put, 
 lastKey=0002588100/info:data/1335463902296/Put, avgKeyLen=30, 
 avgValueLen=1000, entries=1215085, length=1264354417, 
 cur=000248/info:data/1335463994457/Put/vlen=1000/ts=0]
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:135)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95)
   at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:368)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.io.IOException: Invalid HFile block magic: 
 \xEC\xD5\x9D\xB4\xC2bfo
   at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153)
   at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock.init(HFileBlock.java:254)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1779)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:327)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:555)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:651)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130)
   ... 12 more
 Thu Apr 26 11:19:19 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: java.lang.IllegalArgumentException
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1132)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1121)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2420)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.lang.IllegalArgumentException
   at java.nio.Buffer.position(Buffer.java:216)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:630)
   at 
 

[jira] [Commented] (HBASE-5885) Invalid HFile block magic on Local file System

2012-04-27 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264201#comment-13264201
 ] 

Hadoop QA commented on HBASE-5885:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12524960/5885-trunk-v2.txt
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 3 new or modified tests.

+1 hadoop23.  The patch compiles against the hadoop 0.23.x profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 2 new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

+1 core tests.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1678//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1678//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1678//console

This message is automatically generated.

 Invalid HFile block magic on Local file System
 --

 Key: HBASE-5885
 URL: https://issues.apache.org/jira/browse/HBASE-5885
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.0, 0.96.0
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Blocker
 Fix For: 0.94.0, 0.96.0

 Attachments: 5885-trunk-v2.txt, HBASE-5885-94-0.patch, 
 HBASE-5885-94-1.patch, HBASE-5885-trunk-0.patch, HBASE-5885-trunk-1.patch


 ERROR: java.lang.RuntimeException: 
 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after 
 attempts=7, exceptions:
 Thu Apr 26 11:19:18 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: Could not iterate StoreFileScanner[HFileScanner for 
 reader 
 reader=file:/tmp/hbase-eclark/hbase/TestTable/e2d1c846363c75262cbfd85ea278b342/info/bae2681d63734066957b58fe791a0268,
  compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], 
 firstKey=01/info:data/1335463981520/Put, 
 lastKey=0002588100/info:data/1335463902296/Put, avgKeyLen=30, 
 avgValueLen=1000, entries=1215085, length=1264354417, 
 cur=000248/info:data/1335463994457/Put/vlen=1000/ts=0]
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:135)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95)
   at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:368)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.io.IOException: Invalid HFile block magic: 
 \xEC\xD5\x9D\xB4\xC2bfo
   at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153)
   at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock.init(HFileBlock.java:254)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1779)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:327)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:555)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:651)
   at 
 

[jira] [Commented] (HBASE-5890) SplitLog Rescan BusyWaits upon Zk.CONNECTIONLOSS

2012-04-27 Thread Ted Yu (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264205#comment-13264205
 ] 

Ted Yu commented on HBASE-5890:
---

Prakash made a good point.
{code}
this.timeoutMonitor = new TimeoutMonitor(
conf.getInt(hbase.splitlog.manager.timeoutmonitor.period,
1000),
{code}
TimeoutMonitor runs at interval longer than socket timeout. If the default 1 
sec interval (for TimeoutMonitor) is acceptable for the delay, this approach 
would work.

 SplitLog Rescan BusyWaits upon Zk.CONNECTIONLOSS
 

 Key: HBASE-5890
 URL: https://issues.apache.org/jira/browse/HBASE-5890
 Project: HBase
  Issue Type: Bug
Reporter: Nicolas Spiegelberg
Priority: Minor
 Fix For: 0.94.0, 0.96.0, 0.89-fb

 Attachments: HBASE-5890.patch


 We ran into a production issue yesterday where the SplitLogManager tried to 
 create a Rescan node in ZK.  The createAsync() generated a 
 KeeperException.CONNECTIONLOSS that was immedately sent to processResult(), 
 createRescan node with --retry_count was called, and this created a CPU 
 busywait that also clogged up the logs.  We should handle this better.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5874) The HBase do not configure the 'fs.default.name' attribute, the hbck tool and Merge tool throw IllegalArgumentException.

2012-04-27 Thread fulin wang (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264207#comment-13264207
 ] 

fulin wang commented on HBASE-5874:
---

I have tested, It pass.
There is some reason for add 'fs.default.name' attribute.
1) It make hbck tool and Merge tool easy to use.
2) We should supported old version, you can reference the MasterFileSystem 
class.
I suggested to add the attribute.

 The HBase do not configure the 'fs.default.name' attribute, the hbck tool and 
 Merge tool throw IllegalArgumentException.
 

 Key: HBASE-5874
 URL: https://issues.apache.org/jira/browse/HBASE-5874
 Project: HBase
  Issue Type: Bug
  Components: hbck
Affects Versions: 0.90.6
Reporter: fulin wang
Assignee: fulin wang
 Attachments: HBASE-5874-0.90.patch, HBASE-5874-trunk.patch


 The HBase do not configure the 'fs.default.name' attribute, the hbck tool and 
 Merge tool throw IllegalArgumentException.
 the hbck tool and Merge tool, we should add 'fs.default.name' attriubte to 
 the code.
 hbck exception:
 Exception in thread main java.lang.IllegalArgumentException: Wrong FS: 
 hdfs://160.176.0.101:9000/hbase/.META./1028785192/.regioninfo, expected: 
 file:///
   at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:412)
   at 
 org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:59)
   at 
 org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:382)
   at 
 org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:285)
   at 
 org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.init(ChecksumFileSystem.java:128)
   at 
 org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:301)
   at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:489)
   at 
 org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegioninfo(HBaseFsck.java:565)
   at 
 org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionInfos(HBaseFsck.java:596)
   at 
 org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:332)
   at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:360)
   at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:2907)
 
 Merge exception:  
 [2012-05-05 10:48:24,830] [ERROR] [main] [org.apache.hadoop.hbase.util.Merge 
 381] exiting due to error
 java.lang.IllegalArgumentException: Wrong FS: 
 hdfs://160.176.0.101:9000/hbase/.META./1028785192/.regioninfo, expected: 
 file:///
   at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:412)
   at 
 org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:59)
   at 
 org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:382)
   at 
 org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:285)
   at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:823)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkRegioninfoOnFilesystem(HRegion.java:415)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:340)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2679)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2665)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2634)
   at 
 org.apache.hadoop.hbase.util.MetaUtils.openMetaRegion(MetaUtils.java:276)
   at 
 org.apache.hadoop.hbase.util.MetaUtils.scanMetaRegion(MetaUtils.java:261)
   at org.apache.hadoop.hbase.util.Merge.run(Merge.java:115)
   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
   at org.apache.hadoop.hbase.util.Merge.main(Merge.java:379)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-5885) Invalid HFile block magic on Local file System

2012-04-27 Thread stack (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-5885:
-

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Committed 0.94 and trunk.  Thanks for the patch Elliott (See Todd's comment 
above -- add an old hadoop issue when you get a chance)

 Invalid HFile block magic on Local file System
 --

 Key: HBASE-5885
 URL: https://issues.apache.org/jira/browse/HBASE-5885
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.0, 0.96.0
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Blocker
 Fix For: 0.94.0, 0.96.0

 Attachments: 5885-trunk-v2.txt, HBASE-5885-94-0.patch, 
 HBASE-5885-94-1.patch, HBASE-5885-trunk-0.patch, HBASE-5885-trunk-1.patch


 ERROR: java.lang.RuntimeException: 
 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after 
 attempts=7, exceptions:
 Thu Apr 26 11:19:18 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: Could not iterate StoreFileScanner[HFileScanner for 
 reader 
 reader=file:/tmp/hbase-eclark/hbase/TestTable/e2d1c846363c75262cbfd85ea278b342/info/bae2681d63734066957b58fe791a0268,
  compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], 
 firstKey=01/info:data/1335463981520/Put, 
 lastKey=0002588100/info:data/1335463902296/Put, avgKeyLen=30, 
 avgValueLen=1000, entries=1215085, length=1264354417, 
 cur=000248/info:data/1335463994457/Put/vlen=1000/ts=0]
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:135)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95)
   at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:368)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.io.IOException: Invalid HFile block magic: 
 \xEC\xD5\x9D\xB4\xC2bfo
   at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153)
   at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock.init(HFileBlock.java:254)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1779)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:327)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:555)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:651)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130)
   ... 12 more
 Thu Apr 26 11:19:19 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: java.lang.IllegalArgumentException
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1132)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1121)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2420)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.lang.IllegalArgumentException
   at 

[jira] [Commented] (HBASE-5611) Replayed edits from regions that failed to open during recovery aren't removed from the global MemStore size

2012-04-27 Thread Ted Yu (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264210#comment-13264210
 ] 

Ted Yu commented on HBASE-5611:
---

Integrated to 0.92, 0.94 and trunk.

Thanks for the patch, Jieshan.

Thanks for the review, Lars.

 Replayed edits from regions that failed to open during recovery aren't 
 removed from the global MemStore size
 

 Key: HBASE-5611
 URL: https://issues.apache.org/jira/browse/HBASE-5611
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.6
Reporter: Jean-Daniel Cryans
Assignee: Jieshan Bean
Priority: Critical
 Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0

 Attachments: HBASE-5611-92.patch, HBASE-5611-94-minorchange.patch, 
 HBASE-5611-trunk-v2-minorchange.patch


 This bug is rather easy to get if the {{TimeoutMonitor}} is on, else I think 
 it's still possible to hit it if a region fails to open for more obscure 
 reasons like HDFS errors.
 Consider a region that just went through distributed splitting and that's now 
 being opened by a new RS. The first thing it does is to read the recovery 
 files and put the edits in the {{MemStores}}. If this process takes a long 
 time, the master will move that region away. At that point the edits are 
 still accounted for in the global {{MemStore}} size but they are dropped when 
 the {{HRegion}} gets cleaned up. It's completely invisible until the 
 {{MemStoreFlusher}} needs to force flush a region and that none of them have 
 edits:
 {noformat}
 2012-03-21 00:33:39,303 DEBUG 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush thread woke up 
 because memory above low water=5.9g
 2012-03-21 00:33:39,303 ERROR 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flusher failed 
 for entry null
 java.lang.IllegalStateException
 at 
 com.google.common.base.Preconditions.checkState(Preconditions.java:129)
 at 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:199)
 at 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:223)
 at java.lang.Thread.run(Thread.java:662)
 {noformat}
 The {{null}} here is a region. In my case I had so many edits in the 
 {{MemStore}} during recovery that I'm over the low barrier although in fact 
 I'm at 0. It happened yesterday and it still printing this out.
 To fix this we need to be able to decrease the global {{MemStore}} size when 
 the region can't open.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-5611) Replayed edits from regions that failed to open during recovery aren't removed from the global MemStore size

2012-04-27 Thread Lars Hofhansl (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-5611:
-

Resolution: Fixed
Status: Resolved  (was: Patch Available)

 Replayed edits from regions that failed to open during recovery aren't 
 removed from the global MemStore size
 

 Key: HBASE-5611
 URL: https://issues.apache.org/jira/browse/HBASE-5611
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.6
Reporter: Jean-Daniel Cryans
Assignee: Jieshan Bean
Priority: Critical
 Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0

 Attachments: HBASE-5611-92.patch, HBASE-5611-94-minorchange.patch, 
 HBASE-5611-trunk-v2-minorchange.patch


 This bug is rather easy to get if the {{TimeoutMonitor}} is on, else I think 
 it's still possible to hit it if a region fails to open for more obscure 
 reasons like HDFS errors.
 Consider a region that just went through distributed splitting and that's now 
 being opened by a new RS. The first thing it does is to read the recovery 
 files and put the edits in the {{MemStores}}. If this process takes a long 
 time, the master will move that region away. At that point the edits are 
 still accounted for in the global {{MemStore}} size but they are dropped when 
 the {{HRegion}} gets cleaned up. It's completely invisible until the 
 {{MemStoreFlusher}} needs to force flush a region and that none of them have 
 edits:
 {noformat}
 2012-03-21 00:33:39,303 DEBUG 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush thread woke up 
 because memory above low water=5.9g
 2012-03-21 00:33:39,303 ERROR 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flusher failed 
 for entry null
 java.lang.IllegalStateException
 at 
 com.google.common.base.Preconditions.checkState(Preconditions.java:129)
 at 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:199)
 at 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:223)
 at java.lang.Thread.run(Thread.java:662)
 {noformat}
 The {{null}} here is a region. In my case I had so many edits in the 
 {{MemStore}} during recovery that I'm over the low barrier although in fact 
 I'm at 0. It happened yesterday and it still printing this out.
 To fix this we need to be able to decrease the global {{MemStore}} size when 
 the region can't open.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5890) SplitLog Rescan BusyWaits upon Zk.CONNECTIONLOSS

2012-04-27 Thread Lars Hofhansl (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264213#comment-13264213
 ] 

Lars Hofhansl commented on HBASE-5890:
--

Important for 0.94.0? Just say, and I wait.

 SplitLog Rescan BusyWaits upon Zk.CONNECTIONLOSS
 

 Key: HBASE-5890
 URL: https://issues.apache.org/jira/browse/HBASE-5890
 Project: HBase
  Issue Type: Bug
Reporter: Nicolas Spiegelberg
Priority: Minor
 Fix For: 0.94.0, 0.96.0, 0.89-fb

 Attachments: HBASE-5890.patch


 We ran into a production issue yesterday where the SplitLogManager tried to 
 create a Rescan node in ZK.  The createAsync() generated a 
 KeeperException.CONNECTIONLOSS that was immedately sent to processResult(), 
 createRescan node with --retry_count was called, and this created a CPU 
 busywait that also clogged up the logs.  We should handle this better.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5876) TestImportExport has been failing against hadoop 0.23 profile

2012-04-27 Thread Ted Yu (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264212#comment-13264212
 ] 

Ted Yu commented on HBASE-5876:
---

From 
https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK-on-Hadoop-23/156/testReport/org.apache.hadoop.hbase.mapreduce/TestImportTsv/testMROnTableWithCustomMapper/:
{code}
2012-04-27 00:47:47,745 ERROR [pool-1-thread-1] service.CompositeService(72): 
Error starting services org.apache.hadoop.mapred.MiniMRCluster
org.apache.hadoop.yarn.YarnException: java.io.IOException: ResourceManager 
failed to start. Final state is INITED
at 
org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.start(MiniYARNCluster.java:152)
at 
org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
at 
org.apache.hadoop.mapred.MiniMRClientClusterFactory.create(MiniMRClientClusterFactory.java:67)
at org.apache.hadoop.mapred.MiniMRCluster.init(MiniMRCluster.java:180)
at org.apache.hadoop.mapred.MiniMRCluster.init(MiniMRCluster.java:170)
at org.apache.hadoop.mapred.MiniMRCluster.init(MiniMRCluster.java:162)
at org.apache.hadoop.mapred.MiniMRCluster.init(MiniMRCluster.java:154)
at org.apache.hadoop.mapred.MiniMRCluster.init(MiniMRCluster.java:147)
{code}
From 
https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK-on-Hadoop-23/156/testReport/org.apache.hadoop.hbase.mapreduce/TestImportExport/testWithDeletes/:
{code}
java.io.FileNotFoundException: File does not exist: 
/home/hudson/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-core/0.23.2-SNAPSHOT/hadoop-mapreduce-client-core-0.23.2-SNAPSHOT.jar
at 
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:729)
at 
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:208)
at 
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:71)
at 
org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:246)
at 
org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:284)
at 
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:355)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1221)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1218)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1239)
at 
org.apache.hadoop.hbase.mapreduce.TestImportExport.testWithDeletes(TestImportExport.java:181)
{code}

 TestImportExport has been failing against hadoop 0.23 profile
 -

 Key: HBASE-5876
 URL: https://issues.apache.org/jira/browse/HBASE-5876
 Project: HBase
  Issue Type: Bug
Reporter: Zhihong Yu
Assignee: Jonathan Hsieh

 TestImportExport has been failing against hadoop 0.23 profile

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5885) Invalid HFile block magic on Local file System

2012-04-27 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264226#comment-13264226
 ] 

Hudson commented on HBASE-5885:
---

Integrated in HBase-0.94 #153 (See 
[https://builds.apache.org/job/HBase-0.94/153/])
HBASE-5885  Invalid HFile block magic on Local file System (Revision 
1331676)

 Result = FAILURE
stack : 
Files : 
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java
* 
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestChecksum.java


 Invalid HFile block magic on Local file System
 --

 Key: HBASE-5885
 URL: https://issues.apache.org/jira/browse/HBASE-5885
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.0, 0.96.0
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Blocker
 Fix For: 0.94.0, 0.96.0

 Attachments: 5885-trunk-v2.txt, HBASE-5885-94-0.patch, 
 HBASE-5885-94-1.patch, HBASE-5885-trunk-0.patch, HBASE-5885-trunk-1.patch


 ERROR: java.lang.RuntimeException: 
 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after 
 attempts=7, exceptions:
 Thu Apr 26 11:19:18 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: Could not iterate StoreFileScanner[HFileScanner for 
 reader 
 reader=file:/tmp/hbase-eclark/hbase/TestTable/e2d1c846363c75262cbfd85ea278b342/info/bae2681d63734066957b58fe791a0268,
  compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] 
 [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] 
 [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], 
 firstKey=01/info:data/1335463981520/Put, 
 lastKey=0002588100/info:data/1335463902296/Put, avgKeyLen=30, 
 avgValueLen=1000, entries=1215085, length=1264354417, 
 cur=000248/info:data/1335463994457/Put/vlen=1000/ts=0]
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:135)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95)
   at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:368)
   at 
 org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
 Caused by: java.io.IOException: Invalid HFile block magic: 
 \xEC\xD5\x9D\xB4\xC2bfo
   at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153)
   at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock.init(HFileBlock.java:254)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1779)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:327)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:555)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:651)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130)
   ... 12 more
 Thu Apr 26 11:19:19 PDT 2012, 
 org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: 
 java.io.IOException: java.lang.IllegalArgumentException
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1132)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1121)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2420)
   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 

[jira] [Commented] (HBASE-5893) Allow spaces in coprocessor conf (aka trim() className)

2012-04-27 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264227#comment-13264227
 ] 

Hudson commented on HBASE-5893:
---

Integrated in HBase-0.94 #153 (See 
[https://builds.apache.org/job/HBase-0.94/153/])
HBASE-5893 Allow spaces in coprocessor conf (aka trim() className) 
(Revision 1331639)

 Result = FAILURE
stack : 
Files : 
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java


 Allow spaces in coprocessor conf (aka trim() className)
 ---

 Key: HBASE-5893
 URL: https://issues.apache.org/jira/browse/HBASE-5893
 Project: HBase
  Issue Type: Bug
  Components: coprocessors
Affects Versions: 0.92.1, 0.94.0, 0.96.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
  Labels: conf, coprocessors
 Fix For: 0.92.2, 0.94.0

 Attachments: HBASE-5893-v0.patch


 This is annoying especially for coprocessors where you've long class name.
 but maybe is a bug of Configuration.getStrings() that doesn't trim each 
 string.
 When you've comma separated values like in the coprocessors case, you've to 
 pack together your values without spaces (v1,v2,v3,...) otherwise the 
 coprocessor is not loaded because the class name with spaces is not found.
 {code}
 property
namehbase.coprocessor.master.classes/name
value
 org.apache.hadoop.hbase.security.token.TokenProvider,
 org.apache.hadoop.hbase.security.access.AccessController
/value
 /property
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-5611) Replayed edits from regions that failed to open during recovery aren't removed from the global MemStore size

2012-04-27 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264225#comment-13264225
 ] 

Hudson commented on HBASE-5611:
---

Integrated in HBase-0.94 #153 (See 
[https://builds.apache.org/job/HBase-0.94/153/])
HBASE-5611 Replayed edits from regions that failed to open during recovery 
aren't removed from the global MemStore size (Jieshan) (Revision 1331683)

 Result = FAILURE
tedyu : 
Files : 
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerAccounting.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java


 Replayed edits from regions that failed to open during recovery aren't 
 removed from the global MemStore size
 

 Key: HBASE-5611
 URL: https://issues.apache.org/jira/browse/HBASE-5611
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.6
Reporter: Jean-Daniel Cryans
Assignee: Jieshan Bean
Priority: Critical
 Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0

 Attachments: HBASE-5611-92.patch, HBASE-5611-94-minorchange.patch, 
 HBASE-5611-trunk-v2-minorchange.patch


 This bug is rather easy to get if the {{TimeoutMonitor}} is on, else I think 
 it's still possible to hit it if a region fails to open for more obscure 
 reasons like HDFS errors.
 Consider a region that just went through distributed splitting and that's now 
 being opened by a new RS. The first thing it does is to read the recovery 
 files and put the edits in the {{MemStores}}. If this process takes a long 
 time, the master will move that region away. At that point the edits are 
 still accounted for in the global {{MemStore}} size but they are dropped when 
 the {{HRegion}} gets cleaned up. It's completely invisible until the 
 {{MemStoreFlusher}} needs to force flush a region and that none of them have 
 edits:
 {noformat}
 2012-03-21 00:33:39,303 DEBUG 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush thread woke up 
 because memory above low water=5.9g
 2012-03-21 00:33:39,303 ERROR 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flusher failed 
 for entry null
 java.lang.IllegalStateException
 at 
 com.google.common.base.Preconditions.checkState(Preconditions.java:129)
 at 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:199)
 at 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:223)
 at java.lang.Thread.run(Thread.java:662)
 {noformat}
 The {{null}} here is a region. In my case I had so many edits in the 
 {{MemStore}} during recovery that I'm over the low barrier although in fact 
 I'm at 0. It happened yesterday and it still printing this out.
 To fix this we need to be able to decrease the global {{MemStore}} size when 
 the region can't open.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira