[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles with hbck
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-6586:
-
Fix Version/s: (was: 0.95.0)
0.94.2
Fix up after bulk move overwrote some 0.94.2 fix versions w/ 0.95.0 (Noticed by
Lars Hofhansl)
Quarantine Corrupted HFiles with hbck
-
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Affects Versions: 0.92.1
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Fix For: 0.94.2
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8b.patch,
hbase-6586-92-v8.patch, hbase-6586-92-v9.patch, hbase-6586-94-v3.patch,
hbase-6586-94-v9.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch,
hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch,
hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch,
hbase-6586-trunk-v8.patch, hbase-6586-trunk-v9.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles with hbck
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Fix Version/s: (was: 0.94.3)
0.94.2
Quarantine Corrupted HFiles with hbck
-
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Affects Versions: 0.92.1
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Fix For: 0.92.2, 0.96.0, 0.94.2
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8b.patch,
hbase-6586-92-v8.patch, hbase-6586-92-v9.patch, hbase-6586-94-v3.patch,
hbase-6586-94-v9.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch,
hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch,
hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch,
hbase-6586-trunk-v8.patch, hbase-6586-trunk-v9.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles with hbck
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
Quarantine Corrupted HFiles with hbck
-
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Affects Versions: 0.92.1
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Fix For: 0.92.2, 0.96.0, 0.94.2
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8b.patch,
hbase-6586-92-v8.patch, hbase-6586-92-v9.patch, hbase-6586-94-v3.patch,
hbase-6586-94-v9.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch,
hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch,
hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch,
hbase-6586-trunk-v8.patch, hbase-6586-trunk-v9.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Status: Open (was: Patch Available)
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-92-v8.patch, hbase-6586-94-v3.patch,
hbase-6586-94-v8.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch,
hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch,
hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch,
hbase-6586-trunk-v8.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Attachment: hbase-6586-trunk-v8.patch
Previous run didn't pass but when tested locally it passed. Trying again.
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-92-v8.patch, hbase-6586-94-v3.patch,
hbase-6586-94-v8.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch,
hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch,
hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch,
hbase-6586-trunk-v8.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Status: Patch Available (was: Open)
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-92-v8.patch, hbase-6586-94-v3.patch,
hbase-6586-94-v8.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch,
hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch,
hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch,
hbase-6586-trunk-v8.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Attachment: (was: hbase-6586-trunk-v8.patch)
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-92-v8.patch, hbase-6586-94-v3.patch,
hbase-6586-94-v8.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch,
hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch,
hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch,
hbase-6586-trunk-v8.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Attachment: hbase-6586-92-v8a.patch
Previous 92 version was missing new HBaseIOException.java file.
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8.patch,
hbase-6586-94-v3.patch, hbase-6586-94-v8.patch, hbase-6586.patch,
hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch,
hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch,
hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Attachment: hbase-6586-92-v8b.patch
Cleaned up some a merge conflict that was in a comment that I had missed.
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8b.patch,
hbase-6586-92-v8.patch, hbase-6586-94-v3.patch, hbase-6586-94-v8.patch,
hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch,
hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch,
hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Attachment: (was: hbase-6586-trunk-v8.patch)
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8b.patch,
hbase-6586-92-v8.patch, hbase-6586-94-v3.patch, hbase-6586-94-v8.patch,
hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch,
hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch,
hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Attachment: hbase-6586-trunk-v8.patch
Doing this again because hadoopqa picked up the wrong patch.
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8b.patch,
hbase-6586-92-v8.patch, hbase-6586-94-v3.patch, hbase-6586-94-v8.patch,
hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch,
hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch,
hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles with hbck
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Summary: Quarantine Corrupted HFiles with hbck (was: Quarantine Corrupted
HFiles)
Quarantine Corrupted HFiles with hbck
-
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8b.patch,
hbase-6586-92-v8.patch, hbase-6586-94-v3.patch, hbase-6586-94-v8.patch,
hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch,
hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch,
hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles with hbck
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Attachment: hbase-6586-trunk-v9.patch
hbase-6586-94-v9.patch
hbase-6586-92-v9.patch
Versions I've committed to trunk and 0.94. Also the version I was planning on
committing to 0.92 (Ted wants to run the suite before letting the patch in).
Quarantine Corrupted HFiles with hbck
-
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Affects Versions: 0.92.1
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Fix For: 0.92.2, 0.96.0, 0.94.3
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8b.patch,
hbase-6586-92-v8.patch, hbase-6586-92-v9.patch, hbase-6586-94-v3.patch,
hbase-6586-94-v8.patch, hbase-6586-94-v9.patch, hbase-6586.patch,
hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch,
hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch,
hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch,
hbase-6586-trunk-v9.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles with hbck
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Affects Version/s: 0.92.1
Fix Version/s: 0.94.3
0.96.0
0.92.2
Hadoop Flags: Reviewed
Quarantine Corrupted HFiles with hbck
-
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Affects Versions: 0.92.1
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Fix For: 0.92.2, 0.96.0, 0.94.3
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8b.patch,
hbase-6586-92-v8.patch, hbase-6586-92-v9.patch, hbase-6586-94-v3.patch,
hbase-6586-94-v8.patch, hbase-6586-94-v9.patch, hbase-6586.patch,
hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch,
hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch,
hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch,
hbase-6586-trunk-v9.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles with hbck
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Attachment: (was: hbase-6586-94-v8.patch)
Quarantine Corrupted HFiles with hbck
-
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Affects Versions: 0.92.1
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Fix For: 0.92.2, 0.96.0, 0.94.3
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8b.patch,
hbase-6586-92-v8.patch, hbase-6586-92-v9.patch, hbase-6586-94-v3.patch,
hbase-6586-94-v9.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch,
hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch,
hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch,
hbase-6586-trunk-v8.patch, hbase-6586-trunk-v9.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Attachment: hbase-6586-92-v8.patch
hbase-6586-94-v8.patch
Backported to 92 and 94.
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-92-v8.patch, hbase-6586-94-v3.patch,
hbase-6586-94-v8.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch,
hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch,
hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch,
hbase-6586-trunk-v8.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Status: Patch Available (was: Open)
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch,
hbase-6586-trunk-v3.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Attachment: hbase-6586-94-v3.patch
hbase-6586-92-v3.patch
hbase-6586-trunk-v3.patch
v3, including backports to 92/94.
Minor changes due to HBASE-5714 perms checks.
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch,
hbase-6586-trunk-v3.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Attachment: (was: hbase-6586-trunk-v3.patch)
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch,
hbase-6586-trunk-v3.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Status: Open (was: Patch Available)
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch,
hbase-6586-trunk-v3.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Attachment: hbase-6586-trunk-v3.patch
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch,
hbase-6586-trunk-v3.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Status: Patch Available (was: Open)
trying again
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch,
hbase-6586-trunk-v3.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Status: Open (was: Patch Available)
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch,
hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Status: Patch Available (was: Open)
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch,
hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Attachment: hbase-6586-trunk-v4.patch
v4, changes io exception to illegal argument exception.
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch,
hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Attachment: hbase-6586-trunk-v5.patch
v5, addresses ted's comments.
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch,
hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch,
hbase-6586-trunk-v5.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Status: Open (was: Patch Available)
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch,
hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch,
hbase-6586-trunk-v5.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Status: Patch Available (was: Open)
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch,
hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch,
hbase-6586-trunk-v5.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Attachment: (was:
0001-hbase-6586-cosmetic-changes-from-review-part-2.patch)
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch,
hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch,
hbase-6586-trunk-v5.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Attachment: 0001-hbase-6586-cosmetic-changes-from-review-part-2.patch
fixed all spacing problems and other review concerns.
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch,
hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch,
hbase-6586-trunk-v5.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Status: Open (was: Patch Available)
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch,
hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch,
hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Attachment: hbase-6586-trunk-v6.patch
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch,
hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch,
hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Attachment: (was: hbase-6586-trunk-v6.patch)
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch,
hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch,
hbase-6586-trunk-v5.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Attachment: hbase-6586-trunk-v6.patch
(previous version had part of another patch i was working on).
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch,
hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch,
hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Status: Patch Available (was: Open)
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch,
hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch,
hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Attachment: hbase-6586-trunk-v7.patch
more spacing fixes and review updates.
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch,
hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch,
hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch,
hbase-6586-trunk-v7.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Attachment: hbase-6586-trunk-v8.patch
Removed unused SchemaMetrics code from TestHFile
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch,
hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch,
hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch,
hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Status: Open (was: Patch Available)
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch,
hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch,
hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch,
hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Status: Patch Available (was: Open)
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch,
hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch,
hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch,
hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Attachment: 0001-hbase-6568-hbck-quarantine-v6.patch
v2, adds checks/warning for deleted files.
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch,
hbase-6586.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Attachment: hbase-6586.patch
Version for trunk. Tested locally, seems to pass. Will post on review board,
and after it make it through there I plan to backport to 0.94/0.92.
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Bug
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: hbase-6586.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Issue Type: Improvement (was: Bug)
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Improvement
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: hbase-6586.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Description:
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem
in my dev environment but we suspect this may be related to HDFS-3731. On the
HBase side, it seems reasonable to quarantine what are most likely truncated
hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600
(expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
was:
We've encountered a few upgrades that get stuck from 0.90 hbases + 20.2/1.x
hdfs to 0.92 hbases + hdfs 2.x. I haven't been able to duplicate the problem
in my dev environment but we suspect this may be related to HDFS-3731. On the
HBase side, it seems reasonable to quarantine what are most likely truncated
hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
