[
https://issues.apache.org/jira/browse/HBASE-23106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16943845#comment-16943845
]
Michael Stack commented on HBASE-23106:
---------------------------------------
Here is example of what output looks like when we run the WALReader... passing
two files -- first a small recovered.edits file and then a corrupted
recovered.edits file:
{code}
kalashnikov:hbase.apache.git stack$ ./bin/hbase
org.apache.hadoop.hbase.wal.WALReader /tmp/0000000000000932379
2019-10-03 12:28:46,375 WARN [main] util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
/tmp/0000000000000932379
3448 GENIE2_modality_syncdata/3c2fc74c7d67cd20883799b3f7789b3b/932373 3271
11309 GENIE2_modality_syncdata/3c2fc74c7d67cd20883799b3f7789b3b/932374 7767
18565 GENIE2_modality_syncdata/3c2fc74c7d67cd20883799b3f7789b3b/932375 7162
34217 GENIE2_modality_syncdata/3c2fc74c7d67cd20883799b3f7789b3b/932376 15558
44758 GENIE2_modality_syncdata/3c2fc74c7d67cd20883799b3f7789b3b/932377 10447
57357 GENIE2_modality_syncdata/3c2fc74c7d67cd20883799b3f7789b3b/932378 12505
60722 GENIE2_modality_syncdata/3c2fc74c7d67cd20883799b3f7789b3b/932379 3271
kalashnikov:hbase.apache.git stack$ ./bin/hbase
org.apache.hadoop.hbase.wal.WALReader /tmp/0000000000000932380
2019-10-03 12:28:52,110 WARN [main] util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
/tmp/0000000000000932380
2019-10-03 12:28:52,568 WARN [main] hbase.KeyValueUtil:
8\x8E\xF8o\xF4\x03"\xD1\x89\xB8\x86\x5C\xECo\x16\x96\x5C,\xC4EH\xACO\x1E\xA4VF\xA7\xFB\xF9\x7F\xAF](\xFC\xE2\x07QQ\xE6\xA8-\xD1l\x0C\xE7\x13YX_[\x89+n&JS
\xE5\x81\xF4\xFAH/\xBF\xF4\xBD\x95\x1F\x8F*\x96\x09\xC8Q\xEAoJt2\xA3E'\xA2\x01\xAB\xF2y%B\x18\xBAI)\x02\x0B\x0F\xFBv^\xF4\x85$^>\x14\x84\xA7\xBEb\xFE{N\xEB7W~~\xFB\x5C\x0E=\xD6FG\xB1ms"\x04\x81\xC69\x97i\x16m\xD0\x1F\xBC0iR.\xBB6\xEE\xEF\xDE\x05OF>\x029\x01\xF6\xFBJ\x8D\xA7\xDA\xE8x\xB2\x5C\xED20\x85Z\x09y\x82x\x98\xBAQ~,\x80@\x87r!\xAFa\x8F$\xDB\xC7\x06\x84\xE6\xEAe\xD0\xE2xF\x8E\x9A\x17mtbpb\x0F\xF2\xA4\x13\xBAn\x06j\x86\xFA\x92p\x1F\xBB\x97\xD7WZ\x09\xED(\x15\x8B\xAA\xDE\x09\xB8\xFE\x91\x8C\x0F\x17\x19s\x09\xF6\x95\x8DT\xAC\x0A\x12\xDALq@\xAALB\x8C\xB3\x0A\xCE\xA4k\xA5\x9F\xA25\xDBh.\xA5\xED\xD1(\x17s\x01\xCA\x94Gm\xDB%\x19\x9CI\x9FZw\xB2\xB8\xE9:\x8B\x9B\xD2\xF6\xE0N\x07j\xEDE)\x9C\xBA5"\xA5n\x9D\x088qh/\x85\xE6\xC4\x85\x10\xD9\x0A)=1+\x98\x05\x09\xBE\xCA\x99C\xDD!\x9F\x08\x9D\x94\x0F\xD2\xFA\xD3\xFB\xA6\x91,
offset=0, length=1080
2019-10-03 12:28:52,569 WARN [main] wal.ProtobufLogReader: Encountered a
malformed edit, seeking back to last good position in file, from 1261 to 83
java.io.EOFException: EOF while reading 12 WAL KVs; started reading at 177 and
read up to 1261
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:397)
at
org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:98)
at org.apache.hadoop.hbase.wal.WALReader.read(WALReader.java:41)
at org.apache.hadoop.hbase.wal.WALReader.main(WALReader.java:59)
Caused by: java.lang.IllegalArgumentException: Invalid value length in
KeyValue, valueLength=1986,
KeyValueBytesHex=\x00\x00\x00T\x00\x00\x07\xC2\x00$efbb2495-8230-4262-a09d-e3261e54eb51\x04blobpm|com.apple.contact.people|pmap\x7F\xFF\xFF\xFF\xFF\xFF\xFF\xFF\x04\x1F\x8B\x08\x00\x00\x00\x00\x00\x00\x00\xEDW_l\x1CG\x19\x1F\x9F\xFF\xC4\xB1\x1D\xC7I\x13D+H\xF3`J\x03\xD9\xFB\xBB\xB7w\x974\x98\xB3\x0F\x07'\xD7\xC4\xF8\xE2@bJ5\xBB;\xEB\x1Bgwg33k\xDF%P\xB5O\xF0\x82T\x09\x1E\xA8hy\x00!\xD1\xD2\xAAOH
\x90\xFA\xD8\x0A\x04\x0FH}\xA2\xA2\x0F
\xF1\x80\xE0\x85J\x88"\x01\xDF\xEC\x9F\xBB\xB5\xE3\x18\xF3\xCEJ\xB7\xDA\xF9f\xBE\xDF\xF7\xFF\xFB\xE6^\xFF\x0B\x1A\x17\x1C\x9D\xDF\xC2\xDB8\x1FJ\xEA\xE6\x97\x98\xEB\x12KR\xE6\x8B\xF9u\xDFc6u(6]2\xA4?\xBA\x88\x9E\xFF\xD5W\xFE~&\x87F\xDAh\xC4\x92\xE8#m\xC5^P\xEC\x85\xE1\xB1\x8B\xBD\x00\x90KCd\x8B\xF9V\xC89\xF1%\x08I??\x8FE\xF7i\x1C\xCC\xDF\xC4nH\xC4MJv\x1E7\xDA/~\xE9[\xE7.\xE5\x10\xEAqT=$\xFFP\xAE\xC2\xB0_\xFA\xFD\xC9W\xBE\xAE0@\xC5Q\x0F\x07\x12=\x99Qr\x88Tx\x00)V\xFB\x89\xC3\x89M\xE5\x8C\xA2\xD1\x154-\xC8\xA6\x07{Ocqg\x05\xCD$\xABN\x97:r\x03M&K!Qi\xE3\x90\xAA\xCCwb\x1EP\x09!t\x1C~\x1F\x0By\xCA~\x08\x9F\xA4\xECk;\x0B\xCD\xF2w\x1BRyT!\xCD\x81\x81\xF9\xFF\x0Dc\x18\x94\x91e4\xE52l/cK2\x0E\x11zr_$\x97YwD~\x8D\xC07\xC7\xBEl\xC3\xD2Y\xFF\xD1\xF9\xF3\xEFl\xFD9\x0A\xCA\x98\xE8\xFB\x90:\x85\xFD]\x11q\x17vq\xCFw\x80!\x0E\x8E~X\x89\xF3\xD7\x98\xEF`\xCA\x15+\xF9F\xF9O\x9D\xFBoM\xC7i\xA5\x1D\x1AB\xF1\xFE\xF4\xCC\xF7\xBF\xFDZ\xEB\xF6W\x0FJ\xC9\x98\xB7i\x0A`\xB5\xE4\x17B\x12\x12[\xF1v9\xF3\xE9=\xC2\xC1\xFA\xA5pa\xED\x0F`\xFD\x0A\xD4\x9C\xC4\x92\x00\x96q\x08\xAC\xEB;\xBE*\xC0,X\xE5\xFD7\x7F\xF6\x86\xC7^M#\x8A\x16>\x8B\x90\xB8\x8B\x9E\x83\xB4S\xEF\xB9\xFF\xD3\x0E\xA6I\xD4\xB2\x98\x97\xC7A\xE0\x12\xE5y\x09~\xCE\x07\x84\xC1\xF2B\xB9Z\xB4\xEAV\xA3\xA1\xE9Ub\xC2\xAB^\xD3\xCC\x92ch\x95F\xADQ\xAD\x98\xF5Z\xB1b.\xB0K%\xCB\xB1
\x19?\xA5p\x04\xE5\x14`8\xC44\x8F-\x02H\x5CP!\xF3M\x8B\xAC\xC6\x9F*\x827N\xFD\xEB\xB1\x17\xFF\x96\xFBg\x0E\x8Dn\xA0#p\xAE\x85!\x0DPnc\xB1\x8D\x8E%\xCB\xEB\xE6\x16\xB41\x89\xCE\xB4\x01\xB7\xA0p\x0B\xB0S\x00\xA0\x0E\xE1\x14\xBB\xF4\x9EB\xBA\xD8\x8E\xF8o\xF4\x03"\xD1\x89\xB8\x86\x5C\xECo\x16\x96\x5C,\xC4EH\xACO\x1E\xA4VF\xA7\xFB\xF9\x7F\xAF](\xFC\xE2\x07QQ\xE6\xA8-\xD1l\x0C\xE7\x13YX_[\x89+n&JS
\xE5\x81\xF4\xFAH/\xBF\xF4\xBD\x95\x1F\x8F*\x96\x09\xC8Q\xEAoJt2\xA3E'\xA2\x01\xAB\xF2y%B\x18\xBAI)\x02\x0B\x0F\xFBv^\xF4\x85$^>\x14\x84\xA7\xBEb\xFE{N\xEB7W~~\xFB\x5C\x0E=\xD6FG\xB1ms"\x04\x81\xC69\x97i\x16m\xD0\x1F\xBC0iR.\xBB6\xEE\xEF\xDE\x05OF>\x029\x01\xF6\xFBJ\x8D\xA7\xDA\xE8x\xB2\x5C\xED20\x85Z\x09y\x82x\x98\xBAQ~,\x80@\x87r!\xAFa\x8F$\xDB\xC7\x06\x84\xE6\xEAe\xD0\xE2xF\x8E\x9A\x17mtbpb\x0F\xF2\xA4\x13\xBAn\x06j\x86\xFA\x92p\x1F\xBB\x97\xD7WZ\x09\xED(\x15\x8B\xAA\xDE\x09\xB8\xFE\x91\x8C\x0F\x17\x19s\x09\xF6\x95\x8DT\xAC\x0A\x12\xDALq@\xAALB\x8C\xB3\x0A\xCE\xA4k\xA5\x9F\xA25\xDBh.\xA5\xED\xD1(\x17s\x01\xCA\x94Gm\xDB%\x19\x9CI\x9FZw\xB2\xB8\xE9:\x8B\x9B\xD2\xF6\xE0N\x07j\xEDE)\x9C\xBA5"\xA5n\x9D\x088qh/\x85\xE6\xC4\x85\x10\xD9\x0A)=1+\x98\x05\x09\xBE\xCA\x99C\xDD!\x9F\x08\x9D\x94\x0F\xD2\xFA\xD3\xFB\xA6\x91,
offset=0, length=1080
at
org.apache.hadoop.hbase.KeyValueUtil.checkKeyValueBytes(KeyValueUtil.java:556)
at org.apache.hadoop.hbase.KeyValue.<init>(KeyValue.java:344)
at
org.apache.hadoop.hbase.KeyValueUtil.createKeyValueFromInputStream(KeyValueUtil.java:717)
at
org.apache.hadoop.hbase.codec.KeyValueCodecWithTags$KeyValueDecoder.parseCell(KeyValueCodecWithTags.java:81)
at
org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:68)
at org.apache.hadoop.hbase.wal.WALEdit.readFromCells(WALEdit.java:214)
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:382)
... 3 more
{code}
> [HBCK2/hbase-operator-tools] Add a WAL verifier
> -----------------------------------------------
>
> Key: HBASE-23106
> URL: https://issues.apache.org/jira/browse/HBASE-23106
> Project: HBase
> Issue Type: Sub-task
> Reporter: Michael Stack
> Priority: Major
>
> We had a WALPlayer that loads edits up into hbase cluster but what I wanted
> yesterday was a WAL verifier so I could find the bad WAL messing me up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
