[
https://issues.apache.org/jira/browse/HBASE-21379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16664625#comment-16664625
]
Zheng Hu commented on HBASE-21379:
----------------------------------
Decoded the hlog, and read the byte hex carefully. I found something...
{code}
Writer Classes: ProtobufLogWriter AsyncProtobufLogWriter
Cell Codec Class: org.apache.hadoop.hbase.regionserver.wal.WALCellCodec
==> expectedCells: 45
==> KeyValue Cell Hex: \x00\x00\x00z\x00\x00\x00
\x00\x00\x00R\x00\x118242341#02#438917\x01f\x0A^\x00\x00\x01f\xAC\x8A\xBC\xE9\x04{"tp50":5,"avg":5,"min":5,"tp90":5,"max":5,"count":1,"tp99":5,"tp999":5,"error":0}
==> KeyValue Cell Hex:
\x00\x00\x00|\x00\x00\x00"\x00\x00\x00R\x00\x138242741#02#12029960\x01f\x0Ab\x00\x00\x01f\xAC\x8A\xBDY\x04{"tp50":0,"avg":0,"min":0,"tp90":0,"max":0,"count":1,"tp99":0,"tp999":0,"error":0}
==> KeyValue Cell Hex:
\x00\x00\x00}\x00\x00\x00"\x00\x00\x00S\x00\x138242741#02#12029961\x01f\x0Ab\x00\x00\x01f\xAC\x8A\xBDY\x04{"tp50":0,"avg":0,"min":0,"tp90":0,"max":0,"count":89,"tp99":0,"tp999":0,"error":0}
==> KeyValue Cell Hex:
\x00\x00\x00|\x00\x00\x00"\x00\x00\x00R\x00\x138242741#02#12029962\x01f\x0Ab\x00\x00\x01f\xAC\x8A\xBDY\x04{"tp50":0,"avg":0,"min":0,"tp90":0,"max":0,"count":1,"tp99":0,"tp999":0,"error":0}
==> KeyValue Cell Hex:
\x00\x00\x00}\x00\x00\x00"\x00\x00\x00S\x00\x138242741#02#12029963\x01f\x0Ab\x00\x00\x01f\xAC\x8A\xBDY\x04{"tp50":0,"avg":0,"min":0,"tp90":0,"max":0,"count":94,"tp99":0,"tp999":0,"error":0}
==> KeyValue Cell Hex:
\x00\x00\x00}\x00\x00\x00"\x00\x00\x00S\x00\x138242741#02#12029996\x01f\x0Ab\x00\x00\x01f\xAC\x8A\xBDC\x04{"tp50":0,"avg":0,"min":0,"tp90":0,"max":0,"count":93,"tp99":0,"tp999":0,"error":0}
==> KeyValue Cell Hex:
\x00\x00\x00}\x00\x00\x00"\x00\x00\x00S\x00\x138242741#02#12029997\x01f\x0Ab\x00\x00\x01f\xAC\x8A\xBDC\x04{"tp50":0,"avg":0,"min":0,"tp90":0,"max":0,"count":85,"tp99":0,"tp999":0,"error":0}
==> KeyValue Cell Hex:
\x00\x00\x00|\x00\x00\x00"\x00\x00\x00R\x00\x138242741#02#12030005\x01f\x0Ab\x00\x00\x01f\xAC\x8A\xBDC\x04{"tp50":0,"avg":0,"min":0,"tp90":0,"max":0,"count":3,"tp99":0,"tp999":0,"error":0}
==> KeyValue Cell Hex:
\x00\x00\x00}\x00\x00\x00"\x00\x00\x00S\x00\x138242741#02#12030058\x01f\x0Ab\x00\x00\x01f\xAC\x8A\xBD}\x04{"tp50":0,"avg":0,"min":0,"tp90":0,"max":0,"count":75,"tp99":0,"tp999":0,"error":0}
==> KeyValue Cell Hex:
\x00\x00\x00}\x00\x00\x00"\x00\x00\x00S\x00\x138242741#02#12030061\x01f\x0Ab\x00\x00\x01f\xAC\x8A\xBD}\x04{"tp50":0,"avg":0,"min":0,"tp90":0,"max":0,"count":92,"tp99":0,"tp999":0,"error":0}
==> KeyValue Cell Hex:
\x00\x00\x00|\x00\x00\x00"\x00\x00\x00R\x00\x138242741#02#12030066\x01f\x0Ab\x00\x00\x01f\xAC\x8A\xBDZ\x04{"tp50":0,"avg":0,"min":0,"tp90":0,"max":0,"count":1,"tp99":0,"tp999":0,"error":0}
==> KeyValue Cell Hex:
\x00\x00\x00~\x00\x00\x00"\x00\x00\x00T\x00\x138242741#02#12030068\x01f\x0Ab\x00\x00\x01f\xAC\x8A\xBDZ\x04{"tp50":0,"avg":0,"min":0,"tp90":0,"max":0,"count":103,"tp99":0,"tp999":0,"error":0}
==> KeyValue Cell Hex:
\x00\x00\x00|\x00\x00\x00"\x00\x00\x00R\x00\x138242741#02#12030069\x01f\x0Ab\x00\x00\x01f\xAC\x8A\xBDZ\x04{"tp50":0,"avg":0,"min":0,"tp90":0,"max":0,"count":1,"tp99":0,"tp999":0,"error":0}
==> KeyValue Cell Hex:
\x00\x00\x00}\x00\x00\x00"\x00\x00\x00S\x00\x138242741#02#12030070\x01f\x0Ab\x00\x00\x01f\xAC\x8A\xBDZ\x04{"tp50":0,"avg":0,"min":0,"tp90":0,"max":0,"count":92,"tp99":0,"tp999":0,"error":0}
==> KeyValue Cell Hex:
\x00\x00\x00|\x00\x00\x00"\x00\x00\x00R\x00\x138242741#02#12030071\x01f\x0Ab\x00\x00\x01f\xAC\x8A\xBDY\x04{"tp50":0,"avg":0,"min":0,"tp90":0,"max":0,"count":1,"tp99":0,"tp999":0,"error":0}
==> KeyValue Cell Hex:
\x00\x00\x00}\x00\x00\x00"\x00\x00\x00S\x00\x138242741#02#12030098\x01f\x0Ab\x00\x00\x01f\xAC\x8A\xBC\xFE\x04{"tp50":0,"avg":0,"min":0,"tp90":0,"max":0,"count":91,"tp99":0,"tp999":0,"error":0}
==> KeyValue Cell Hex:
\x00\x00\x00}99":2,"error":0}\x00\x00\x00{\x00\x00\x00\x1F\x00\x00\x00T\x00\x108484941#02#39103\x01f\x0A_\x00\x00\x01f\xAC\x8A\xBDr\x04{"tp50":0,"avg":1,"min":0,"tp90":0,"max":2,"count":103,"tp99":2,"t
==> KeyValue Cell Hex:
\x00\x00\x00|\x00\x00\x00"\x00\x00\x00R\x00\x138242741#02#12030139\x01f\x0Ab\x00\x00\x01f\xAC\x8A\xBC\xDA\x04{"tp50":0,"avg":0,"min":0,"tp90":0,"max":0,"count":1,"tp99":0,"tp999":0,"error":0}
==> KeyValue Cell Hex:
\x00\x00\x00}\x00\x00\x00"\x00\x00\x00S\x00\x138242741#02#12030141\x01f\x0Ab\x00\x00\x01f\xAC\x8A\xBC\xDA\x04{"tp50":0,"avg":0,"min":0,"tp90":0,"max":0,"count":84,"tp99":0,"tp999":0,"error":0}
==> KeyValue Cell Hex:
\x00\x00\x00|or":0}\x00\x00\x00y\x00\x00\x00\x1F\x00\x00\x00R\x00\x10894737#02#225822\x01f\x0A^\x00\x00\x01f\xAC\x8A\xBC\x9F\x04{"tp50":1,"avg":1,"min":1,"tp90":1,"max":1,"count":1,"tp99":1,"tp999":1,"er
==> KeyValue Cell Hex:
\x00\x00\x00}"error":0}\x00\x00\x00y\x00\x00\x00\x1F\x00\x00\x00R\x00\x10894737#02#419456\x01f\x0A^\x00\x00\x01f\xAC\x8A\xBCi\x04{"tp50":0,"avg":0,"min":0,"tp90":0,"max":0,"count":1,"tp99":0,"tp999":0,
==> KeyValue Cell Hex:
\x00\x00\x00{:0}\x00\x00\x00x\x00\x00\x00\x1E\x00\x00\x00R\x00\x0F894737#02#82463\x01f\x0A^\x00\x00\x01f\xAC\x8A\xBC\x9F\x04{"tp50":2,"avg":2,"min":2,"tp90":2,"max":2,"count":1,"tp99":2,"tp999":2,"error
==> KeyValue Cell Hex: \x00\x00\x00|\x00\x00\x00
\x00\x00\x00T\x00\x118242741#02#164028\x01f\x0Ab\x00\x00\x01f\xAC\x8A\xBDY\x04{"tp50":0,"avg":0,"min":0,"tp90":0,"max":0,"count":105,"tp99":0,"tp999":0,"error":0}
==> KeyValue Cell Hex: \x00\x00\x00z\x00\x00\x00
\x00\x00\x00R\x00\x118242741#02#170112\x01f\x0Ab\x00\x00\x01f\xAC\x8A\xBDC\x04{"tp50":0,"avg":0,"min":0,"tp90":0,"max":0,"count":1,"tp99":0,"tp999":0,"error":0}
==> KeyValue Cell Hex: \x00\x00\x00|\x00\x00\x00
\x00\x00\x00T\x00\x118242741#02#186505\x01f\x0Ab\x00\x00\x01f\xAC\x8A\xBDC\x04{"tp50":0,"avg":0,"min":0,"tp90":0,"max":0,"count":101,"tp99":0,"tp999":0,"error":0}
==> KeyValue Cell Hex: \x00\x00\x00|\x00\x00\x00
\x00\x00\x00T\x00\x118242741#02#186560\x01f\x0Ab\x00\x00\x01f\xAC\x8A\xBDZ\x04{"tp50":0,"avg":0,"min":0,"tp90":0,"max":0,"count":103,"tp99":0,"tp999":0,"error":0}
==> KeyValue Cell Hex: \x00\x00\x00{\x00\x00\x00
\x00\x00\x00S\x00\x118242741#02#194715\x01f\x0Ab\x00\x00\x01f\xAC\x8A\xBDC\x04{"tp50":0,"avg":1,"min":0,"tp90":0,"max":1,"count":89,"tp99":0,"tp999":0,"error":0}
==> KeyValue Cell Hex: \x00\x00\x00{\x00\x00\x00
\x00\x00\x00S\x00\x118242741#02#198872\x01f\x0Ab\x00\x00\x01f\xAC\x8A\xBD}\x04{"tp50":0,"avg":0,"min":0,"tp90":0,"max":0,"count":77,"tp99":0,"tp999":0,"error":0}
==> KeyValue Cell Hex:
\x00\x00\x00{99":0,"error":0}\x00\x00\x00x\x00\x00\x00\x1E\x00\x00\x00R\x00\x0F894737#02#97152\x01f\x0A^\x00\x00\x01f\xAC\x8A\xBCi\x04{"tp50":0,"avg":0,"min":0,"tp90":0,"max":0,"count":2,"tp99":0,"tp
==> KeyValue Cell Hex:
\x00\x00\x00{":2,"error":0}\x00\x00\x00}\x00\x00\x00"\x00\x00\x00S\x00\x138484941#02#12030070\x01f\x0A_\x00\x00\x01f\xAC\x8A\xBDs\x04{"tp50":0,"avg":1,"min":0,"tp90":0,"max":2,"count":79,"tp99":0,
==> KeyValue Cell Hex:
\x00\x00\x00z\x00\x00\x00\x1F\x00\x00\x00S\x00\x108242741#02#28859\x01f\x0Ab\x00\x00\x01f\xAC\x8A\xBDY\x04{"tp50":0,"avg":0,"min":0,"tp90":0,"max":0,"count":91,"tp99":0,"tp999":0,"error":0}
==> KeyValue Cell Hex:
\x00\x00\x00z:0,"error":0}\x00\x00\x00x\x00\x00\x00\x1E\x00\x00\x00R\x00\x0F894737#02#64003\x01f\x0A^\x00\x00\x01f\xAC\x8A\xBCi\x04{"tp50":1,"avg":1,"min":1,"tp90":1,"max":1,"count":1,"tp99":1,"tp99
==> KeyValue Cell Hex:
\x00\x00\x00z,"error":0}\x00\x00\x00y\x00\x00\x00\x1F\x00\x00\x00R\x00\x10894737#02#419461\x01f\x0A^\x00\x00\x01f\xAC\x8A\xBCi\x04{"tp50":0,"avg":0,"min":0,"tp90":0,"max":0,"count":2,"tp99":0,"tp999
==> KeyValue Cell Hex: \x00\x00\x00z\x00\x00\x00
\x00\x00\x00R\x00\x118242741#02#331999\x01f\x0Ab\x00\x00\x01f\xAC\x8A\xBD}\x04{"tp50":0,"avg":0,"min":0,"tp90":0,"max":0,"count":1,"tp99":0,"tp999":0,"error":0}
==> KeyValue Cell Hex: \x00\x00\x00z\x00\x00\x00
\x00\x00\x00R\x00\x118242741#02#350430\x01f\x0Ab\x00\x00\x01f\xAC\x8A\xBD}\x04{"tp50":0,"avg":0,"min":0,"tp90":0,"max":0,"count":1,"tp99":0,"tp999":0,"error":0}
==> KeyValue Cell Hex:
\x00\x00\x00{\x00\x00\x00\x1F\x00\x00\x00T\x00\x108242741#02#37061\x01f\x0Ab\x00\x00\x01f\xAC\x8A\xBDZ\x04{"tp50":0,"avg":0,"min":0,"tp90":0,"max":0,"count":104,"tp99":0,"tp999":0,"error":0}
==> KeyValue Cell Hex: \x00\x00\x00z\x00\x00\x00
\x00\x00\x00R\x00\x118242741#02#374983\x01f\x0Ab\x00\x00\x01f\xAC\x8A\xBDZ\x04{"tp50":0,"avg":0,"min":0,"tp90":0,"max":0,"count":2,"tp99":0,"tp999":0,"error":0}
==> KeyValue Cell Hex:
\x00\x00\x00z\x00\x00\x00\x1F\x00\x00\x00S\x00\x108242741#02#39103\x01f\x0Ab\x00\x00\x01f\xAC\x8A\xBDY\x04{"tp50":0,"avg":0,"min":0,"tp90":0,"max":0,"count":81,"tp99":0,"tp999":0,"error":0}
==> KeyValue Cell Hex: \x00\x00\x00z\x00\x00\x00
\x00\x00\x00R\x00\x118242741#02#397353\x01f\x0Ab\x00\x00\x01f\xAC\x8A\xBD\x1E\x04{"tp50":0,"avg":0,"min":0,"tp90":0,"max":0,"count":1,"tp99":0,"tp999":0,"error":0}
==> KeyValue Cell Hex:
\x00\x00\x00z\x00\x00\x00\x1F\x00\x00\x00S\x00\x108242741#02#41154\x01f\x0Ab\x00\x00\x01f\xAC\x8A\xBDZ\x04{"tp50":0,"avg":0,"min":0,"tp90":0,"max":0,"count":90,"tp99":0,"tp999":0,"error":0}
==> KeyValue Cell Hex:
\x00\x00\x00z\x00\x00\x00y\x00\x00\x00\x1F\x00\x00\x00R\x00\x10894737#02#434277\x01f\x0A^\x00\x00\x01f\xAC\x8A\xBC\xE7\x04{"tp50":0,"avg":1,"min":0,"tp90":0,"max":1,"count":2,"tp99":0,"tp999":0,"error"
==> KeyValue Cell Hex:
\x00\x00\x00y"error":0}\x00\x00\x00y\x00\x00\x00\x1F\x00\x00\x00R\x00\x10894737#02#399619\x01f\x0A^\x00\x00\x01f\xAC\x8A\xBCi\x04{"tp50":1,"avg":1,"min":1,"tp90":1,"max":1,"count":1,"tp99":1,"tp999
==> KeyValue Cell Hex:
\x00\x00\x00y\x00\x00\x00\x1F\x00\x00\x00R\x00\x108242741#02#61453\x01f\x0Ab\x00\x00\x01f\xAC\x8A\xBC\xDA\x04{"tp50":0,"avg":0,"min":0,"tp90":0,"max":0,"count":1,"tp99":0,"tp999":0,"error":0}
==> KeyValue Cell Hex:
\x00\x00\x00z\x00\x00\x00\x1F\x00\x00\x00S\x00\x108242741#02#69643\x01f\x0Ab\x00\x00\x01f\xAC\x8A\xBC\xDA\x04{"tp50":0,"avg":0,"min":0,"tp90":0,"max":0,"count":81,"tp99":0,"tp999":0,"error":0}
==> KeyValue Cell Hex:
\x00\x00\x00y\x00\x00\x00\x1F\x00\x00\x00R\x00\x108242741#02#84109\x01f\x0Ab\x00\x00\x01f\xAC\x8A\xBDC\x04{"tp50":0,"avg":0,"min":0,"tp90":0,"max":0,"count":2,"tp99":0,"tp999":0,"error":0}
{code}
For example, the cell:
{code}
\x00\x00\x00}99":2,"error":0}\x00\x00\x00{\x00\x00\x00\x1F\x00\x00\x00T\x00\x108484941#02#39103\x01f\x0A_\x00\x00\x01f\xAC\x8A\xBDr\x04{"tp50":0,"avg":1,"min":0,"tp90":0,"max":2,"count":103,"tp99":2,"t
{code}
some bytes are rotated from right to left..... the full json should be
{code}
{"tp50":0,"avg":1,"min":0,"tp90":0,"max":2,"count":103,"tp99":2,"t999":2,"error":0}
{code}
But the bytes removed the prefix 9, and move the `99":2,"error":0}` to the
front of the byte array ...
All the *mis-encoded cells* are fit the same rule above.... seems there's a bug
when encode the cell's byte array....
> RegionServer Stop by ArrayIndexOutOfBoundsException of WAL when replication
> enabled
> -----------------------------------------------------------------------------------
>
> Key: HBASE-21379
> URL: https://issues.apache.org/jira/browse/HBASE-21379
> Project: HBase
> Issue Type: Bug
> Components: Replication
> Affects Versions: 2.0.0
> Reporter: justice
> Priority: Major
> Attachments: hbase-wal-p.tgz, log.tgz, wal.tgz
>
>
> log as follow:
> {code:java}
> //代码占位符
> 2018-10-24 09:22:42,381 INFO [regionserver/11-3-19-10:16020]
> wal.AbstractFSWAL: New WAL
> /hbase/WALs/11-3-19-10.jd.local,16020,1540344155469/11-3-19-10.jd.local%2C16020%2C1540344155469.1540344162124
> │
> 2018-10-24 09:23:05,151 ERROR
> [regionserver/11-3-19-10:16020.replicationSource.11-3-19-10.jd.local%2C16020%2C1540344155469,2.replicationSource.wal-reader.11-3-19-10.jd.local%2C16020%2C1540344155469,2]
> region│
> server.ReplicationSource: Unexpected exception in
> regionserver/11-3-19-10:16020.replicationSource.11-3-19-10.jd.local%2C16020%2C1540344155469,2.replicationSource.wal-reader.11-3-19-10.jd.local%2C16020%2C1540│
> 344155469,2
> currentPath=hdfs://11-3-18-67.JD.LOCAL:9000/hbase/WALs/11-3-19-10.jd.local,16020,1540344155469/11-3-19-10.jd.local%2C16020%2C1540344155469.1540344162124
> │
> java.lang.ArrayIndexOutOfBoundsException: 8830 │
> at org.apache.hadoop.hbase.KeyValue.getFamilyLength(KeyValue.java:1365) │
> at org.apache.hadoop.hbase.KeyValue.getFamilyLength(KeyValue.java:1358) ┤
> at org.apache.hadoop.hbase.CellUtil.cloneFamily(CellUtil.java:114) │
> at
> org.apache.hadoop.hbase.replication.ScopeWALEntryFilter.filterCell(ScopeWALEntryFilter.java:54)
> │
> at
> org.apache.hadoop.hbase.replication.ChainWALEntryFilter.filterCells(ChainWALEntryFilter.java:90)
> │
> at
> org.apache.hadoop.hbase.replication.ChainWALEntryFilter.filter(ChainWALEntryFilter.java:77)
> │
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.filterEntry(ReplicationSourceWALReader.java:234)
> │
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:170)
> │ at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:133)
> │
> 2018-10-24 09:23:05,153 INFO
> [regionserver/11-3-19-10:16020.replicationSource.11-3-19-10.jd.local%2C16020%2C1540344155469,2.replicationSource.wal-reader.11-3-19-10.jd.local%2C16020%2C1540344155469,2]
> region│
> server.HRegionServer: ***** STOPPING region server
> '11-3-19-10.jd.local,16020,1540344155469' *****
> {code}
> hbase wal -p output
> {code:java}
> //代码占位符
> writer Classes: ProtobufLogWriter AsyncProtobufLogWriter
> Cell Codec Class: org.apache.hadoop.hbase.regionserver.wal.WALCellCodec
> Sequence=15 , region=fee7a9465ced6ce9e319d37e9d71c63c at write timestamp=Wed
> Oct 24 09:22:49 CST 2018
> row=80000000, column=METAFAMILY:HBASE::REGION_EVENT
> value: \x08\x00\x12\x1Cmlaas:ump_host_second_181029\x1A
> fee7a9465ced6ce9e319d37e9d71c63c
> \x0E*\x06\x0A\x01f\x12\x01f2\x1F\x0A\x1311-3-19-10.JD.LOCAL\x10\x94}\x18\xCD\x9A\xAA\x9D\xEA,:Umlaas:ump_host_second_181029,80000000,1540271129253.fee7a9465ced6ce9e319d37e9d71c63c.
> Sequence=9 , region=ba6684888d826328a6373435124dc1cd at write timestamp=Wed
> Oct 24 09:22:49 CST 2018
> row=91000000, column=METAFAMILY:HBASE::REGION_EVENT
> ...
> row=34975#00, column=f:\x09,
> value:
> {"tp50":1,"avg":2,"min":0,"tp90":1,"max":3,"count":13,"tp99":2,"tp999":2,"error":0}
> row=349824#00, column=f:\x08\xFA
> value:
> {"tp50":2,"avg":2,"min":0,"tp90":2,"max":98,"count":957,"tp99":3,"tp999":34,"error":0}
> row=349824#00, column=f:\x08\xD2
> value:
> {"tp50":2,"avg":2,"min":0,"tp90":2,"max":43,"count":1842,"tp99":2,"tp999":31,"error":0}
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 8830
> at org.apache.hadoop.hbase.KeyValue.getFamilyLength(KeyValue.java:1365)
> at org.apache.hadoop.hbase.KeyValue.getFamilyLength(KeyValue.java:1358)
> at
> org.apache.hadoop.hbase.wal.WALPrettyPrinter.toStringMap(WALPrettyPrinter.java:336)
> at
> org.apache.hadoop.hbase.wal.WALPrettyPrinter.processFile(WALPrettyPrinter.java:290)
> at org.apache.hadoop.hbase.wal.WALPrettyPrinter.run(WALPrettyPrinter.java:421)
> at
> org.apache.hadoop.hbase.wal.WALPrettyPrinter.main(WALPrettyPrinter.java:356)
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
