[
https://issues.apache.org/jira/browse/HBASE-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16726398#comment-16726398
]
Zheng Hu commented on HBASE-21401:
----------------------------------
bq. I don't like that we are adding friction to KV making because it will
impinge on perf.
Yeah, I understand you consideration. Our HBASE-21459 indicated that there's
extremely small performance loss for diff kinds of keyvalue, so I think should
be OK.
bq. What happens if we do not commit this? For blocks coming from hdfs (hfile
and/or WAL), they should be checksummed so read should fail if bits are
corrupted between writing and re-reading. Is the reading from socket when we
read from client? That seems like a good time to do this extensive check.
The sanity check isn't designed for discovering the bits corruption in network
transferring or disk IO. It is designed to detect bugs inside HBase in
advance. Let see an example:
1. client mis-encoded a KeyValue in some rare case;
2. client made a checksum and sent to RS;
3. RS checked the checksum (checksum was OK), and wrapped the byte[] as cell
(KeyValue constructioin)
4. RS tried to get a family but found that the offset overflow.....
The problem is, we cannot ensure bug was caused by RegionServer's KeyValue
encoding or mis-encoding by client ? If we had the sanity check, in step.3 ,
we would got a exception, it mis-encoded definitly by client .
The HFile reading or WAL reading (replication) will also benefit from the
sanity check too.
Thanks.
> Sanity check when constructing the KeyValue
> -------------------------------------------
>
> Key: HBASE-21401
> URL: https://issues.apache.org/jira/browse/HBASE-21401
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
> Reporter: Zheng Hu
> Assignee: Zheng Hu
> Priority: Critical
> Fix For: 3.0.0, 2.2.0, 2.1.3, 2.0.5
>
> Attachments: HBASE-21401.v1.patch, HBASE-21401.v2.patch,
> HBASE-21401.v3.patch, HBASE-21401.v4.patch, HBASE-21401.v4.patch,
> HBASE-21401.v5.patch, HBASE-21401.v6.patch, HBASE-21401.v7.patch
>
>
> In KeyValueDecoder & ByteBuffKeyValueDecoder, we pass a byte buffer to
> initialize the Cell without a sanity check (check each field's offset&len
> exceed the byte buffer or not), so ArrayIndexOutOfBoundsException may happen
> when read the cell's fields, such as HBASE-21379, it's hard to debug this
> kind of bug.
> An earlier check will help to find such kind of bugs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
