Sorry for the delay (US Thanksgiving Holidays). We are on 1.2.0-cdh5.9.2 On Thu, Nov 22, 2018 at 6:54 PM Jaanai Zhang <cloud.pos...@gmail.com> wrote:
> It seems that is a bug about FastDiff in HBase, what' the version number > of your HBase? I saw a similar question: > http://mail-archives.apache.org/mod_mbox/hbase-user/201504.mbox/%3c44434b16-823d-43ef-aab1-337bfba6f...@5dlab.com%3E > > > ---------------------------------------- > Jaanai Zhang > Best regards! > > > > William Shen <wills...@marinsoftware.com> 于2018年11月22日周四 上午8:28写道： > >> Further narrowed down the HBase row, and was able to reproduce the >> Exception in HBase Shell. The error also shows up in the region server log. >> Would appreciate any pointers on next step to investigate the issue on the >> HBase level. Thank you! >> >> hbase(main):007:0> get 'qa2.ADGROUPS', >> "\x05\x80\x00\x00\x00\x00\x1F\x54\x9C\x80\x00\x00\x00\x00\x1C\x7D\x45\x00\x04\x80\x00\x00\x00\x00\x1D\x0F\x19\x80\x00\x00\x00\x00\x4A\x64\x6F\x80\x00\x00\x00\x01\xD9\xDB\xCE" >> COLUMN CELL >> >> ERROR: java.io.IOException >> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2215) >> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109) >> at >> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185) >> at >> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165) >> Caused by: java.lang.IndexOutOfBoundsException >> at java.nio.Buffer.checkBounds(Buffer.java:567) >> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:149) >> at >> org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decode(FastDiffDeltaEncoder.java:465) >> at >> org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decodeNext(FastDiffDeltaEncoder.java:516) >> at >> org.apache.hadoop.hbase.io.encoding.BufferedDataBlockEncoder$BufferedEncodedSeeker.next(BufferedDataBlockEncoder.java:618) >> at >> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.next(HFileReaderV2.java:1277) >> at >> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:180) >> at >> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:108) >> at >> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:588) >> at >> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:147) >> at >> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5706) >> at >> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:5865) >> at >> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5643) >> at >> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:5620) >> at >> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:5606) >> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6801) >> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6779) >> at >> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2029) >> at >> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644) >> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170) >> ... 3 more >> >> >> On Wed, Nov 21, 2018 at 11:19 AM William Shen <wills...@marinsoftware.com> >> wrote: >> > Hi Jaanai, >>> >>> We are running Phoenix 4.13, and actually upon investigating the issue >>> more, we are not so sure if it is that row that's having issue. It seems >>> like the data issue might be on a different row than the one HBase >>> compaction is complaining about because when we do a point look up, the >>> data comes back, but it does not return if it is a range scan (#1). Not >>> totally sure why HBase complained about that row when the failure is >>> somewhere else... >>> >>> I don't have the specific Phoenix DDL handy, but here is describe in >>> HBase (#2), and also selective info from SYSTEMS.CATALOGS (#3). Hopefully >>> that's helpful to make sense of the data types / keys, etc. Is there >>> anything specific that you want to see? >>> >>> >>> >>> *#1: Range scan vs point look up* >>> >>> 0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> explain select "stts", >>> "cstId", "cltId", "pubId", "accId","cpgnId", "id" from "qa2".adgroups where >>> "cstId" = 2053276 and "cltId" = 1867077 and "id" = 30496531 and "pubId" = 4 >>> and "accId" = 1904409 and "cpgnId" = 4811421; >>> >>> >>> +------------------------------------------------------------------------------------------------------+-----------------+----------------+----------------+ >>> >>> | PLAN >>> | EST_BYTES_READ | EST_ROWS_READ | >>> EST_INFO_TS | >>> >>> >>> +------------------------------------------------------------------------------------------------------+-----------------+----------------+----------------+ >>> >>> | CLIENT 1-CHUNK 1 ROWS 8052 BYTES PARALLEL 1-WAY ROUND ROBIN POINT >>> LOOKUP ON 1 KEY OVER qa2.ADGROUPS | 8052 | 1 | >>> 1542827075237 | >>> >>> >>> +------------------------------------------------------------------------------------------------------+-----------------+----------------+----------------+ >>> >>> 1 row selected (0.048 seconds) >>> >>> 0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> select "stts", "cstId", >>> "cltId", "pubId", "accId","cpgnId", "id" from "qa2".adgroups where "cstId" >>> = 2053276 and "cltId" = 1867077 and "id" = 30496531 and "pubId" = 4 and >>> "accId" = 1904409 and "cpgnId" = 4811421; >>> >>> >>> +---------+----------+----------+--------+----------+----------+-----------+ >>> >>> | stts | cstId | cltId | pubId | accId | cpgnId | id >>> | >>> >>> >>> +---------+----------+----------+--------+----------+----------+-----------+ >>> >>> | ACTIVE | 2053276 | 1867077 | 4 | 1904409 | 4811421 | >>> 30496531 | >>> >>> >>> +---------+----------+----------+--------+----------+----------+-----------+ >>> >>> 1 row selected (0.072 seconds) >>> >>> 0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> explain select "stts", >>> "cstId", "cltId", "pubId", "accId","cpgnId", "id" from "qa2".adgroups where >>> "id" = 30496531; >>> >>> >>> +----------------------------------------------------------------------------------------------------------------------+-----------------+----------------+----------------+ >>> >>> | PLAN >>> | EST_BYTES_READ | >>> EST_ROWS_READ | EST_INFO_TS | >>> >>> >>> +----------------------------------------------------------------------------------------------------------------------+-----------------+----------------+----------------+ >>> >>> | CLIENT 364-CHUNK 36227938 ROWS 34037631025 BYTES PARALLEL 10-WAY >>> ROUND ROBIN RANGE SCAN OVER qa2.ADGROUPS [0] - [9] | 34037631025 | >>> 36227938 >>> | 1542823263107 | >>> >>> | SERVER FILTER BY "id" = 30496531 >>> | 34037631025 | 36227938 >>> | 1542823263107 | >>> >>> >>> +----------------------------------------------------------------------------------------------------------------------+-----------------+----------------+----------------+ >>> >>> 2 rows selected (0.063 seconds) >>> >>> 0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> select "stts", "cstId", >>> "cltId", "pubId", "accId","cpgnId", "id" from "qa2".adgroups where "id" = >>> 30496531; >>> >>> *[Hangs]* >>> >>> #2 >>> >>> hbase(main):001:0> describe "qa2.ADGROUPS" >>> >>> Table qa2.ADGROUPS is ENABLED >>> >>> >>> >>> >>> qa2.ADGROUPS, {TABLE_ATTRIBUTES => {coprocessor$1 => >>> '|org.apache.phoenix.coprocessor.ScanRegionObserver|805306366|', >>> coprocessor$2 => >>> '|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|805306366|', >>> coprocessor$ >>> >>> 3 => >>> '|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|805306366|', >>> coprocessor$4 => >>> '|org.apache.phoenix.coprocessor.ServerCachingEndpointImpl|805306366|', >>> coprocessor$5 => '|org.apache.phoenix.hbase.index.Index >>> >>> er|805306366|org.apache.hadoop.hbase.index.codec.class=org.apache.phoenix.index.PhoenixIndexCodec,index.builder=org.apache.phoenix.index.PhoenixIndexBuilder'} >>> >>> >>> COLUMN FAMILIES DESCRIPTION >>> >>> >>> >>> >>> {NAME => 'AG', BLOOMFILTER => 'NONE', VERSIONS => '1', IN_MEMORY => >>> 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DIFF', >>> TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => >>> 'true' >>> >>> , BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} >>> >>> >>> >>> >>> 1 row(s) in 0.4210 seconds >>> >>> #3 >>> >> 0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> select KEY_SEQ, >>> ORDINAL_POSITION, COLUMN_QUALIFIER, DATA_TYPE from SYSTEM.CATALOG >>> where TABLE_SCHEM = 'qa2' and TABLE_NAME = 'ADGROUPS' order by >>> ORDINAL_POSITION ; >>> >>> +----------+-------------------+-------------------+------------+ >>> >>> | KEY_SEQ | ORDINAL_POSITION | COLUMN_QUALIFIER | DATA_TYPE | >>> >>> +----------+-------------------+-------------------+------------+ >>> >>> | null | null | | null | >>> >>> | null | null | | null | >>> >>> | null | null | | null | >>> >>> | null | null | | null | >>> >>> | 1 | 1 | | -5 | >>> >>> | 2 | 2 | | -5 | >>> >>> | 3 | 3 | | 13 | >>> >>> | 4 | 4 | | -5 | >>> >>> | 5 | 5 | | -5 | >>> >>> | 6 | 6 | | -5 | >>> >>> | null | 7 | 800b | 12 | >>> >>> | null | 8 | 800c | -5 | >>> >>> | null | 9 | 800d | 2995 | >>> >>> | null | 10 | 800e | -5 | >>> >>> | null | 11 | 800f | 12 | >>> >>> | null | 12 | 8010 | 1 | >>> >>> | null | 13 | 8011 | 12 | >>> >>> | null | 14 | 8012 | 12 | >>> >>> | null | 15 | 8013 | 12 | >>> >>> | null | 16 | 8014 | 12 | >>> >>> | null | 17 | 8015 | 93 | >>> >>> | null | 18 | 8016 | 93 | >>> >>> | null | 19 | 8017 | 12 | >>> >>> | null | 20 | 8018 | -5 | >>> >>> | null | 21 | 8019 | 12 | >>> >>> | null | 22 | 801a | 12 | >>> >>> | null | 23 | 801b | -3 | >>> >>> | null | 24 | 801c | 12 | >>> >>> | null | 25 | 801d | 12 | >>> >>> | null | 26 | 801e | 12 | >>> >>> | null | 27 | 801f | 12 | >>> >>> | null | 28 | 8020 | -5 | >>> >>> | null | 29 | 8021 | 12 | >>> >>> | null | 30 | 8022 | 12 | >>> >>> | null | 31 | 8023 | -5 | >>> >>> | null | 32 | 8024 | 16 | >>> >>> | null | 33 | 8025 | 12 | >>> >>> | null | 34 | 8026 | -3 | >>> >>> | null | 35 | 8027 | 93 | >>> >>> | null | 36 | 8028 | -5 | >>> >>> | null | 37 | 8029 | 93 | >>> >>> | null | 38 | 802a | -5 | >>> >>> | null | 39 | 802b | 16 | >>> >>> | null | 40 | 802c | 12 | >>> >>> | null | 41 | 802d | 16 | >>> >> | null | 42 <span >>> class="m_-4528002918779368419m_1004149540963671125m_-6947968 >>> >>
