Hi Jaanai,
We are running Phoenix 4.13, and actually upon investigating the issue
more, we are not so sure if it is that row that's having issue. It seems
like the data issue might be on a different row than the one HBase
compaction is complaining about because when we do a point look up, the
data comes back, but it does not return if it is a range scan (#1). Not
totally sure why HBase complained about that row when the failure is
somewhere else...
I don't have the specific Phoenix DDL handy, but here is describe in HBase
(#2), and also selective info from SYSTEMS.CATALOGS (#3). Hopefully that's
helpful to make sense of the data types / keys, etc. Is there anything
specific that you want to see?
*#1: Range scan vs point look up*
0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> explain select "stts",
"cstId", "cltId", "pubId", "accId","cpgnId", "id" from "qa2".adgroups where
"cstId" = 2053276 and "cltId" = 1867077 and "id" = 30496531 and "pubId" = 4
and "accId" = 1904409 and "cpgnId" = 4811421;
+------------------------------------------------------------------------------------------------------+-----------------+----------------+----------------+
| PLAN
| EST_BYTES_READ | EST_ROWS_READ |
EST_INFO_TS |
+------------------------------------------------------------------------------------------------------+-----------------+----------------+----------------+
| CLIENT 1-CHUNK 1 ROWS 8052 BYTES PARALLEL 1-WAY ROUND ROBIN POINT LOOKUP
ON 1 KEY OVER qa2.ADGROUPS | 8052 | 1 |
1542827075237 |
+------------------------------------------------------------------------------------------------------+-----------------+----------------+----------------+
1 row selected (0.048 seconds)
0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> select "stts", "cstId",
"cltId", "pubId", "accId","cpgnId", "id" from "qa2".adgroups where "cstId"
= 2053276 and "cltId" = 1867077 and "id" = 30496531 and "pubId" = 4 and
"accId" = 1904409 and "cpgnId" = 4811421;
+---------+----------+----------+--------+----------+----------+-----------+
| stts | cstId | cltId | pubId | accId | cpgnId | id |
+---------+----------+----------+--------+----------+----------+-----------+
| ACTIVE | 2053276 | 1867077 | 4 | 1904409 | 4811421 | 30496531 |
+---------+----------+----------+--------+----------+----------+-----------+
1 row selected (0.072 seconds)
0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> explain select "stts",
"cstId", "cltId", "pubId", "accId","cpgnId", "id" from "qa2".adgroups where
"id" = 30496531;
+----------------------------------------------------------------------------------------------------------------------+-----------------+----------------+----------------+
| PLAN
| EST_BYTES_READ |
EST_ROWS_READ | EST_INFO_TS |
+----------------------------------------------------------------------------------------------------------------------+-----------------+----------------+----------------+
| CLIENT 364-CHUNK 36227938 ROWS 34037631025 BYTES PARALLEL 10-WAY ROUND
ROBIN RANGE SCAN OVER qa2.ADGROUPS [0] - [9] | 34037631025 | 36227938
| 1542823263107 |
| SERVER FILTER BY "id" = 30496531
| 34037631025 | 36227938
| 1542823263107 |
+----------------------------------------------------------------------------------------------------------------------+-----------------+----------------+----------------+
2 rows selected (0.063 seconds)
0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> select "stts", "cstId",
"cltId", "pubId", "accId","cpgnId", "id" from "qa2".adgroups where "id" =
30496531;
*[Hangs]*
#2
hbase(main):001:0> describe "qa2.ADGROUPS"
Table qa2.ADGROUPS is ENABLED
qa2.ADGROUPS, {TABLE_ATTRIBUTES => {coprocessor$1 =>
'|org.apache.phoenix.coprocessor.ScanRegionObserver|805306366|',
coprocessor$2 =>
'|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|805306366|',
coprocessor$
3 =>
'|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|805306366|',
coprocessor$4 =>
'|org.apache.phoenix.coprocessor.ServerCachingEndpointImpl|805306366|',
coprocessor$5 => '|org.apache.phoenix.hbase.index.Index
er|805306366|org.apache.hadoop.hbase.index.codec.class=org.apache.phoenix.index.PhoenixIndexCodec,index.builder=org.apache.phoenix.index.PhoenixIndexBuilder'}
COLUMN FAMILIES DESCRIPTION
{NAME => 'AG', BLOOMFILTER => 'NONE', VERSIONS => '1', IN_MEMORY =>
'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DIFF',
TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE =>
'true'
, BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}
1 row(s) in 0.4210 seconds
#3
0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> select KEY_SEQ,
ORDINAL_POSITION, COLUMN_QUALIFIER, DATA_TYPE from SYSTEM.CATALOG where
TABLE_SCHEM = 'qa2' and TABLE_NAME = 'ADGROUPS' order by ORDINAL_POSITION ;
+----------+-------------------+-------------------+------------+
| KEY_SEQ | ORDINAL_POSITION | COLUMN_QUALIFIER | DATA_TYPE |
+----------+-------------------+-------------------+------------+
| null | null | | null |
| null | null | | null |
| null | null | | null |
| null | null | | null |
| 1 | 1 | | -5 |
| 2 | 2 | | -5 |
| 3 | 3 | | 13 |
| 4 | 4 | | -5 |
| 5 | 5 | | -5 |
| 6 | 6 | | -5 |
| null | 7 | 800b | 12 |
| null | 8 | 800c | -5 |
| null | 9 | 800d | 2995 |
| null | 10 | 800e | -5 |
| null | 11 | 800f | 12 |
| null | 12 | 8010 | 1 |
| null | 13 | 8011 | 12 |
| null | 14 | 8012 | 12 |
| null | 15 | 8013 | 12 |
| null | 16 | 8014 | 12 |
| null | 17 | 8015 | 93 |
| null | 18 | 8016 | 93 |
| null | 19 | 8017 | 12 |
| null | 20 | 8018 | -5 |
| null | 21 | 8019 | 12 |
| null | 22 | 801a | 12 |
| null | 23 | 801b | -3 |
| null | 24 | 801c | 12 |
| null | 25 | 801d | 12 |
| null | 26 | 801e | 12 |
| null | 27 | 801f | 12 |
| null | 28 | 8020 | -5 |
| null | 29 | 8021 | 12 |
| null | 30 | 8022 | 12 |
| null | 31 | 8023 | -5 |
| null | 32 | 8024 | 16 |
| null | 33 | 8025 | 12 |
| null | 34 | 8026 | -3 |
| null | 35 | 8027 | 93 |
| null | 36 | 8028 | -5 |
| null | 37 | 8029 | 93 |
| null | 38 | 802a | -5 |
| null | 39 | 802b | 16 |
| null | 40 | 802c | 12 |
| null | 41 | 802d | 16 |
| null | 42 | 802e | 93 |
| null | 43 | 802f | 12 |
| null | 44 | 8030 | -5 |
| null | 45 | 8031 | 12 |
| null | 46 | 8032 | 4 |
| null | 47 | 8033 | 6 |
| null | 48 | 8034 | 12 |
| null | 49 | 8035 | 16 |
| null | 50 | 8036 | 3012 |
| null | 51 | 8037 | 93 |
| null | 52 | 8038 | 16 |
| null | 53 | 8039 | 4 |
| null | 54 | 803a | 12 |
| null | 55 | 803b | 12 |
| null | 56 | 803c | 12 |
| null | 57 | 803d | 3012 |
| null | 58 | 803e | -3 |
| null | 59 | 803f | 12 |
| null | 60 | 8040 | -5 |
| null | 61 | 8041 | 8 |
| null | 62 | 8042 | 8 |
| null | 63 | 8043 | 12 |
| null | 64 | 8044 | -5 |
| null | 65 | 8045 | 12 |
| null | 66 | 8046 | 4 |
| null | 67 | 8047 | 4 |
| null | 68 | 8048 | -5 |
| null | 69 | 8049 | -6 |
| null | 70 | 804a | 12 |
| null | 71 | 804b | 12 |
| null | 72 | 804c | 4 |
| null | 73 | 804d | -5 |
| null | 74 | 804e | -5 |
| null | 75 | 804f | 12 |
| null | 76 | 8050 | 12 |
| null | 77 | 8051 | 4 |
| null | 78 | 8052 | 16 |
| null | 79 | 8053 | -5 |
| null | 80 | 8054 | 12 |
| null | 81 | 8055 | 12 |
| null | 82 | 8056 | 12 |
| null | 83 | 8057 | 12 |
| null | 84 | 8058 | 12 |
+----------+-------------------+-------------------+------------+
Thanks
On Wed, Nov 21, 2018 at 1:23 AM Jaanai Zhang <cloud.pos...@gmail.com> wrote:
> Can you please attach Create the table/indexes SQL？ is this Phoneix's
> version? Are you sure many rows data had been corrupted or only this one
> row?
>
> ----------------------------------------
> Jaanai Zhang
> Best regards!
>
>
>
> William Shen <wills...@marinsoftware.com> 于2018年11月21日周三 上午10:20写道：
>
>> Hi there,
>>
>> We've encountered the following compaction failure(#1) for a Phoenix
>> table, and are not sure how to make sense of it. Using the HBase row key
>> from the error, we are able to query data directly from hbase shell, and by
>> examining the data, there arent anything immediately obvious about the data
>> as they seem to be stored consistent with the Phoenix data type (#2). When
>> querying Phoenix for the given row through sqline, the row would return if
>> only primary key columns are selected, and the query would not return if
>> non primary key columns are selected (#3).
>>
>> Few questions hoping to find some help on:
>>
>> a. Are we correct in understanding the error message to indicate an issue
>> with data for the row key (
>> \x05\x80\x00\x00\x00\x00\x1FT\x9C\x80\x00\x00\x00\x00\x1C}E\x00\x04\x80\x00\x00\x00\x00\x1D\x0F\x19\x80\x00\x00\x00\x00Ij\x9D\x80\x00\x00\x00\x01\xD1W\x13)?
>> We are not sure what to make sense of the string "
>> 1539019716378.3dcf2b1e057915feb74395d9711ba4ad." that is included with
>> the row key...
>>
>> b. What is out of bound here? It's not apparently clear here what
>> StatisticsScanner and FastDiffDeltaEncoder are tripping over...
>>
>> c. Is it normal for hbase shell to return some parts of the hex string as
>> ascii characters? We are seeing that in the row key as well as column name
>> encoding and value. We are not sure if that is causing any issues, or if
>> that was just a display issue that we can safely ignore.
>>
>> *#1 Compaction Failure*
>>
>> Compaction failed Request =
>> regionName=qa2.ADGROUPS,\x05\x80\x00\x00\x00\x00\x1FT\x9C\x80\x00\x00\x00\x00\x1C}E\x00\x04\x80\x00\x00\x00\x00\x1D\x0F\x19\x80\x00\x00\x00\x00Ij\x9D\x80\x00\x00\x00\x01\xD1W\x13,1539019716378.3dcf2b1e057915feb74395d9711ba4ad.,
>> storeName=AG, fileCount=4, fileSize=316.0 M (315.8 M, 188.7 K, 6.8 K, 14.2
>> K), priority=1, time=40613533856170784
>> java.lang.IndexOutOfBoundsException
>> at java.nio.Buffer.checkBounds(Buffer.java:567)
>> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:149)
>> at
>> org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decode(FastDiffDeltaEncoder.java:465)
>> at
>> org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decodeNext(FastDiffDeltaEncoder.java:516)
>> at
>> org.apache.hadoop.hbase.io.encoding.BufferedDataBlockEncoder$BufferedEncodedSeeker.next(BufferedDataBlockEncoder.java:618)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.next(HFileReaderV2.java:1277)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:180)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:108)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:588)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:458)
>> at
>> org.apache.phoenix.schema.stats.StatisticsScanner.next(StatisticsScanner.java:69)
>> at
>> org.apache.phoenix.schema.stats.StatisticsScanner.next(StatisticsScanner.java:76)
>> at
>> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:334)
>> at
>> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:106)
>> at
>> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:131)
>> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1245)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1852)
>> at
>> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:529)
>> at
>> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:566)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:748)
>>
>>
>> *#2 Data on the HBase level*
>>
>> hbase(main):002:0> get 'qa2.ADGROUPS',
>> "\x05\x80\x00\x00\x00\x00\x1FT\x9C\x80\x00\x00\x00\x00\x1C}E\x00\x04\x80\x00\x00\x00\x00\x1D\x0F\x19\x80\x00\x00\x00\x00Ij\x9D\x80\x00\x00\x00\x01\xD1W\x13"
>> COLUMN CELL
>> AG:\x00\x00\x00\x00 timestamp=1539019457506, value=x
>> AG:\x80\x0F timestamp=1539019457506,
>> value=D:USA_AK:Nightmute:1456903:hotel
>> AG:\x80\x12 timestamp=1539019457506, value=ACTIVE
>> AG:\x80\x13 timestamp=1539019457506, value=ADD
>> AG:\x80# timestamp=1539019457506,
>> value=\x80\x00\x00\x00\x00\x00'\x10
>> AG:\x80' timestamp=1539019457506,
>> value=\x80\x00\x01[\x97\x02\x02X\x00\x00\x00\x00
>> AG:\x80( timestamp=1539019457506,
>> value=\x80\x00\x00\x00\x00\x1D\xEC\xA4
>> AG:\x808 timestamp=1539019457506, value=\x00
>> 8 row(s) in 0.0510 seconds
>>
>>
>> *#3 Data in Phoenix via sqline*
>>
>> 0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> select "cstId", "cltId",
>> "pubId", "accId","cpgnId", "id" from "qa2".adgroups where "id" = 30496531;
>> +----------+----------+--------+----------+----------+-----------+
>> | cstId | cltId | pubId | accId | cpgnId | id |
>> +----------+----------+--------+----------+----------+-----------+
>> | 2053276 | 1867077 | 4 | 1904409 | 4811421 | 30496531 |
>> +----------+----------+--------+----------+----------+-----------+
>> 1 row selected (0.095 seconds)
>>
>> 0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> select "cstId", "cltId",
>> "pubId", "accId","cpgnId", "id", "stts" from "qa2".adgroups where "id" =
>> 30496531;
>>
>> *[hangs]*
>>
>>
>> Thanks in advance for your help!
>>
>
