Exception in HBase get

[Graceline Abigail Prem Kumar]

Hi

We have got an IllegalStateException during HBase get operation. And we're
not sure about the cause. Here is the exception trace. What could the
problem be?

Caused by: java.lang.IllegalStateException: isDelete failed:
deleteBuffer=22, qualifier=21, timestamp=1487055525513, comparison result:
1
at
org.apache.hadoop.hbase.regionserver.ScanDeleteTracker.isDeleted(ScanDeleteTracker.java:147)
at
org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.match(ScanQueryMatcher.java:395)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:529)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:150)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5731)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:5894)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5668)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:5645)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:5631)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6829)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6807)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2049)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2188)
... 4 more

Mon Jul 03 04:34:07 PDT 2017,
RpcRetryingCaller{globalStartTime=1499081647183, pause=100, retries=35},
java.io.IOException: java.io.IOException: isDelete failed: deleteBuffer=22,
qualifier=21, timestamp=1487055525513, comparison result: 1
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2239)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
at java.lang.Thread.run(Thread.java:745)

Regards,
Graceline Abigail P

Re: Problem during hbase read

[mukund murrali]

Hi Stack

We are currently using hbase-1.2.5(Stable). I was not able to read the file
using HFile tool. It throws the same error. But I downloaded the file using
dfs commands. The file had very less data. Ya I saw that comment. More
details on the issue and fix will be helpful. Adding him to this thread.

Regards,

Mukund Murrali

On Wed, Jul 5, 2017 at 8:30 PM, Stack  wrote:

> (What version of hbase?)
>
> Unfortunately the exception does not print the file name. Can you figure
> which file or region it is? Could try reading content w/ hfile tool to see
> if it complains also.
>
> Problem is:
>
> On-disk size without header provided is
> 36901, but block header contains 859321901. Block offset: 440164791, data
> starts with: 1-9d66-4386-83bc-48b335630d0e\x88\x01\x01
> at org.apache.hadoop.hbase.io.hfile.HFileBlock.
> validateOnDiskSizeWithoutHeader(HFileBlock.java:521)
>
> I see the above exception in HBASE-13830 where Nils perhaps fixes it by
> upgrading hbase (see end comment).
>
> St.Ack
>
>
>
> On Wed, Jul 5, 2017 at 1:00 AM, mukund murrali 
> wrote:
>
> > Hi,
> >
> > Recently we faced an issue in our HBase Cluster and following is the
> > exception trace in client. Any idea what could be the cause and how to
> > rectify this?
> >
> > RpcRetryingCaller{globalStartTime=1499165478610, pause=100, retries=35},
> > java.io.IOException: java.io.IOException: Could not reseek
> > StoreFileScanner[org.apache.hadoop.hbase.io.
> HalfStoreFileReader$1@2e508d2b
> > ,
> > cur=xxx/dataCF:5f1e00cd-cbea-45d6-8542-72b8b00960ff/
> > 149888597/Maximum/vlen=0/seqid=0] to key
> xxx/dataCF:5f1e00cd-cbea-
> > 45d6-8542-72b8b00960ff/LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0
> > at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> > reseek(StoreFileScanner.java:217)
> > at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> > enforceSeek(StoreFileScanner.java:408)
> > at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
> > pollRealKV(KeyValueHeap.java:363)
> > at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
> > next(KeyValueHeap.java:123)
> > at org.apache.hadoop.hbase.regionserver.StoreScanner.
> > next(StoreScanner.java:588)
> > at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
> > next(KeyValueHeap.java:150)
> > at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
> > populateResult(HRegion.java:5731)
> > at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
> > nextInternal(HRegion.java:5894)
> > at org.apache.hadoop.hbase.regionserver.HRegion$
> > RegionScannerImpl.nextRaw(HRegion.java:5668)
> > at org.apache.hadoop.hbase.regionserver.HRegion$
> > RegionScannerImpl.next(
> > HRegion.java:5645)
> > at org.apache.hadoop.hbase.regionserver.HRegion$
> > RegionScannerImpl.next(
> > HRegion.java:5631)
> > at org.apache.hadoop.hbase.regionserver.HRegion.get(
> HRegion.java:6829)
> > at org.apache.hadoop.hbase.regionserver.HRegion.get(
> HRegion.java:6807)
> > at org.apache.hadoop.hbase.regionserver.RSRpcServices.
> > get(RSRpcServices.java:2049)
> > at org.apache.hadoop.hbase.protobuf.generated.
> > ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644)
> > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2188)
> > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> > at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(
> > RpcExecutor.java:133)
> > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.
> java:108)
> > at java.lang.Thread.run(Thread.java:745)
> > Caused by: java.io.IOException: On-disk size without header provided is
> > 36901, but block header contains 859321901. Block offset: 440164791, data
> > starts with: 1-9d66-4386-83bc-48b335630d0e\x88\x01\x01
> > at org.apache.hadoop.hbase.io.hfile.HFileBlock.
> > validateOnDiskSizeWithoutHeader(HFileBlock.java:521)
> > at org.apache.hadoop.hbase.io.hfile.HFileBlock.access$700(
> > HFileBlock.java:88)
> > at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.
> > readBlockDataInternal(HFileBlock.java:1665)
> > at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.
> > readBlockData(HFileBlock.java:1532)
> > at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(
> > HFileReaderV2.java:452)
> > at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$
> BlockIndexReader.
> > loadDataBlockWithScanInfo(HFileBlockIndex.java:271)
> > at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$
> > AbstractScannerV2.seekTo(HFileReaderV2.java:649)
> > at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$
> > AbstractScannerV2.reseekTo(HFileReaderV2.java:629)
> > at org.apache.hadoop.hbase.io.HalfStoreFileReader$1.
> > reseekTo(HalfStoreFileReader.java:293)
> > at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> > reseekAtOrAfter(StoreFileScanner.java:288)
> > at 

RE: [ANNOUNCE] Devaraj Das joins the Apache HBase PMC

[Pankaj kr]

Congratulations Devaraj..!! 

Thanks & Regards,
Pankaj

HUAWEI TECHNOLOGIES CO.LTD.
Huawei Tecnologies India Pvt. Ltd.
Near EPIP Industrial Area, Kundalahalli Village
Whitefield, Bangalore-560066
www.huawei.com
-
This e-mail and its attachments contain confidential information from HUAWEI, 
which 
is intended only for the person or entity whose address is listed above. Any 
use of the 
information contained herein in any way (including, but not limited to, total 
or partial 
disclosure, reproduction, or dissemination) by persons other than the intended 
recipient(s) is prohibited. If you receive this e-mail in error, please notify 
the sender by 
phone or email immediately and delete it!


-Original Message-
From: Josh Elser [mailto:els...@apache.org] 
Sent: Thursday, July 06, 2017 12:27 AM
To: d...@hbase.apache.org; user@hbase.apache.org
Subject: [ANNOUNCE] Devaraj Das joins the Apache HBase PMC

I'm pleased to announce yet another PMC addition in the form of Devaraj Das. 
One of the "old guard" in the broader Hadoop umbrella, he's also a 
long-standing member in our community. We all look forward to the continued 
contributions and project leadership.

Please join me in welcoming Devaraj!

- Josh (on behalf of the PMC)

Re: NPE in ReplicationZKLockCleanerChore

[Ted Yu]

Which hbase release are you using (so that I can find the exact location
where NPE came from) ?

Do you see NPE repeatedly ?
If so, replication lock cleaning would not make progress.

On Wed, Jul 5, 2017 at 5:29 PM, Kang Minwoo  wrote:

> Hello, HBase Users.
>
> While I am watching HMaster log, I found NullPointerException Logs.
>
> 
>
> 2017-07-06 09:05:02,579 DEBUG [,1498445640728_ChoreService_1]
> cleaner.CleanerChore: Removing: hdfs://*** from archive
> 2017-07-06 09:05:02,585 ERROR [,1498445640728_ChoreService_2]
> hbase.ScheduledChore: Caught error
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.master.cleaner.
> ReplicationZKLockCleanerChore.chore(ReplicationZKLockCleanerChore.java:80)
> at org.apache.hadoop.hbase.ScheduledChore.run(
> ScheduledChore.java:185)
> at java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(
> FutureTask.java:304)
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$
> JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecu
> torImpl.java:110)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 2017-07-06 09:05:02,585 DEBUG [,1498445640728_ChoreService_1]
> cleaner.CleanerChore: Removing: hdfs://*** from archive
> 2017-07-06 09:05:02,586 DEBUG [,1498445640728_ChoreService_1]
> cleaner.CleanerChore: Removing: hdfs://*** from archive
>
> 
>
> I wonder what side effects this error gives HBase.
>
> Best regards,
> Minwoo Kang
>

NPE in ReplicationZKLockCleanerChore

[Kang Minwoo]

Hello, HBase Users.

While I am watching HMaster log, I found NullPointerException Logs.



2017-07-06 09:05:02,579 DEBUG [,1498445640728_ChoreService_1] 
cleaner.CleanerChore: Removing: hdfs://*** from archive
2017-07-06 09:05:02,585 ERROR [,1498445640728_ChoreService_2] 
hbase.ScheduledChore: Caught error
java.lang.NullPointerException
at 
org.apache.hadoop.hbase.master.cleaner.ReplicationZKLockCleanerChore.chore(ReplicationZKLockCleanerChore.java:80)
at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:185)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at 
org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:110)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2017-07-06 09:05:02,585 DEBUG [,1498445640728_ChoreService_1] 
cleaner.CleanerChore: Removing: hdfs://*** from archive
2017-07-06 09:05:02,586 DEBUG [,1498445640728_ChoreService_1] 
cleaner.CleanerChore: Removing: hdfs://*** from archive



I wonder what side effects this error gives HBase.

Best regards,
Minwoo Kang

Re: [ANNOUNCE] Devaraj Das joins the Apache HBase PMC

[Jerry He]

Congrats, Devaraj !


Thanks,

Jerry

On Wed, Jul 5, 2017 at 1:36 PM, Nick Dimiduk  wrote:
> Congratulations Devaraj!
>
> On Wed, Jul 5, 2017 at 9:27 AM, Josh Elser  wrote:
>
>> I'm pleased to announce yet another PMC addition in the form of Devaraj
>> Das. One of the "old guard" in the broader Hadoop umbrella, he's also a
>> long-standing member in our community. We all look forward to the continued
>> contributions and project leadership.
>>
>> Please join me in welcoming Devaraj!
>>
>> - Josh (on behalf of the PMC)
>>

Re: [ANNOUNCE] Devaraj Das joins the Apache HBase PMC

[Nick Dimiduk]

Congratulations Devaraj!

On Wed, Jul 5, 2017 at 9:27 AM, Josh Elser  wrote:

> I'm pleased to announce yet another PMC addition in the form of Devaraj
> Das. One of the "old guard" in the broader Hadoop umbrella, he's also a
> long-standing member in our community. We all look forward to the continued
> contributions and project leadership.
>
> Please join me in welcoming Devaraj!
>
> - Josh (on behalf of the PMC)
>

Re: [ANNOUNCE] Chunhui Shen joins the Apache HBase PMC

[Stephen Jiang]

Congratulations!

Thanks
Stephen

On Wed, Jul 5, 2017 at 9:35 AM, Ashish Singhi <
ashish.singhi.apa...@gmail.com> wrote:

> Congratulations!
>
> Sent from my iPhone
>
> > On 04-Jul-2017, at 10:54 AM, Yu Li  wrote:
> >
> > On behalf of the Apache HBase PMC I am pleased to announce that Chunhui
> Shen
> > has accepted our invitation to become a PMC member on the Apache
> > HBase project. He has been an active contributor to HBase for past many
> > years. Looking forward for many more contributions from him.
> >
> > Please join me in welcoming Chunhui to the HBase PMC!
> >
> > Best Regards,
> > Yu
>

Re: [ANNOUNCE] Devaraj Das joins the Apache HBase PMC

[Andrew Purtell]

Congratulations, Devaraj!

On Wed, Jul 5, 2017 at 9:27 AM, Josh Elser  wrote:

> I'm pleased to announce yet another PMC addition in the form of Devaraj
> Das. One of the "old guard" in the broader Hadoop umbrella, he's also a
> long-standing member in our community. We all look forward to the continued
> contributions and project leadership.
>
> Please join me in welcoming Devaraj!
>
> - Josh (on behalf of the PMC)
>



-- 
Best regards,
Andrew

Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
   - A23, Crosstalk

Re: [ANNOUNCE] Chunhui Shen joins the Apache HBase PMC

[Ashish Singhi]

Congratulations!

Sent from my iPhone

> On 04-Jul-2017, at 10:54 AM, Yu Li  wrote:
> 
> On behalf of the Apache HBase PMC I am pleased to announce that Chunhui Shen
> has accepted our invitation to become a PMC member on the Apache
> HBase project. He has been an active contributor to HBase for past many
> years. Looking forward for many more contributions from him.
> 
> Please join me in welcoming Chunhui to the HBase PMC!
> 
> Best Regards,
> Yu

Re: [ANNOUNCE] Devaraj Das joins the Apache HBase PMC

[Ashish Singhi]

Congratulations!

Sent from my iPhone

> On 05-Jul-2017, at 9:57 PM, Josh Elser  wrote:
> 
> I'm pleased to announce yet another PMC addition in the form of Devaraj Das. 
> One of the "old guard" in the broader Hadoop umbrella, he's also a 
> long-standing member in our community. We all look forward to the continued 
> contributions and project leadership.
> 
> Please join me in welcoming Devaraj!
> 
> - Josh (on behalf of the PMC)

Re: [ANNOUNCE] Devaraj Das joins the Apache HBase PMC

[Ted Yu]

Congratulations, Devaraj !

On Wed, Jul 5, 2017 at 9:27 AM, Josh Elser  wrote:

> I'm pleased to announce yet another PMC addition in the form of Devaraj
> Das. One of the "old guard" in the broader Hadoop umbrella, he's also a
> long-standing member in our community. We all look forward to the continued
> contributions and project leadership.
>
> Please join me in welcoming Devaraj!
>
> - Josh (on behalf of the PMC)
>

[ANNOUNCE] Devaraj Das joins the Apache HBase PMC

[Josh Elser]

I'm pleased to announce yet another PMC addition in the form of Devaraj 
Das. One of the "old guard" in the broader Hadoop umbrella, he's also a 
long-standing member in our community. We all look forward to the 
continued contributions and project leadership.


Please join me in welcoming Devaraj!

- Josh (on behalf of the PMC)

Re: Problem during hbase read

[Stack]

(What version of hbase?)

Unfortunately the exception does not print the file name. Can you figure
which file or region it is? Could try reading content w/ hfile tool to see
if it complains also.

Problem is:

On-disk size without header provided is
36901, but block header contains 859321901. Block offset: 440164791, data
starts with: 1-9d66-4386-83bc-48b335630d0e\x88\x01\x01
at org.apache.hadoop.hbase.io.hfile.HFileBlock.
validateOnDiskSizeWithoutHeader(HFileBlock.java:521)

I see the above exception in HBASE-13830 where Nils perhaps fixes it by
upgrading hbase (see end comment).

St.Ack



On Wed, Jul 5, 2017 at 1:00 AM, mukund murrali 
wrote:

> Hi,
>
> Recently we faced an issue in our HBase Cluster and following is the
> exception trace in client. Any idea what could be the cause and how to
> rectify this?
>
> RpcRetryingCaller{globalStartTime=1499165478610, pause=100, retries=35},
> java.io.IOException: java.io.IOException: Could not reseek
> StoreFileScanner[org.apache.hadoop.hbase.io.HalfStoreFileReader$1@2e508d2b
> ,
> cur=xxx/dataCF:5f1e00cd-cbea-45d6-8542-72b8b00960ff/
> 149888597/Maximum/vlen=0/seqid=0] to key xxx/dataCF:5f1e00cd-cbea-
> 45d6-8542-72b8b00960ff/LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> reseek(StoreFileScanner.java:217)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> enforceSeek(StoreFileScanner.java:408)
> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
> pollRealKV(KeyValueHeap.java:363)
> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
> next(KeyValueHeap.java:123)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.
> next(StoreScanner.java:588)
> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
> next(KeyValueHeap.java:150)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
> populateResult(HRegion.java:5731)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
> nextInternal(HRegion.java:5894)
> at org.apache.hadoop.hbase.regionserver.HRegion$
> RegionScannerImpl.nextRaw(HRegion.java:5668)
> at org.apache.hadoop.hbase.regionserver.HRegion$
> RegionScannerImpl.next(
> HRegion.java:5645)
> at org.apache.hadoop.hbase.regionserver.HRegion$
> RegionScannerImpl.next(
> HRegion.java:5631)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6829)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6807)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.
> get(RSRpcServices.java:2049)
> at org.apache.hadoop.hbase.protobuf.generated.
> ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2188)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(
> RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: On-disk size without header provided is
> 36901, but block header contains 859321901. Block offset: 440164791, data
> starts with: 1-9d66-4386-83bc-48b335630d0e\x88\x01\x01
> at org.apache.hadoop.hbase.io.hfile.HFileBlock.
> validateOnDiskSizeWithoutHeader(HFileBlock.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileBlock.access$700(
> HFileBlock.java:88)
> at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.
> readBlockDataInternal(HFileBlock.java:1665)
> at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.
> readBlockData(HFileBlock.java:1532)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(
> HFileReaderV2.java:452)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.
> loadDataBlockWithScanInfo(HFileBlockIndex.java:271)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$
> AbstractScannerV2.seekTo(HFileReaderV2.java:649)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$
> AbstractScannerV2.reseekTo(HFileReaderV2.java:629)
> at org.apache.hadoop.hbase.io.HalfStoreFileReader$1.
> reseekTo(HalfStoreFileReader.java:293)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> reseekAtOrAfter(StoreFileScanner.java:288)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> reseek(StoreFileScanner.java:200)
> ... 19 more
>
> ThanksU
> Regards,
>
> Mukund Murrali
>

Problem during hbase read

[mukund murrali]

Hi,

Recently we faced an issue in our HBase Cluster and following is the
exception trace in client. Any idea what could be the cause and how to
rectify this?

RpcRetryingCaller{globalStartTime=1499165478610, pause=100, retries=35},
java.io.IOException: java.io.IOException: Could not reseek
StoreFileScanner[org.apache.hadoop.hbase.io.HalfStoreFileReader$1@2e508d2b,
cur=xxx/dataCF:5f1e00cd-cbea-45d6-8542-72b8b00960ff/
149888597/Maximum/vlen=0/seqid=0] to key xxx/dataCF:5f1e00cd-cbea-
45d6-8542-72b8b00960ff/LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
reseek(StoreFileScanner.java:217)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
enforceSeek(StoreFileScanner.java:408)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
pollRealKV(KeyValueHeap.java:363)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
next(KeyValueHeap.java:123)
at org.apache.hadoop.hbase.regionserver.StoreScanner.
next(StoreScanner.java:588)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
next(KeyValueHeap.java:150)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
populateResult(HRegion.java:5731)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
nextInternal(HRegion.java:5894)
at org.apache.hadoop.hbase.regionserver.HRegion$
RegionScannerImpl.nextRaw(HRegion.java:5668)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(
HRegion.java:5645)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(
HRegion.java:5631)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6829)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6807)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.
get(RSRpcServices.java:2049)
at org.apache.hadoop.hbase.protobuf.generated.
ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2188)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(
RpcExecutor.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: On-disk size without header provided is
36901, but block header contains 859321901. Block offset: 440164791, data
starts with: 1-9d66-4386-83bc-48b335630d0e\x88\x01\x01
at org.apache.hadoop.hbase.io.hfile.HFileBlock.
validateOnDiskSizeWithoutHeader(HFileBlock.java:521)
at org.apache.hadoop.hbase.io.hfile.HFileBlock.access$700(
HFileBlock.java:88)
at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.
readBlockDataInternal(HFileBlock.java:1665)
at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.
readBlockData(HFileBlock.java:1532)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(
HFileReaderV2.java:452)
at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.
loadDataBlockWithScanInfo(HFileBlockIndex.java:271)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$
AbstractScannerV2.seekTo(HFileReaderV2.java:649)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$
AbstractScannerV2.reseekTo(HFileReaderV2.java:629)
at org.apache.hadoop.hbase.io.HalfStoreFileReader$1.
reseekTo(HalfStoreFileReader.java:293)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
reseekAtOrAfter(StoreFileScanner.java:288)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
reseek(StoreFileScanner.java:200)
... 19 more

ThanksU
Regards,

Mukund Murrali

Re: [ANNOUNCE] Chunhui Shen joins the Apache HBase PMC

[Phil Yang]

Congratulations!

Thanks,
Phil


2017-07-05 13:05 GMT+08:00 ramkrishna vasudevan <
ramkrishna.s.vasude...@gmail.com>:

> Congratulations !!!
>
> On Wed, Jul 5, 2017 at 10:33 AM, QI Congyun  cn
> > wrote:
>
> > Congratulations, Chunhui.
> >
> > -Original Message-
> > From: Jerry He [mailto:jerry...@gmail.com]
> > Sent: Wednesday, July 05, 2017 1:02 PM
> > To: d...@hbase.apache.org; user@hbase.apache.org
> > Subject: Re: [ANNOUNCE] Chunhui Shen joins the Apache HBase PMC
> >
> > Congrats,  Chunhui!
> >
> > Thanks
> >
> > Jerry
> >
> > On Tue, Jul 4, 2017 at 8:37 PM Anoop John  wrote:
> >
> > > Congrats Chunhui..
> > >
> > > On Wed, Jul 5, 2017 at 6:55 AM, Pankaj kr 
> wrote:
> > > > Congratulations Chunhui..!!
> > > >
> > > > Regards,
> > > > Pankaj
> > > >
> > > >
> > > > -Original Message-
> > > > From: Yu Li [mailto:car...@gmail.com]
> > > > Sent: Tuesday, July 04, 2017 1:24 PM
> > > > To: d...@hbase.apache.org; Hbase-User
> > > > Subject: [ANNOUNCE] Chunhui Shen joins the Apache HBase PMC
> > > >
> > > > On behalf of the Apache HBase PMC I am pleased to announce that
> > > > Chunhui
> > > Shen has accepted our invitation to become a PMC member on the Apache
> > > HBase project. He has been an active contributor to HBase for past many
> > years.
> > > Looking forward for many more contributions from him.
> > > >
> > > > Please join me in welcoming Chunhui to the HBase PMC!
> > > >
> > > > Best Regards,
> > > > Yu
> > >
> >
>