[
https://issues.apache.org/jira/browse/HBASE-2077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12795248#action_12795248
]
Sam Pullara commented on HBASE-2077:
------------------------------------
jdcryans helped narrow it down to an issue in Leases where the lease would
polled from the queue at the same time it was being renewed. since there is no
sychronization protection there is a race condition. I am attaching a patch
that should fix the problem though it is very difficult to reproduce.
> NullPointException with an open scanner that expired causing an immediate
> region server shutdown
> ------------------------------------------------------------------------------------------------
>
> Key: HBASE-2077
> URL: https://issues.apache.org/jira/browse/HBASE-2077
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.20.2
> Environment: Hadoop 0.20.0, Mac OS X, Java 6
> Reporter: Sam Pullara
> Fix For: 0.20.3
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> 2009-12-29 18:05:55,432 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
> -4250070597157694417 lease expired
> 2009-12-29 18:05:55,443 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1310)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:136)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:127)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:117)
> at
> java.util.PriorityQueue.siftDownUsingComparator(PriorityQueue.java:641)
> at java.util.PriorityQueue.siftDown(PriorityQueue.java:612)
> at java.util.PriorityQueue.poll(PriorityQueue.java:523)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:113)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:1776)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1719)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1944)
> at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> 2009-12-29 18:05:55,446 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 7 on 55260, call next(-4250070597157694417, 10000) from
> 192.168.1.90:54011: error: java.io.IOException: java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:869)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:859)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1965)
> at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1310)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:136)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:127)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:117)
> at
> java.util.PriorityQueue.siftDownUsingComparator(PriorityQueue.java:641)
> at java.util.PriorityQueue.siftDown(PriorityQueue.java:612)
> at java.util.PriorityQueue.poll(PriorityQueue.java:523)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:113)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:1776)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1719)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1944)
> ... 5 more
> 2009-12-29 18:05:55,447 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
> Responder, call next(-4250070597157694417, 10000) from 192.168.1.90:54011:
> output error
> 2009-12-29 18:05:55,448 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 7 on 55260 caught: java.nio.channels.ClosedChannelException
> at
> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1125)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:615)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:679)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:943)
> 2009-12-29 18:05:56,322 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> server on 55260
> 2009-12-29 18:05:56,322 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC
> Server listener on 55260
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
