[
https://issues.apache.org/jira/browse/HBASE-20280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16412460#comment-16412460
]
Zach York commented on HBASE-20280:
-----------------------------------
[~apurtell] Could you make sure I am understanding the prefetch logic correctly
(especially around canceling)? Trying to understand the code, it looks like
when a ClosedByInterruptException is thrown it first sets the thread to
interrupted then throws the exception.
>From
>[https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java#L270]
> it looks like after the exception is thrown (and caught by BucketCache) this
>should break out of the loop. However, I saw two exceptions being thrown back
>to back by apparently the same thread (by name at least) which would indicate
>that this didn't actually break out of the loop. Do you have any idea what is
>going on here:
2018-03-17 12:03:31,986 ERROR [hfile-prefetch-1521244956862]
bucket.BucketCache: Failed reading block
f1512d0ed56546939771d8e7f5e72c6a_129839 from bucket cache
java.nio.channels.ClosedByInterruptException at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:746) at
sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:727) at
org.apache.hadoop.hbase.io.hfile.bucket.FileIOEngine$FileReadAccessor.access(FileIOEngine.java:276)
at
org.apache.hadoop.hbase.io.hfile.bucket.FileIOEngine.accessFile(FileIOEngine.java:196)
at
org.apache.hadoop.hbase.io.hfile.bucket.FileIOEngine.read(FileIOEngine.java:113)
at
org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getBlock(BucketCache.java:506)
at
org.apache.hadoop.hbase.io.hfile.CombinedBlockCache.getBlock(CombinedBlockCache.java:84)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2.getCachedBlock(HFileReaderV2.java:279)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:420)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2$1.run(HFileReaderV2.java:209) at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at
java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) 2018-03-17 12:03:31,986 ERROR
[hfile-prefetch-1521244956862] bucket.BucketCache: Failed reading block
f1512d0ed56546939771d8e7f5e72c6a_129839 from bucket cache
java.nio.channels.ClosedByInterruptException at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:746) at
sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:727) at
org.apache.hadoop.hbase.io.hfile.bucket.FileIOEngine$FileReadAccessor.access(FileIOEngine.java:276)
at
org.apache.hadoop.hbase.io.hfile.bucket.FileIOEngine.accessFile(FileIOEngine.java:196)
at
org.apache.hadoop.hbase.io.hfile.bucket.FileIOEngine.read(FileIOEngine.java:113)
at
org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getBlock(BucketCache.java:506)
at
org.apache.hadoop.hbase.io.hfile.CombinedBlockCache.getBlock(CombinedBlockCache.java:84)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2.getCachedBlock(HFileReaderV2.java:279)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:420)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2$1.run(HFileReaderV2.java:209) at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at
java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
> Fix possibility of deadlocking in refreshFileConnections when prefetch is
> enabled
> ---------------------------------------------------------------------------------
>
> Key: HBASE-20280
> URL: https://issues.apache.org/jira/browse/HBASE-20280
> Project: HBase
> Issue Type: Bug
> Components: BucketCache
> Affects Versions: 2.0.0-beta-2, 1.4.2
> Reporter: Zach York
> Assignee: Zach York
> Priority: Major
> Attachments: HBASE-20280.master.001.patch
>
>
> When prefetch on open is specified, there is a deadlocking case
> where if the prefetch is cancelled, the PrefetchExecutor interrupts
> the threads if necessary, when that happens in FileIOEngine, it
> causes an ClosedByInterruptException which is a subclass of
> ClosedChannelException. If we retry all ClosedChannelExceptions,
> this will lock as this access is expected to be interrupted.
> This change removes calling refreshFileConnections for
> ClosedByInterruptExceptions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
