[
https://issues.apache.org/jira/browse/HBASE-27511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17640864#comment-17640864
]
Wellington Chevreuil commented on HBASE-27511:
----------------------------------------------
Thanks, [~zhangduo] . Yeah, I noticed this just happens in the case of preads
and specifically to S3, but I wonder if this could be handled at
StoreFileScanner by always creating a new StoreFileReader just like we do with
stream reads. Or would you think that would break pread?
> Lock contention when doing multiple parallel preads due to StoreFileReader
> reuse
> --------------------------------------------------------------------------------
>
> Key: HBASE-27511
> URL: https://issues.apache.org/jira/browse/HBASE-27511
> Project: HBase
> Issue Type: Bug
> Reporter: Wellington Chevreuil
> Assignee: Wellington Chevreuil
> Priority: Major
> Attachments: rs-stack-lock-contention
>
>
> In HStoreFile, we reuse the StoreFileReader created during HStoreFile
> initialization when creating a StoreFileScanner for preads:
> [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStoreFile.java#L545]
> When using S3 as hbase storage, we noticed this caused lock contention when
> multiple clients were doing preads in parallel:
> {noformat}
> ...
> "RpcServer.default.FPBQ.Fifo.handler=38,queue=8,port=16020" #125 daemon
> prio=5 os_prio=0 tid=0x00007fc11d83c000 nid=0x73f2 waiting for monitor entry
> [0x00007fc1154e6000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:73)
> - waiting to lock <0x00007fc4c2769660> (a
> org.apache.hadoop.fs.s3a.S3AInputStream)
> ...
> "RpcServer.default.FPBQ.Fifo.handler=37,queue=7,port=16020" #124 daemon
> prio=5 os_prio=0 tid=0x00007fc11d83a000 nid=0x73f1 waiting for monitor entry
> [0x00007fc1155e7000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:73)
> - waiting to lock <0x00007fc4c2769660> (a
> org.apache.hadoop.fs.s3a.S3AInputStream)
> ...
> "RpcServer.default.FPBQ.Fifo.handler=36,queue=6,port=16020" #123 daemon
> prio=5 os_prio=0 tid=0x00007fc11d838000 nid=0x73f0 runnable
> [0x00007fc1156e8000]
> java.lang.Thread.State: RUNNABLE
> at java.net.SocketInputStream.socketRead0(Native Method)
> ...
> at org.apache.hadoop.fs.s3a.S3AInputStream.reopen(S3AInputStream.java:216)
> - locked <0x00007fc4c2769660> (a
> org.apache.hadoop.fs.s3a.S3AInputStream)
> ... {noformat}
> We should create a new instance of StoreFileReader for each StoreFileScanner
> when doing preads, instead,
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
