Wellington Chevreuil created HBASE-27511: --------------------------------------------
Summary: Lock contention when doing multiple parallel preads due to StoreFileReader reuse Key: HBASE-27511 URL: https://issues.apache.org/jira/browse/HBASE-27511 Project: HBase Issue Type: Bug Reporter: Wellington Chevreuil Assignee: Wellington Chevreuil Attachments: rs-stack-lock-contention In HStoreFile, we reuse the StoreFileReader created during HStoreFile initialization when creating a StoreFileScanner for preads: [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStoreFile.java#L545] When using S3 as hbase storage, we noticed this caused lock contention when multiple clients were doing preads in parallel: {noformat} ... "RpcServer.default.FPBQ.Fifo.handler=38,queue=8,port=16020" #125 daemon prio=5 os_prio=0 tid=0x00007fc11d83c000 nid=0x73f2 waiting for monitor entry [0x00007fc1154e6000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:73) - waiting to lock <0x00007fc4c2769660> (a org.apache.hadoop.fs.s3a.S3AInputStream) ... "RpcServer.default.FPBQ.Fifo.handler=37,queue=7,port=16020" #124 daemon prio=5 os_prio=0 tid=0x00007fc11d83a000 nid=0x73f1 waiting for monitor entry [0x00007fc1155e7000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:73) - waiting to lock <0x00007fc4c2769660> (a org.apache.hadoop.fs.s3a.S3AInputStream) ... "RpcServer.default.FPBQ.Fifo.handler=36,queue=6,port=16020" #123 daemon prio=5 os_prio=0 tid=0x00007fc11d838000 nid=0x73f0 runnable [0x00007fc1156e8000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) ... at org.apache.hadoop.fs.s3a.S3AInputStream.reopen(S3AInputStream.java:216) - locked <0x00007fc4c2769660> (a org.apache.hadoop.fs.s3a.S3AInputStream) ... {noformat} We should create a new instance of StoreFileReader for each StoreFileScanner when doing preads, instead, -- This message was sent by Atlassian Jira (v8.20.10#820010)