[jira] [Updated] (HDFS-13564) PreAllocator for DfsClientShm
[ https://issues.apache.org/jira/browse/HDFS-13564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HDFS-13564: - Fix Version/s: (was: 3.0.2) > PreAllocator for DfsClientShm > - > > Key: HDFS-13564 > URL: https://issues.apache.org/jira/browse/HDFS-13564 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.0.2 >Reporter: Gang Xie >Assignee: Lisheng Sun >Priority: Minor > > When we do a stress test against Short-Circuit Local Reads, and found a > bottleneck that allocating new DfsClientShm blocks a lot of slot allocatings > on it. > Currently, there are 128 slots per shm which means at most, 128 reads could > be blocked by the shm allocation. Especially when stressed, the domain socket > communication to datanode gets slow, and datanode could also have GC, which > could cause some hundreds ms to allocate 1 shm, in turn, the reads. This is > bad for some latency sensitive service, like Hbase. > I'm working on the prototype and will upload the code and test result later. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13564) PreAllocator for DfsClientShm
[ https://issues.apache.org/jira/browse/HDFS-13564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HDFS-13564: - Target Version/s: (was: 3.0.2) > PreAllocator for DfsClientShm > - > > Key: HDFS-13564 > URL: https://issues.apache.org/jira/browse/HDFS-13564 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.0.2 >Reporter: Gang Xie >Assignee: Lisheng Sun >Priority: Minor > > When we do a stress test against Short-Circuit Local Reads, and found a > bottleneck that allocating new DfsClientShm blocks a lot of slot allocatings > on it. > Currently, there are 128 slots per shm which means at most, 128 reads could > be blocked by the shm allocation. Especially when stressed, the domain socket > communication to datanode gets slow, and datanode could also have GC, which > could cause some hundreds ms to allocate 1 shm, in turn, the reads. This is > bad for some latency sensitive service, like Hbase. > I'm working on the prototype and will upload the code and test result later. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
