[
https://issues.apache.org/jira/browse/HDFS-4515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
WenJin Ma updated HDFS-4515:
----------------------------
Description:
1.I use programs simulate 3000 users to write 100K small files in HDFS。（3
datanodes）
2.Found that after a period of time the client and datanodes a lot of timeout
errors or errors caused by the socket timeout
example：file:log1.txt
3.dump datanode java stack log（view file:java.log）
cat java.log |grep BLOCKED|wc -l
2635
A large number of threads blocking in ReplicaMap, because there used
synchronize guarantee thread safety
{code}
"DataXceiver for client DFSClient_NONMAPREDUCE_-1528766469_27 at
/10.28.171.254:59064 [Receiving block
BP-560172827-10.28.171.226-1360119691522:blk_4662142724658079555_330234]"
daemon prio=10 tid=0x00000000428d2800 nid=0x9e32 waiting for monitor entry
[0x00007f20ea56d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:670)
- waiting to lock <0x00000000f81c05d0> (a
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:89)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:159)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:393)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:98)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:66)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:219)
at java.lang.Thread.run(Thread.java:662)
{code}
Linux users resources may be exhausted after running for some time.
HDFS this realization in our test environment can only guarantee stable
operation in 800 users.
4. I will be synchronized mechanism is changed to read-write locks, and try to
run in the 4500 user does not appear a large number of thread blocks.
5. dump java stack log(view file:java3.log).
cat java3.log |grep BLOCKED|wc -l
0
was:
1.I use programs simulate 3000 users to write 100K small files in HDFS。（3
datanodes）
2.Found that after a period of time the client and datanodes a lot of timeout
errors or errors caused by the socket timeout
example：log1.txt
3.dump datanode java stack log（view file:java.log）
cat java.log |grep BLOCKED|wc -l
2635
A large number of threads blocking in ReplicaMap, because there used
synchronize guarantee thread safety
{code}
"DataXceiver for client DFSClient_NONMAPREDUCE_-1528766469_27 at
/10.28.171.254:59064 [Receiving block
BP-560172827-10.28.171.226-1360119691522:blk_4662142724658079555_330234]"
daemon prio=10 tid=0x00000000428d2800 nid=0x9e32 waiting for monitor entry
[0x00007f20ea56d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:670)
- waiting to lock <0x00000000f81c05d0> (a
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:89)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:159)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:393)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:98)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:66)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:219)
at java.lang.Thread.run(Thread.java:662)
{code}
Linux users resources may be exhausted after running for some time.
HDFS this realization in our test environment can only guarantee stable
operation in 800 users.
4. I will be synchronized mechanism is changed to read-write locks, and try to
run in the 4500 user does not appear a large number of thread blocks.
5. dump java stack log(view file java3.log).
cat java3.log |grep BLOCKED|wc -l
0
> ReplicaMap thread-safe synchronization lead to a large number of threads
> blocking
> ---------------------------------------------------------------------------------
>
> Key: HDFS-4515
> URL: https://issues.apache.org/jira/browse/HDFS-4515
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Affects Versions: 2.0.3-alpha
> Environment: CentOS release 6.3
> Reporter: WenJin Ma
> Attachments: java3.log, java.log, logs1.txt
>
> Original Estimate: 612h
> Remaining Estimate: 612h
>
> 1.I use programs simulate 3000 users to write 100K small files in HDFS。（3
> datanodes）
> 2.Found that after a period of time the client and datanodes a lot of timeout
> errors or errors caused by the socket timeout
> example：file:log1.txt
> 3.dump datanode java stack log（view file:java.log）
> cat java.log |grep BLOCKED|wc -l
> 2635
> A large number of threads blocking in ReplicaMap, because there used
> synchronize guarantee thread safety
> {code}
> "DataXceiver for client DFSClient_NONMAPREDUCE_-1528766469_27 at
> /10.28.171.254:59064 [Receiving block
> BP-560172827-10.28.171.226-1360119691522:blk_4662142724658079555_330234]"
> daemon prio=10 tid=0x00000000428d2800 nid=0x9e32 waiting for monitor entry
> [0x00007f20ea56d000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:670)
> - waiting to lock <0x00000000f81c05d0> (a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:89)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:159)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:393)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:98)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:66)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:219)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> Linux users resources may be exhausted after running for some time.
> HDFS this realization in our test environment can only guarantee stable
> operation in 800 users.
> 4. I will be synchronized mechanism is changed to read-write locks, and try
> to run in the 4500 user does not appear a large number of thread blocks.
> 5. dump java stack log(view file:java3.log).
> cat java3.log |grep BLOCKED|wc -l
> 0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
