[
https://issues.apache.org/jira/browse/HDFS-7358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14197254#comment-14197254
]
stack commented on HDFS-7358:
-----------------------------
With the patch in place, it takes longer to hit the stall and then when I do, I
get the loggings about wake up:
{code}
2014-11-04 16:55:57,202 DEBUG [ResponseProcessor for block
BP-410607956-10.20.84.26-1391491814882:blk_1075488681_1099513376802]
util.ByteArrayManager: recycle: array.length=131072, [131072: 9999/10000,
free=0], freeQueue.offer, freeQueueSize=1
2014-11-04 16:55:57,202 DEBUG [sync.0] util.ByteArrayManager: allocate(65565):
count=60367, aboveThreshold, [131072: 9998/10000, free=1], recycled? true
2014-11-04 16:55:57,202 DEBUG [ResponseProcessor for block
BP-410607956-10.20.84.26-1391491814882:blk_1075488681_1099513376802]
util.ByteArrayManager: recycle: array.length=131072, [131072: 9999/10000,
free=0], freeQueue.offer, freeQueueSize=1
2014-11-04 16:55:57,202 DEBUG [ResponseProcessor for block
BP-410607956-10.20.84.26-1391491814882:blk_1075488681_1099513376802]
util.ByteArrayManager: recycle: array.length=131072, [131072: 9998/10000,
free=1], freeQueue.offer, freeQueueSize=2
2014-11-04 16:55:57,202 DEBUG [sync.3] util.ByteArrayManager: allocate(65565):
count=60368, aboveThreshold, [131072: 9997/10000, free=2], recycled? true
2014-11-04 16:55:57,202 DEBUG [ResponseProcessor for block
BP-410607956-10.20.84.26-1391491814882:blk_1075488681_1099513376802]
util.ByteArrayManager: recycle: array.length=131072, [131072: 9998/10000,
free=1], freeQueue.offer, freeQueueSize=2
2014-11-04 16:55:57,202 DEBUG [ResponseProcessor for block
BP-410607956-10.20.84.26-1391491814882:blk_1075488681_1099513376802]
util.ByteArrayManager: recycle: array.length=131072, [131072: 9997/10000,
free=2], freeQueue.offer, freeQueueSize=3
2014-11-04 16:55:57,203 DEBUG [sync.2] util.ByteArrayManager: allocate(65565):
count=60369, aboveThreshold, [131072: 9996/10000, free=3], recycled? true
2014-11-04 16:55:57,203 DEBUG [ResponseProcessor for block
BP-410607956-10.20.84.26-1391491814882:blk_1075488681_1099513376802]
util.ByteArrayManager: recycle: array.length=131072, [131072: 9997/10000,
free=2], freeQueue.offer, freeQueueSize=3
2014-11-04 16:55:57,203 DEBUG [sync.1] util.ByteArrayManager: allocate(65565):
count=60370, aboveThreshold, [131072: 9996/10000, free=3], recycled? true
2014-11-04 16:55:57,203 DEBUG [sync.4] util.ByteArrayManager: allocate(65565):
count=60371, aboveThreshold, [131072: 9997/10000, free=2], recycled? true
2014-11-04 16:55:57,203 DEBUG [sync.0] util.ByteArrayManager: allocate(65565):
count=60372, aboveThreshold, [131072: 9998/10000, free=1], recycled? true
2014-11-04 16:55:57,203 DEBUG [ResponseProcessor for block
BP-410607956-10.20.84.26-1391491814882:blk_1075488681_1099513376802]
util.ByteArrayManager: recycle: array.length=131072, [131072: 9999/10000,
free=0], freeQueue.offer, freeQueueSize=1
2014-11-04 16:55:57,203 DEBUG [sync.2] util.ByteArrayManager: allocate(65565):
count=60373, aboveThreshold, [131072: 9998/10000, free=1], recycled? true
2014-11-04 16:55:57,203 DEBUG [sync.3] util.ByteArrayManager: allocate(65565):
count=60374, aboveThreshold, [131072: 9999/10000, free=0], recycled? false
2014-11-04 16:55:57,204 DEBUG [ResponseProcessor for block
BP-410607956-10.20.84.26-1391491814882:blk_1075488681_1099513376802]
util.ByteArrayManager: recycle: array.length=131072, [131072: 10000/10000,
free=0], notifyAll, freeQueue.offer, freeQueueSize=1
2014-11-04 16:55:57,204 DEBUG [sync.4] util.ByteArrayManager: allocate(65565):
count=60375, aboveThreshold, [131072: 9999/10000, free=1], recycled? true
2014-11-04 16:55:57,204 DEBUG [sync.1] util.ByteArrayManager: allocate(65565):
count=60376, aboveThreshold, [131072: 10000/10000, free=0]: wait ...
2014-11-04 16:55:57,204 DEBUG [ResponseProcessor for block
BP-410607956-10.20.84.26-1391491814882:blk_1075488681_1099513376802]
util.ByteArrayManager: recycle: array.length=131072, [131072: 10000/10000,
free=0], notifyAll, freeQueue.offer, freeQueueSize=1
2014-11-04 16:55:57,204 DEBUG [sync.1] util.ByteArrayManager: wake up: [131072:
9999/10000, free=1], recycled? true
2014-11-04 16:55:57,204 DEBUG [ResponseProcessor for block
BP-410607956-10.20.84.26-1391491814882:blk_1075488681_1099513376802]
util.ByteArrayManager: recycle: array.length=131072, [131072: 10000/10000,
free=0], notifyAll, freeQueue.offer, freeQueueSize=1
2014-11-04 16:55:57,204 DEBUG [sync.0] util.ByteArrayManager: allocate(65565):
count=60377, aboveThreshold, [131072: 9999/10000, free=1], recycled? true
2014-11-04 16:55:57,204 DEBUG [sync.2] util.ByteArrayManager: allocate(65565):
count=60378, aboveThreshold, [131072: 10000/10000, free=0]: wait ...
2014-11-04 16:55:57,204 DEBUG [ResponseProcessor for block
BP-410607956-10.20.84.26-1391491814882:blk_1075488681_1099513376802]
util.ByteArrayManager: recycle: array.length=131072, [131072: 10000/10000,
free=0], notifyAll, freeQueue.offer, freeQueueSize=1
2014-11-04 16:55:57,205 DEBUG [sync.2] util.ByteArrayManager: wake up: [131072:
9999/10000, free=1], recycled? true
2014-11-04 16:55:57,205 DEBUG [sync.3] util.ByteArrayManager: allocate(65565):
count=60379, aboveThreshold, [131072: 10000/10000, free=0]: wait ...
2014-11-04 16:55:57,205 DEBUG [ResponseProcessor for block
BP-410607956-10.20.84.26-1391491814882:blk_1075488681_1099513376802]
util.ByteArrayManager: recycle: array.length=131072, [131072: 10000/10000,
free=0], notifyAll, freeQueue.offer, freeQueueSize=1
2014-11-04 16:55:57,205 DEBUG [sync.3] util.ByteArrayManager: wake up: [131072:
9999/10000, free=1], recycled? true
2014-11-04 16:55:57,205 DEBUG [sync.1] util.ByteArrayManager: allocate(65565):
count=60380, aboveThreshold, [131072: 10000/10000, free=0]: wait ...
2014-11-04 16:55:57,205 DEBUG [ResponseProcessor for block
BP-410607956-10.20.84.26-1391491814882:blk_1075488681_1099513376802]
util.ByteArrayManager: recycle: array.length=131072, [131072: 10000/10000,
free=0], notifyAll, freeQueue.offer, freeQueueSize=1
2014-11-04 16:55:57,205 DEBUG [sync.1] util.ByteArrayManager: wake up: [131072:
9999/10000, free=1], recycled? true
2014-11-04 16:55:57,205 DEBUG [sync.4] util.ByteArrayManager: allocate(65565):
count=60381, aboveThreshold, [131072: 10000/10000, free=0]: wait ...
2014-11-04 16:55:57,206 DEBUG [ResponseProcessor for block
BP-410607956-10.20.84.26-1391491814882:blk_1075488681_1099513376802]
util.ByteArrayManager: recycle: array.length=131072, [131072: 10000/10000,
free=0], notifyAll, freeQueue.offer, freeQueueSize=1
2014-11-04 16:55:57,206 DEBUG [sync.4] util.ByteArrayManager: wake up: [131072:
9999/10000, free=1], recycled? true
2014-11-04 16:55:57,206 DEBUG [sync.0] util.ByteArrayManager: allocate(65565):
count=60382, aboveThreshold, [131072: 10000/10000, free=0]: wait ...
2014-11-04 16:55:57,206 DEBUG [ResponseProcessor for block
BP-410607956-10.20.84.26-1391491814882:blk_1075488681_1099513376802]
util.ByteArrayManager: recycle: array.length=131072, [131072: 10000/10000,
free=0], notifyAll, freeQueue.offer, freeQueueSize=1
2014-11-04 16:55:57,206 DEBUG [sync.0] util.ByteArrayManager: wake up: [131072:
9999/10000, free=1], recycled? true
2014-11-04 16:55:57,206 DEBUG [sync.2] util.ByteArrayManager: allocate(65565):
count=60383, aboveThreshold, [131072: 10000/10000, free=0]: wait ...
2014-11-04 16:55:57,206 DEBUG [ResponseProcessor for block
BP-410607956-10.20.84.26-1391491814882:blk_1075488681_1099513376802]
util.ByteArrayManager: recycle: array.length=131072, [131072: 10000/10000,
free=0], notifyAll, freeQueue.offer, freeQueueSize=1
2014-11-04 16:55:57,206 DEBUG [sync.2] util.ByteArrayManager: wake up: [131072:
9999/10000, free=1], recycled? true
2014-11-04 16:55:57,206 DEBUG [sync.3] util.ByteArrayManager: allocate(65565):
count=60384, aboveThreshold, [131072: 10000/10000, free=0]: wait ...
2014-11-04 16:55:57,206 DEBUG [ResponseProcessor for block
BP-410607956-10.20.84.26-1391491814882:blk_1075488681_1099513376802]
util.ByteArrayManager: recycle: array.length=131072, [131072: 10000/10000,
free=0], notifyAll, freeQueue.offer, freeQueueSize=1
2014-11-04 16:55:57,207 DEBUG [sync.3] util.ByteArrayManager: wake up: [131072:
9999/10000, free=1], recycled? true
2014-11-04 16:55:57,207 DEBUG [sync.1] util.ByteArrayManager: allocate(65565):
count=60385, aboveThreshold, [131072: 10000/10000, free=0]: wait ...
2014-11-04 16:55:57,207 DEBUG [ResponseProcessor for block
BP-410607956-10.20.84.26-1391491814882:blk_1075488681_1099513376802]
util.ByteArrayManager: recycle: array.length=131072, [131072: 10000/10000,
free=0], notifyAll, freeQueue.offer, freeQueueSize=1
2014-11-04 16:55:57,207 DEBUG [sync.1] util.ByteArrayManager: wake up: [131072:
9999/10000, free=1], recycled? true
2014-11-04 16:55:57,207 DEBUG [sync.4] util.ByteArrayManager: allocate(65565):
count=60386, aboveThreshold, [131072: 10000/10000, free=0]: wait ...
2014-11-04 16:55:57,207 DEBUG [ResponseProcessor for block
BP-410607956-10.20.84.26-1391491814882:blk_1075488681_1099513376802]
util.ByteArrayManager: recycle: array.length=131072, [131072: 10000/10000,
free=0], notifyAll, freeQueue.offer, freeQueueSize=1
2014-11-04 16:55:57,207 DEBUG [sync.4] util.ByteArrayManager: wake up: [131072:
9999/10000, free=1], recycled? true
2014-11-04 16:55:57,207 DEBUG [sync.1] util.ByteArrayManager: allocate(65565):
count=60387, aboveThreshold, [131072: 10000/10000, free=0]: wait ...
2014-11-04 16:55:57,208 DEBUG [ResponseProcessor for block
BP-410607956-10.20.84.26-1391491814882:blk_1075488681_1099513376802]
util.ByteArrayManager: recycle: array.length=131072, [131072: 10000/10000,
free=0], notifyAll, freeQueue.offer, freeQueueSize=1
2014-11-04 16:55:57,208 DEBUG [sync.1] util.ByteArrayManager: wake up: [131072:
9999/10000, free=1], recycled? true
2014-11-04 16:55:57,208 DEBUG [sync.0] util.ByteArrayManager: allocate(65565):
count=60388, aboveThreshold, [131072: 10000/10000, free=0]: wait ...
2014-11-04 16:55:57,209 DEBUG [ResponseProcessor for block
BP-410607956-10.20.84.26-1391491814882:blk_1075488681_1099513376802]
util.ByteArrayManager: recycle: array.length=131072, [131072: 10000/10000,
free=0], notifyAll, freeQueue.offer, freeQueueSize=1
2014-11-04 16:55:57,209 DEBUG [sync.0] util.ByteArrayManager: wake up: [131072:
9999/10000, free=1], recycled? true
2014-11-04 16:55:57,209 DEBUG [sync.2] util.ByteArrayManager: allocate(65565):
count=60389, aboveThreshold, [131072: 10000/10000, free=0]: wait ...
2014-11-04 16:55:58,209 DEBUG [sync.2] util.ByteArrayManager: wake up: [131072:
10000/10000, free=0]: wait ...
2014-11-04 16:55:59,209 DEBUG [sync.2] util.ByteArrayManager: wake up: [131072:
10000/10000, free=0]: wait ...
2014-11-04 16:56:00,210 DEBUG [sync.2] util.ByteArrayManager: wake up: [131072:
10000/10000, free=0]: wait ...
2014-11-04 16:56:01,210 DEBUG [sync.2] util.ByteArrayManager: wake up: [131072:
10000/10000, free=0]: wait ...
2014-11-04 16:56:02,210 DEBUG [sync.2] util.ByteArrayManager: wake up: [131072:
10000/10000, free=0]: wait ...
2014-11-04 16:56:03,210 DEBUG [sync.2] util.ByteArrayManager: wake up: [131072:
10000/10000, free=0]: wait ...
2014-11-04 16:56:04,210 DEBUG [sync.2] util.ByteArrayManager: wake up: [131072:
10000/10000, free=0]: wait ...
2014-11-04 16:56:05,210 DEBUG [sync.2] util.ByteArrayManager: wake up: [131072:
10000/10000, free=0]: wait ...
2014-11-04 16:56:06,210 DEBUG [sync.2] util.ByteArrayManager: wake up: [131072:
10000/10000, free=0]: wait ...
2014-11-04 16:56:07,211 DEBUG [sync.2] util.ByteArrayManager: wake up: [131072:
10000/10000, free=0]: wait ...
2014-11-04 16:56:08,211 DEBUG [sync.2] util.ByteArrayManager: wake up: [131072:
10000/10000, free=0]: wait ...
2014-11-04 16:56:09,211 DEBUG [sync.2] util.ByteArrayManager: wake up: [131072:
10000/10000, free=0]: wait ...
2014-11-04 16:56:10,211 DEBUG [sync.2] util.ByteArrayManager: wake up: [131072:
10000/10000, free=0]: wait ...
2014-11-04 16:56:11,211 DEBUG [sync.2] util.ByteArrayManager: wake up: [131072:
10000/10000, free=0]: wait ...
2014-11-04 16:56:12,212 DEBUG [sync.2] util.ByteArrayManager: wake up: [131072:
10000/10000, free=0]: wait ...
2014-11-04 16:56:13,212 DEBUG [sync.2] util.ByteArrayManager: wake up: [131072:
10000/10000, free=0]: wait ...
....
{code}
> Clients may get stuck waiting when using ByteArrayManager
> ---------------------------------------------------------
>
> Key: HDFS-7358
> URL: https://issues.apache.org/jira/browse/HDFS-7358
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
> Reporter: Tsz Wo Nicholas Sze
> Assignee: Tsz Wo Nicholas Sze
> Attachments: h7358_20141104_wait_timeout.patch
>
>
> [~stack] reported that clients might get stuck waiting when using
> ByteArrayManager; see [his
> comments|https://issues.apache.org/jira/browse/HDFS-7276?focusedCommentId=14197036&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14197036].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
