Nilotpal Nandi created HDDS-934:
-----------------------------------
Summary: freon run hung and did not terminate when run on
non-functional pipeline
Key: HDDS-934
URL: https://issues.apache.org/jira/browse/HDDS-934
Project: Hadoop Distributed Data Store
Issue Type: Bug
Components: Ozone Client
Reporter: Nilotpal Nandi
Attachments: jstack.txt
steps taken:
# created docker cluster with 3 datanodes where datanodes are not able to
communicate with each other.
# ran freon to write 1 key
freon run threw following exception but it did not terminate.
exception:
------------------
{noformat}
2018-12-18 10:38:06 INFO RandomKeyGenerator:227 - Number of Threads: 10
2018-12-18 10:38:06 INFO RandomKeyGenerator:233 - Number of Volumes: 1.
2018-12-18 10:38:06 INFO RandomKeyGenerator:234 - Number of Buckets per Volume:
1.
2018-12-18 10:38:06 INFO RandomKeyGenerator:235 - Number of Keys per Bucket: 1.
2018-12-18 10:38:06 INFO RandomKeyGenerator:236 - Key size: 10240 bytes
2018-12-18 10:38:06 INFO RandomKeyGenerator:266 - Starting progress bar Thread.
0.00% |█ | 0/1 Time: 0:00:002018-12-18 10:38:06 INFO RpcClient:250 - Creating
Volume: vol-0-74492, with hadoop as owner and quota set to 1152921504606846976
bytes.
2018-12-18 10:38:06 INFO RpcClient:379 - Creating Bucket:
vol-0-74492/bucket-0-16002, with Versioning false and Storage Type set to DISK
0.00% |█ | 0/1 Time: 0:02:402018-12-18 10:40:46 ERROR
ChunkGroupOutputStream:275 - Try to allocate more blocks for write failed,
already allocated 0 blocks for this write.
2018-12-18 10:40:46 ERROR RandomKeyGenerator:624 - Exception while adding key:
key-0-28925 in bucket: org.apache.hadoop.ozone.client.OzoneBucket@1675c402 of
volume: org.apache.hadoop.ozone.client.OzoneVolume@5b6bfafd.
java.io.IOException: Allocate block failed, error:INTERNAL_ERROR
at
org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.allocateBlock(OzoneManagerProtocolClientSideTranslatorPB.java:620)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.allocateNewBlock(ChunkGroupOutputStream.java:437)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleWrite(ChunkGroupOutputStream.java:272)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleException(ChunkGroupOutputStream.java:377)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleFlushOrClose(ChunkGroupOutputStream.java:473)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleFlushOrClose(ChunkGroupOutputStream.java:474)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleFlushOrClose(ChunkGroupOutputStream.java:474)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleFlushOrClose(ChunkGroupOutputStream.java:474)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleWrite(ChunkGroupOutputStream.java:309)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.write(ChunkGroupOutputStream.java:255)
at
org.apache.hadoop.ozone.client.io.OzoneOutputStream.write(OzoneOutputStream.java:49)
at java.io.OutputStream.write(OutputStream.java:75)
at
org.apache.hadoop.ozone.freon.RandomKeyGenerator$OfflineProcessor.run(RandomKeyGenerator.java:606)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
0.00% |█ | 0/1 Time: 0:11:01
0.00% |█ | 0/1 Time: 0:11:19
{noformat}
Here is the jstack for the freon process :
[^jstack.txt]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
