Sandeep Nemuri created HDDS-1315:
------------------------------------
Summary: datanode process dies if it runs out of disk space
Key: HDDS-1315
URL: https://issues.apache.org/jira/browse/HDDS-1315
Project: Hadoop Distributed Data Store
Issue Type: Bug
Components: Ozone Datanode
Reporter: Sandeep Nemuri
As of now the datanode process dies if it runs out of disk space which makes
the data present in that DN is inaccessible.
datanode logs:
{code:java}
2019-03-11 04:01:27,141 ERROR org.apache.ratis.server.storage.RaftLogWorker:
Terminating with exit status 1:
fb635e52-e2eb-46b1-b109-a831c10d3bf8-RaftLogWorker failed.
java.io.FileNotFoundException:
/opt/data/meta/ratis/68e315f3-312c-4c9f-a7bd-590194deb5e7/current/log_inprogress_8705582
(No space left on device)
at java.io.RandomAccessFile.open0(Native Method)
at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
at
org.apache.ratis.server.storage.LogOutputStream.<init>(LogOutputStream.java:66)
at
org.apache.ratis.server.storage.RaftLogWorker$StartLogSegment.execute(RaftLogWorker.java:436)
at
org.apache.ratis.server.storage.RaftLogWorker.run(RaftLogWorker.java:219)
at java.lang.Thread.run(Thread.java:745)
{code}
{code:java}
2019-03-11 04:01:25,531 [grpc-default-executor-9192] INFO - Operation:
WriteChunk : Trace ID: : Message: java.nio.file.FileSystemException:
/opt/data/hdds/a83a7108-91c7-4357-9f68-46753641d429/current/containerDir0/88/chunks/ba29bb91559179cbf7ab5d86cac47ba1_stream_9fb1e802-dca6-46e0-be12-5ac743d8563d_chunk_1.tmp.11076.8705539:
No space left on device : Result: IO_EXCEPTION
2019-03-11 04:01:25,543 [grpc-default-executor-9192] INFO - Operation:
WriteChunk : Trace ID: : Message: java.nio.file.FileSystemException:
/opt/data/hdds/a83a7108-91c7-4357-9f68-46753641d429/current/containerDir0/86/chunks/19ef3c1d36eadbc9538116c68c6e494f_stream_c58e8b91-dc18-4b61-918f-ab1eeda41c02_chunk_1.tmp.11076.8705540:
No space left on device : Result: IO_EXCEPTION
2019-03-11 04:01:25,546 [grpc-default-executor-9192] INFO - Operation:
WriteChunk : Trace ID: : Message: java.nio.file.FileSystemException:
/opt/data/hdds/a83a7108-91c7-4357-9f68-46753641d429/current/containerDir0/87/chunks/83a6a81f2f703f49a7e0a1413eebfc4c_stream_cae1ed30-c613-4278-8404-c9e37d0b690f_chunk_1.tmp.11076.8705541:
No space left on device : Result: IO_EXCEPTION
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
