[
https://issues.apache.org/jira/browse/RATIS-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Gui updated RATIS-1377:
----------------------------
Description:
We are using ozone with ratis for our services and we hit an issue with disk
out of space. We checked the log and think that it is that ratis has run out of
space and ozone pipelines (raftgroups for ratis) created on the full disk are
not able to close because it has to take a final snapshot. short log appended
below.
{code:java}
2021-05-25 19:10:47,171
[492bc1be-439e-45db-856f-2e58336e2528@group-B26E6BC26E24-StateMachineUpdater]
INFO
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine:
group-B26E6BC26E24: Taking a snapshot at:(t:5, i:419) file
/data1/ratis/c5a9bc6e-fee1-48a8-9100-b26e6bc26e24/sm/snapshot.5_419 2021-05-25
19:10:47,171
[492bc1be-439e-45db-856f-2e58336e2528@group-B26E6BC26E24-StateMachineUpdater]
ERROR
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine:
group-B26E6BC26E24: Failed to write snapshot at:(t:5, i:419) file
/data1/ratis/c5a9bc6e-fee1-48a8-9100-b26e6bc26e24/sm/snapshot.5_419 2021-05-25
19:10:47,171
[492bc1be-439e-45db-856f-2e58336e2528@group-B26E6BC26E24-StateMachineUpdater]
ERROR org.apache.ratis.server.impl.StateMachineUpdater:
492bc1be-439e-45db-856f-2e58336e2528@group-B26E6BC26E24-StateMachineUpdater:
Failed to take snapshot java.io.IOException: No space left on device at
java.io.FileOutputStream.writeBytes(Native Method) at
java.io.FileOutputStream.write(FileOutputStream.java:326) at
org.apache.ratis.thirdparty.com.google.protobuf.CodedOutputStream$OutputStreamEncoder.doFlush(CodedOutputStream.java:3062)
at
org.apache.ratis.thirdparty.com.google.protobuf.CodedOutputStream$OutputStreamEncoder.flushIfNotAvailable(CodedOutputStream.java:3057)
at
org.apache.ratis.thirdparty.com.google.protobuf.CodedOutputStream$OutputStreamEncoder.writeUInt64NoTag(CodedOutputStream.java:2897)
at
org.apache.ratis.thirdparty.com.google.protobuf.CodedOutputStream.writeInt64NoTag(CodedOutputStream.java:414)
at
org.apache.ratis.thirdparty.com.google.protobuf.FieldSet.writeElementNoTag(FieldSet.java:657)
at
org.apache.ratis.thirdparty.com.google.protobuf.FieldSet.writeElement(FieldSet.java:634)
at
org.apache.ratis.thirdparty.com.google.protobuf.MapEntryLite.writeTo(MapEntryLite.java:110)
at
org.apache.ratis.thirdparty.com.google.protobuf.MapEntry.writeTo(MapEntry.java:154)
at
org.apache.ratis.thirdparty.com.google.protobuf.CodedOutputStream$OutputStreamEncoder.writeMessageNoTag(CodedOutputStream.java:2855)
at
org.apache.ratis.thirdparty.com.google.protobuf.CodedOutputStream$OutputStreamEncoder.writeMessage(CodedOutputStream.java:2824)
at
org.apache.ratis.thirdparty.com.google.protobuf.GeneratedMessageV3.serializeMapTo(GeneratedMessageV3.java:3224)
at
org.apache.ratis.thirdparty.com.google.protobuf.GeneratedMessageV3.serializeLongMapTo(GeneratedMessageV3.java:3140)
at
org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$Container2BCSIDMapProto.writeTo(ContainerProtos.java:14633)
at
org.apache.ratis.thirdparty.com.google.protobuf.AbstractMessageLite.writeTo(AbstractMessageLite.java:83)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.persistContainerSet(ContainerStateMachine.java:270)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.takeSnapshot(ContainerStateMachine.java:294)
at
org.apache.ratis.server.impl.StateMachineUpdater.takeSnapshot(StateMachineUpdater.java:265)
at
org.apache.ratis.server.impl.StateMachineUpdater.checkAndTakeSnapshot(StateMachineUpdater.java:257)
at
org.apache.ratis.server.impl.StateMachineUpdater.run(StateMachineUpdater.java:183)
at java.lang.Thread.run(Thread.java:748)
{code}
So I think as the consumer of the disk, ratis should be able to mange the
free/used space and have some guarantee that operations should not be partial
completed due to out of space. We may build a reserved space for each disk in
ratis and filter out disks which reach the defined threshold for new raftgroup
allocation. Although the problem we hit happened on ozone side, but as the
comsumer of the metadata disks, this should better be done in ratis.
was:
We are using ozone with ratis for our services and we hit an issue with disk
out of space. We checked the log and think that it is that ratis has run out of
space and ozone pipelines (raftgroups for ratis) created on the full disk are
not able to close because it has to take a final snapshot. short log appended
below.
```
2021-05-25 19:10:47,171
[492bc1be-439e-45db-856f-2e58336e2528@group-B26E6BC26E24-StateMachineUpdater]
INFO
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine:
group-B26E6BC26E24: Taking a snapshot at:(t:5, i:419) file
/data1/ratis/c5a9bc6e-fee1-48a8-9100-b26e6bc26e24/sm/snapshot.5_419 2021-05-25
19:10:47,171
[492bc1be-439e-45db-856f-2e58336e2528@group-B26E6BC26E24-StateMachineUpdater]
ERROR
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine:
group-B26E6BC26E24: Failed to write snapshot at:(t:5, i:419) file
/data1/ratis/c5a9bc6e-fee1-48a8-9100-b26e6bc26e24/sm/snapshot.5_419 2021-05-25
19:10:47,171
[492bc1be-439e-45db-856f-2e58336e2528@group-B26E6BC26E24-StateMachineUpdater]
ERROR org.apache.ratis.server.impl.StateMachineUpdater:
492bc1be-439e-45db-856f-2e58336e2528@group-B26E6BC26E24-StateMachineUpdater:
Failed to take snapshot java.io.IOException: No space left on device at
java.io.FileOutputStream.writeBytes(Native Method) at
java.io.FileOutputStream.write(FileOutputStream.java:326) at
org.apache.ratis.thirdparty.com.google.protobuf.CodedOutputStream$OutputStreamEncoder.doFlush(CodedOutputStream.java:3062)
at
org.apache.ratis.thirdparty.com.google.protobuf.CodedOutputStream$OutputStreamEncoder.flushIfNotAvailable(CodedOutputStream.java:3057)
at
org.apache.ratis.thirdparty.com.google.protobuf.CodedOutputStream$OutputStreamEncoder.writeUInt64NoTag(CodedOutputStream.java:2897)
at
org.apache.ratis.thirdparty.com.google.protobuf.CodedOutputStream.writeInt64NoTag(CodedOutputStream.java:414)
at
org.apache.ratis.thirdparty.com.google.protobuf.FieldSet.writeElementNoTag(FieldSet.java:657)
at
org.apache.ratis.thirdparty.com.google.protobuf.FieldSet.writeElement(FieldSet.java:634)
at
org.apache.ratis.thirdparty.com.google.protobuf.MapEntryLite.writeTo(MapEntryLite.java:110)
at
org.apache.ratis.thirdparty.com.google.protobuf.MapEntry.writeTo(MapEntry.java:154)
at
org.apache.ratis.thirdparty.com.google.protobuf.CodedOutputStream$OutputStreamEncoder.writeMessageNoTag(CodedOutputStream.java:2855)
at
org.apache.ratis.thirdparty.com.google.protobuf.CodedOutputStream$OutputStreamEncoder.writeMessage(CodedOutputStream.java:2824)
at
org.apache.ratis.thirdparty.com.google.protobuf.GeneratedMessageV3.serializeMapTo(GeneratedMessageV3.java:3224)
at
org.apache.ratis.thirdparty.com.google.protobuf.GeneratedMessageV3.serializeLongMapTo(GeneratedMessageV3.java:3140)
at
org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$Container2BCSIDMapProto.writeTo(ContainerProtos.java:14633)
at
org.apache.ratis.thirdparty.com.google.protobuf.AbstractMessageLite.writeTo(AbstractMessageLite.java:83)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.persistContainerSet(ContainerStateMachine.java:270)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.takeSnapshot(ContainerStateMachine.java:294)
at
org.apache.ratis.server.impl.StateMachineUpdater.takeSnapshot(StateMachineUpdater.java:265)
at
org.apache.ratis.server.impl.StateMachineUpdater.checkAndTakeSnapshot(StateMachineUpdater.java:257)
at
org.apache.ratis.server.impl.StateMachineUpdater.run(StateMachineUpdater.java:183)
at java.lang.Thread.run(Thread.java:748)
```
So I think as the consumer of the disk, ratis should be able to mange the
free/used space and have some guarantee that operations should not be partial
completed due to out of space. We may build a reserved space for each disk in
ratis and filter out disks which reach the defined threshold for new raftgroup
allocation. Although the problem we hit happened on ozone side, but as the
comsumer of the metadata disks, this should better be done in ratis.
> Ratis reserved space for storage dirs
> -------------------------------------
>
> Key: RATIS-1377
> URL: https://issues.apache.org/jira/browse/RATIS-1377
> Project: Ratis
> Issue Type: Improvement
> Reporter: Mark Gui
> Assignee: Mark Gui
> Priority: Major
>
> We are using ozone with ratis for our services and we hit an issue with disk
> out of space. We checked the log and think that it is that ratis has run out
> of space and ozone pipelines (raftgroups for ratis) created on the full disk
> are not able to close because it has to take a final snapshot. short log
> appended below.
> {code:java}
> 2021-05-25 19:10:47,171
> [492bc1be-439e-45db-856f-2e58336e2528@group-B26E6BC26E24-StateMachineUpdater]
> INFO
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine:
> group-B26E6BC26E24: Taking a snapshot at:(t:5, i:419) file
> /data1/ratis/c5a9bc6e-fee1-48a8-9100-b26e6bc26e24/sm/snapshot.5_419
> 2021-05-25 19:10:47,171
> [492bc1be-439e-45db-856f-2e58336e2528@group-B26E6BC26E24-StateMachineUpdater]
> ERROR
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine:
> group-B26E6BC26E24: Failed to write snapshot at:(t:5, i:419) file
> /data1/ratis/c5a9bc6e-fee1-48a8-9100-b26e6bc26e24/sm/snapshot.5_419
> 2021-05-25 19:10:47,171
> [492bc1be-439e-45db-856f-2e58336e2528@group-B26E6BC26E24-StateMachineUpdater]
> ERROR org.apache.ratis.server.impl.StateMachineUpdater:
> 492bc1be-439e-45db-856f-2e58336e2528@group-B26E6BC26E24-StateMachineUpdater:
> Failed to take snapshot java.io.IOException: No space left on device at
> java.io.FileOutputStream.writeBytes(Native Method) at
> java.io.FileOutputStream.write(FileOutputStream.java:326) at
> org.apache.ratis.thirdparty.com.google.protobuf.CodedOutputStream$OutputStreamEncoder.doFlush(CodedOutputStream.java:3062)
> at
> org.apache.ratis.thirdparty.com.google.protobuf.CodedOutputStream$OutputStreamEncoder.flushIfNotAvailable(CodedOutputStream.java:3057)
> at
> org.apache.ratis.thirdparty.com.google.protobuf.CodedOutputStream$OutputStreamEncoder.writeUInt64NoTag(CodedOutputStream.java:2897)
> at
> org.apache.ratis.thirdparty.com.google.protobuf.CodedOutputStream.writeInt64NoTag(CodedOutputStream.java:414)
> at
> org.apache.ratis.thirdparty.com.google.protobuf.FieldSet.writeElementNoTag(FieldSet.java:657)
> at
> org.apache.ratis.thirdparty.com.google.protobuf.FieldSet.writeElement(FieldSet.java:634)
> at
> org.apache.ratis.thirdparty.com.google.protobuf.MapEntryLite.writeTo(MapEntryLite.java:110)
> at
> org.apache.ratis.thirdparty.com.google.protobuf.MapEntry.writeTo(MapEntry.java:154)
> at
> org.apache.ratis.thirdparty.com.google.protobuf.CodedOutputStream$OutputStreamEncoder.writeMessageNoTag(CodedOutputStream.java:2855)
> at
> org.apache.ratis.thirdparty.com.google.protobuf.CodedOutputStream$OutputStreamEncoder.writeMessage(CodedOutputStream.java:2824)
> at
> org.apache.ratis.thirdparty.com.google.protobuf.GeneratedMessageV3.serializeMapTo(GeneratedMessageV3.java:3224)
> at
> org.apache.ratis.thirdparty.com.google.protobuf.GeneratedMessageV3.serializeLongMapTo(GeneratedMessageV3.java:3140)
> at
> org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$Container2BCSIDMapProto.writeTo(ContainerProtos.java:14633)
> at
> org.apache.ratis.thirdparty.com.google.protobuf.AbstractMessageLite.writeTo(AbstractMessageLite.java:83)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.persistContainerSet(ContainerStateMachine.java:270)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.takeSnapshot(ContainerStateMachine.java:294)
> at
> org.apache.ratis.server.impl.StateMachineUpdater.takeSnapshot(StateMachineUpdater.java:265)
> at
> org.apache.ratis.server.impl.StateMachineUpdater.checkAndTakeSnapshot(StateMachineUpdater.java:257)
> at
> org.apache.ratis.server.impl.StateMachineUpdater.run(StateMachineUpdater.java:183)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> So I think as the consumer of the disk, ratis should be able to mange the
> free/used space and have some guarantee that operations should not be partial
> completed due to out of space. We may build a reserved space for each disk in
> ratis and filter out disks which reach the defined threshold for new
> raftgroup allocation. Although the problem we hit happened on ozone side, but
> as the comsumer of the metadata disks, this should better be done in ratis.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
