[
https://issues.apache.org/jira/browse/RATIS-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz-wo Sze resolved RATIS-1377.
-------------------------------
Resolution: Fixed
I just have merged the pull request. Thanks, [~markgui]!
> Ratis min free space for storage dirs
> -------------------------------------
>
> Key: RATIS-1377
> URL: https://issues.apache.org/jira/browse/RATIS-1377
> Project: Ratis
> Issue Type: Improvement
> Components: server
> Reporter: Mark Gui
> Assignee: Mark Gui
> Priority: Major
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> We are using ozone with ratis for our services and we hit an issue with disk
> out of space. We checked the log and think that it is that ratis has run out
> of space and ozone pipelines (raftgroups for ratis) created on the full disk
> are not able to close because it has to take a final snapshot. short log
> appended below.
> {code:java}
> 2021-05-25 19:10:47,171
> [492bc1be-439e-45db-856f-2e58336e2528@group-B26E6BC26E24-StateMachineUpdater]
> INFO
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine:
> group-B26E6BC26E24: Taking a snapshot at:(t:5, i:419) file
> /data1/ratis/c5a9bc6e-fee1-48a8-9100-b26e6bc26e24/sm/snapshot.5_419
> 2021-05-25 19:10:47,171
> [492bc1be-439e-45db-856f-2e58336e2528@group-B26E6BC26E24-StateMachineUpdater]
> ERROR
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine:
> group-B26E6BC26E24: Failed to write snapshot at:(t:5, i:419) file
> /data1/ratis/c5a9bc6e-fee1-48a8-9100-b26e6bc26e24/sm/snapshot.5_419
> 2021-05-25 19:10:47,171
> [492bc1be-439e-45db-856f-2e58336e2528@group-B26E6BC26E24-StateMachineUpdater]
> ERROR org.apache.ratis.server.impl.StateMachineUpdater:
> 492bc1be-439e-45db-856f-2e58336e2528@group-B26E6BC26E24-StateMachineUpdater:
> Failed to take snapshot java.io.IOException: No space left on device at
> java.io.FileOutputStream.writeBytes(Native Method) at
> java.io.FileOutputStream.write(FileOutputStream.java:326) at
> org.apache.ratis.thirdparty.com.google.protobuf.CodedOutputStream$OutputStreamEncoder.doFlush(CodedOutputStream.java:3062)
> at
> org.apache.ratis.thirdparty.com.google.protobuf.CodedOutputStream$OutputStreamEncoder.flushIfNotAvailable(CodedOutputStream.java:3057)
> at
> org.apache.ratis.thirdparty.com.google.protobuf.CodedOutputStream$OutputStreamEncoder.writeUInt64NoTag(CodedOutputStream.java:2897)
> at
> org.apache.ratis.thirdparty.com.google.protobuf.CodedOutputStream.writeInt64NoTag(CodedOutputStream.java:414)
> at
> org.apache.ratis.thirdparty.com.google.protobuf.FieldSet.writeElementNoTag(FieldSet.java:657)
> at
> org.apache.ratis.thirdparty.com.google.protobuf.FieldSet.writeElement(FieldSet.java:634)
> at
> org.apache.ratis.thirdparty.com.google.protobuf.MapEntryLite.writeTo(MapEntryLite.java:110)
> at
> org.apache.ratis.thirdparty.com.google.protobuf.MapEntry.writeTo(MapEntry.java:154)
> at
> org.apache.ratis.thirdparty.com.google.protobuf.CodedOutputStream$OutputStreamEncoder.writeMessageNoTag(CodedOutputStream.java:2855)
> at
> org.apache.ratis.thirdparty.com.google.protobuf.CodedOutputStream$OutputStreamEncoder.writeMessage(CodedOutputStream.java:2824)
> at
> org.apache.ratis.thirdparty.com.google.protobuf.GeneratedMessageV3.serializeMapTo(GeneratedMessageV3.java:3224)
> at
> org.apache.ratis.thirdparty.com.google.protobuf.GeneratedMessageV3.serializeLongMapTo(GeneratedMessageV3.java:3140)
> at
> org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$Container2BCSIDMapProto.writeTo(ContainerProtos.java:14633)
> at
> org.apache.ratis.thirdparty.com.google.protobuf.AbstractMessageLite.writeTo(AbstractMessageLite.java:83)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.persistContainerSet(ContainerStateMachine.java:270)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.takeSnapshot(ContainerStateMachine.java:294)
> at
> org.apache.ratis.server.impl.StateMachineUpdater.takeSnapshot(StateMachineUpdater.java:265)
> at
> org.apache.ratis.server.impl.StateMachineUpdater.checkAndTakeSnapshot(StateMachineUpdater.java:257)
> at
> org.apache.ratis.server.impl.StateMachineUpdater.run(StateMachineUpdater.java:183)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> So I think as the consumer of the disk, ratis should be able to mange the
> free/used space and have some guarantee that operations should not be partial
> completed due to out of space. We may build a reserved space for each disk in
> ratis and filter out disks which reach the defined threshold for new
> raftgroup allocation. Although the problem we hit happened on ozone side, but
> as the comsumer of the metadata disks, this should better be done in ratis.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
