Hi, We have ruled that out by monitoring use. It is happening during checkpointing. So a continuing process were old checkpoints get deleted after new ones are made. There are many checkpoints before
I messed things up in my first mail, so it wasn't clear why I talked about space. Sometimes they they just get this (first number is MPI rank): 222: forrtl: Input/output error 222: forrtl: severe (28): CLOSE error, unit 10, file "Unknown" Sometimes they get: 33: forrtl: No space left on device 14: forrtl: No space left on device 08: forrtl: Input/output error 08: forrtl: severe (28): CLOSE error, unit 10, file "Unknown" Info: We have a ZFS snapshot of the osts and mdt. It's ZFS 0.6.5.7. Cheers, Hans Henrik On 09-06-2017 08:41, Thomas Roth wrote: > Hi, > > I don't know about the error messages. But are you sure that the > imbalance of the OST filling isn't due to some extremely large files > written overnight or so (- with default striping, one file -> one OST). > Our users are able to do that, without realizing. > > Regards, > Thomas > > On 08.06.2017 10:11, Hans Henrik Happe wrote: >> Hi, >> >> We are on Lustre 2.8 with ZFS. >> >> Our users have seen some unexplainable errors: >> >> 062: forrtl: Input/output error >> >> Or >> >> 062: forrtl: severe (28): CLOSE error, unit 10, file “Unknown" >> >> >> From attached 'lfs df -h' you can see that the OSTs are unbalanced and >> OST0001 but far from being full. We are using default allocation setting >> so we should be in weighted mode. >> >> I've tried to find an LU matching this but no luck. Also, log on >> affected nodes and on servers are empty. >> >> Any suggestions about how to debug this? >> >> Cheers, >> Hans Henrik >> >> >> >> _______________________________________________ >> lustre-discuss mailing list >> [email protected] >> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org >> > _______________________________________________ lustre-discuss mailing list [email protected] http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
