We were able to resolve our issue by freeing up inodes using the process outlined in https://wiki.lustre.org/ZFS_MDT_ENOSPC_Recovery. This resolved the "rm: cannot remove XXX: No space left on device" problem. Then we grew the MDT zpool by adding more disks using a standard `zpool add` command. We were a little concerned about that operation in particular since it was not clear if our older Lustre version supports dynamically increasing the size of the MDT. We found a few vague mentions of folks doing so, but no specific commands or versions were listed. It worked fine and we now have twice as many inodes as we had before.
Upgrading the Lustre version on our servers now has a higher priority so we can get to a version that lets us do file level backups of the MDT. -- Regards, -liam -There are uncountably more irrational fears than rational ones. -P. Dolan Liam Forbes [email protected] ph: 907.450.8618 UAF GI Research Computing Systems Manager https://calendly.com/ualoforbes/30min
_______________________________________________ lustre-discuss mailing list [email protected] http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
