Hi Daniel, isn't this expected: on your lustrefs-OST0001, usage seems to have hit the limit (perhaps if you do 'lfs quota -g somegroup...', it will show you by how many bytes).
If one part of the distributed quota is exceeded, Lustre should report that with the * - although the total across the file system is still below the limit.
Obviously your 'somegroup' is at the quota limit on all visible OSTs, so my guess is that would be the same on the missing two OSTs. So, either have some data removed or increase the limit. Best regards Thomas On 26.10.22 16:52, Daniel Szkola via lustre-discuss wrote:
Hello all, We recently removed an OSS/OST node that was spontaneously shutting down so hardware testing could be performed. I have no idea how long it will be out, so I followed the procedure for permanent removal. Since then space usage is being calculated correctly, but 'lfs quota' will show groups as exceeding quota, despite being under both soft and hard limits. A verbose listing shows that all OST limits are met and I have no idea how to reset the limits now that the two OSTs on the removed OSS node are not part of the equation. Due to the heavy usage of the Lustre filesystem, no clients have been unmounted and no MDS or OST nodes have been restarted. The underlying filesystem is ZFS. Looking for ideas on how to correct this. Example: # lfs quota -gh somegroup -v /lustre1 Disk quotas for grp somegroup (gid NNNN): Filesystem used quota limit grace files quota limit grace /lustre1 21.59T* 27T 30T 6d23h39m15s 2250592 2621440 3145728 - lustrefs-MDT0000_UUID 1.961G - 1.962G - 2250592 - 2359296 - lustrefs-OST0000_UUID 2.876T - 2.876T - - - - - lustrefs-OST0001_UUID 2.611T* - 2.611T - - - - - lustrefs-OST0002_UUID 4.794T - 4.794T - - - - - lustrefs-OST0003_UUID 4.587T - 4.587T - - - - - quotactl ost4 failed. quotactl ost5 failed. lustrefs-OST0006_UUID 3.21T - 3.21T - - - - - lustrefs-OST0007_UUID 3.515T - 3.515T - - - - - Total allocated inode limit: 2359296, total allocated block limit: 21.59T Some errors happened when getting quota info. Some devices may be not working or deactivated. The data in "[]" is inaccurate. -- Dan Szkola FNAL _______________________________________________ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
_______________________________________________ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org