Re: [lustre-discuss] Recommendation on number of OSTs

2024-01-12 Thread Backer via lustre-discuss
Thanks for the advice! On Fri, 12 Jan 2024 at 19:23, Andreas Dilger wrote: > I would recommend *not* to use too many OSTs as this causes fragmentation > of the free space, and excess overhead in managing the connections. Today, > single OSTs can be up to 500TiB in size (or larger, though not

Re: [lustre-discuss] Mixing ZFS and LDISKFS

2024-01-12 Thread Backer via lustre-discuss
Sounds good. Thank you! On Fri, 12 Jan 2024 at 19:28, Andreas Dilger wrote: > All of the OSTs and MDTs are "independently managed" (have their own > connection state between each client and target) so this should be > possible, though I don't know of sites that are doing this. Possibly this >

Re: [lustre-discuss] Mixing ZFS and LDISKFS

2024-01-12 Thread Andreas Dilger via lustre-discuss
All of the OSTs and MDTs are "independently managed" (have their own connection state between each client and target) so this should be possible, though I don't know of sites that are doing this. Possibly this makes sense to put NVMe flash OSTs on ldiskfs, and HDD OSTs on ZFS, and then put

Re: [lustre-discuss] Recommendation on number of OSTs

2024-01-12 Thread Andreas Dilger via lustre-discuss
I would recommend *not* to use too many OSTs as this causes fragmentation of the free space, and excess overhead in managing the connections. Today, single OSTs can be up to 500TiB in size (or larger, though not necessarily optimal for performance). Depending on your cluster size and total

Re: [lustre-discuss] Mixing ZFS and LDISKFS

2024-01-12 Thread Backer via lustre-discuss
Thank you Andreas! How about mixing OSTs? The requirement is to do RAID with small volumes using ZFS and have a large OST. This is to reduce the number of OSTs overall as the cluster being extended. On Fri, 12 Jan 2024 at 11:26, Andreas Dilger wrote: > Yes, some systems use ldiskfs for the MDT

Re: [lustre-discuss] Mixing ZFS and LDISKFS

2024-01-12 Thread Andreas Dilger via lustre-discuss
Yes, some systems use ldiskfs for the MDT (for performance) and ZFS for the OSTs (for low-cost RAID). The IOPS performance of ZFS is low vs. ldiskfs, but the streaming bandwidth is fine. Cheers, Andreas > On Jan 12, 2024, at 08:40, Backer via lustre-discuss > wrote: > >  > Hi, > > Could

[lustre-discuss] Mixing ZFS and LDISKFS

2024-01-12 Thread Backer via lustre-discuss
Hi, Could we mix ZFS and LDISKFS together in a cluster? Thank you, ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] Recommendation on number of OSTs

2024-01-12 Thread Backer via lustre-discuss
Hi All, What is the recommendation on the total number of OSTs? In order to maximize throughput, go for more number of OSS with small OSTs. This means that it will end up with 1000s of OSTs. Any suggestions or recommendations? Thank you, ___