Gagan:
I actually tried what Alexander mentioned below, as a separate experiment.
I have two AMD Ryzen 9 5950X compute nodes and one AMD Ryzen 9 7950X compute
node, and each node has 128 GB of RAM and also a Mellanox ConnectX-4 100 Gbps
Infiniband network card, and I was using I think one Intel 670p 1 TB NVMe SSD
and two Silicon Power US70 1 TB NVMe SSD.
>From the Ceph perspective, giving it quote "this much hardware", didn't
>significantly improve the performance.
Of course, 100 Gbps Infiniband is faster than 1 GbE, but Ceph barely even
noticed.
Where it played a more significant role was when I was doing live migrations of
VMs and LXCs where it was able to use the 100 Gbps IB network speeds more so
for migrating RAM states between nodes than the actual virtual disk drive of
the VM. Ceph won't help with the RAM state. And since the VM disk was already
sitting on the shared, distributed, Ceph storage, I wasn't moving the disk over
100 Gbps IB, just the RAM state.
So depending on how you've set it up, throwing more hardware at Ceph, might not
improve performance much.
Thanks.
Sincerely,
Ewen
________________________________
From: Gluster-users <gluster-users-boun...@gluster.org> on behalf of Alexander
Schreiber <a...@thangorodrim.ch>
Sent: April 17, 2025 7:54 AM
To: gagan tiwari <gagan.tiw...@mathisys-india.com>
Cc: gluster-users@gluster.org <gluster-users@gluster.org>
Subject: Re: [Gluster-users] Gluster with ZFS
On Thu, Apr 17, 2025 at 02:44:28PM +0530, gagan tiwari wrote:
> HI Alexander,
> Thanks for the update. Initially, I also
> thought of deploying Ceph but ceph is quite difficult to set-up and manage.
> Moreover, it's also hardware demanding. I think it's most suitable for a
> very large set-up with hundreds of clients.
I strongly disagree. I run a small (3 nodes) Ceph cluster in my homelab
and following the official docs it's pretty easy to set up. The hardware
demands mostly depend on what performance one needs - the more performance
(e.g. NVMe storage and 100 GBit networking) one wants, the more powerful
hardware one has to provide, as usual. My nodes are Intel D-1521 with
64G of (ECC, of course) RAM and Connect-X 4 cards running at 10 GBit
and storage on HDDs which provide reasonable performance for my needs - not
a HPC setup, of course.
> What do you think of MooseFS ? Have you or anyone else tried MooseFS. If
> yes, how was its performance?
Last time I looked, MooseFS needs a commercial license for the full feature
set (e.g. highly available metadata (_that_ is not negotiable!), erasure
oding, Windows clients) which killed it for my non-commercial use case.
Kind regards,
Alex.
--
"Opportunity is missed by most people because it is dressed in overalls and
looks like work." -- Thomas A. Edison
________
Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users
________
Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users
