Hi Lindsay, With NVME journals on a 3 node 4 OSD cluster if I do a quick dd of a 1GB file on a VM I can see 2.34Gbps on the storage network straight away so if I was only using 1Gbps here the network would be a bottlekneck. If I perform the same in 2 VMs traffic hits 4.19Gbps on the storage network.
The throughput in the VM is 1073741824 bytes (1.1 GB) copied, 3.43556 s, 313 MB/s (R=3) Would be very interested in hearing more about your gluster setup.. I don't know anything about it - how many nodes are involved? On Fri, Apr 22, 2016 at 3:02 PM, Lindsay Mathieson <[email protected]> wrote: > On 22/04/2016 11:31 PM, Brian :: wrote: >> >> 10Gbps or faster at a minimum or you will have pain. Even using 4 >> nodes with 4 spinner disks in each node and you will be maxing out >> 1Gbps network. > > > > Can't say I saw that on our cluster. > > - 3 Nodes > - 3 OSD's per Node > - SSD journals for each OSD. > - 2*1G Eth in LACP Bond dedicated to Ceph > - 1G Admin Net > - Size = 3 (Replica 3) > > Never came close to maxing out 1 1Gbps connection, write throughput and IOPS > are terrible. Read is pretty good though. > > Currently trialing Gluster 3.7.11 on ZFS Bricks, Replica 3 also. Triple the > throughput ans IOPS I was getting with Ceph, maxes out the 2*1G connection, > also seems to deal with VM I/O spikes better too, not letting other VM's be > stalled. > > Not convinced its as robust as Ceph yet, give it a few more weeks. It does > cope very well with failover and brick heals (using 64MB shards). > > -- > Lindsay Mathieson > > > _______________________________________________ > pve-user mailing list > [email protected] > http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user _______________________________________________ pve-user mailing list [email protected] http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
