[email protected] wrote: > Ok I am dealing with two John's here :) > > John Terpstra, > > Thanks for your reply. > > whoa ... 90 Mbytes/sec.... I would give an arm to get that kind of > throughput. Right now I am getting a little over 1% of that throughput :-D > > Can you tell me what nic's, switch and cabling are you using in your setup?
I gave you the NICs in the cTDB cluster (shown below). My home network has the following NICs: datastore: 01:00.0 Ethernet controller: Attansic Technology Corp. L1 Gigabit Ethernet Adapter (rev b0) Windows client: Realtec RTL8168/8111 PCI-Express Gigabit Ethernet NIC (Built into ASUS M2A-VM mobo) Linux client: 02:00.0 Ethernet controller: Attansic Technology Corp. L1 Gigabit Ethernet Adapter (rev b0) (Built into ASUS P5QC mobo) Cat5 cabling - home-grown. Switch: D-Link 1 Gigabit Model DGS-2208 > I am using lustre as the backend to samba CTDB. I dont know how much you > know about lustre so I wont bore you with a sermon :-). (I am not a lustre > expert myself - not yet ;-) ) I'm always ready to listen to an expert talk about something that excites him/her. > But my initial tests of lustre do give promising results. Using a native > lustre client on linux, I can get high throughputs. The only limitation > being the network interconnect on the client side. Using a 4 port PCI-E > Intel EEPRO's bonded togeather should give me 30x4=120MB/s (in a worst > case scenario considering inferior eqpt) Using the bonded Intel quad-port NIC (Intel Corporation 82571EB Gigabit Ethernet Controller) over two bonded ports I've seen rates of 90 MBytes/sec aggregate in each direction. That is doing DRBD sync of two block volumes each way. i.e.: 2 x 4TB from machine A to B, and 2 x 4TB from machine B to A - both running at the same time. But I would not expect to see that high a throughput over a Samba cluster. Your mileage might vary! > As I mentioned in my last post, The issue is whether samba ctdb can scale > to that bandwidth. Right now on a standard samba install on debian etch I > am trying to get more than 1.4 MB/s (NFS etc works fine at 50MB/s). This is not a Samba problem. You have a hardware issue of some sort. > Its only when I resolve this can I look at ctdb with lustre. I'd like to be kept updated on your progress with this. > If I had the luxury of using native lustre client, I am sure of meeting or > exceeding the 100MB/s objective. Unfortunately Mac's dont have a native > lustre client yet. Of course the whole system will still have to be > carefully handbuilt and tuned with each component whether its the > nic,Motherboard,switch, cabling etc; all chosen to work optimally with > each other. > > I guess my current speed issue is related to the samba version. Don't guess - prove it. :-) There have been way too many such suspicions on this list. It's time this got put to bed. > I am using 3.0.24 that comes standard with debian etch. I need to upgrade. > I will be testing out various updated versions of Samba today. > > I will keep the list posted with results of my samba-ctdb, lustre trials > as and when they are conducted as well as my current samba speed issue. Please email me (off-list) your smb.conf configuration, and your CTDB config files. I'd like to compare notes. Cheers, John T. > Thanks, > Godwin Monis > > >> [email protected] wrote: >>> John, thanks once again for the quick reply. >>> >> ... [snip]... >> >>> I am eager to understand how you are getting 50MB/s on samba transfers >>> as >>> considering the overheads added by the samba protocol, you should be >>> getting 60-75 MB/s using scp/rsync/nfs. >> My home network running Samba-3.3.1 over 1 Gigabit ethernet transfers up >> to 90 Mbytes/sec. The file system is ext3 on a Multidisk RAID5 array >> that consists of 4x1TB Seagate SATA drives. >> >> When transferring lots of small files the rate drops dramatically. When >> I run rsync over the same link between the same systems the transfer >> rate for large files hovers around 55 Mbytes/sec and drops to as low as >> 1.5 Mbytes/sec when it hits directories with lots of small files (<100 >> Kbytes). >> >>> I am also working on a COTS storage solution using lustre and Samba >>> CTDB. >>> The aim is to provide 1000MB/s to clients (100MB/s to each client) so >>> they >> Have been working on two Samba-cTDB cluster installations. One of these >> is based on RHEL and has Samba-cTDB on a front-end cluster that sits on >> top of RHCS, over a GFS2 file system, over LVM, over iSCSI. The >> back-end consists of two systems that each have 32TB of data that is >> mirrored using DRBD. The DRBD nodes are exported as iSCSI targets. >> >> So far with 2 active front-end nodes (each has 8 CPU cores) and running >> the NetBench workload using smbtorture, the highest peak I/O I have seen >> is 58 MBytes/sec. The iSCSI framework is using bonded multiple 1 >> Gigabit ethernet adaptors and the cluster front-end also uses multiple 1 >> Gigabit ethernet. >> >> I would love to find a way to get some more speed out of the cluster, >> and hence if you can meet your 100 Mbtes/sec objective I'd love to know >> how you did that! >> >> PS: Using Samba-3.3.1 with CTDB 1.0.70. >> >>> can edit video online. The solution also needs to be scalable in terms >>> of >>> IO and storage capacity and built out of open source components and COTS >>> so there is no vendor lock in. Initial tests on lustre using standard >>> dell >>> desktop hw are very good. However I need samba ctdb to communicate with >> The moment you introduce global file locking I believe you will see a >> sharp decline in throughput. Clusters are good for availability and >> reliability, but throughput is a bit elusive. >> >>> the clients as they are Apple macs. I havent reached samba ctdb >>> configuration yet. But the gigabit ethernet issue had me scared till I >>> received your reply. Now I see a lot of hope ;-). >>> >>> Once again thanks for your help and do let me know if I can reciprocate >>> your kindness. >>> >>> Cheers, >>> Godwin Monis >>> >>> >>>>> Now, if its not asking for too much, can you let me know >>>>> 1. the network chipsets used on your server and client >>>>> >>>> Main servers >>>> fileserv ~ # lspci | grep Giga >>>> 02:09.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 >>>> Gigabit Ethernet (rev 03) >>>> 02:09.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 >>>> Gigabit Ethernet (rev 03) >> NICs on the cluster servers I am working with are: >> >> nVidia Corporation MCP55 Ethernet - dual ports on mobos >> Intel Corporation 82571EB Gigabit Ethernet Controller - quad port >> >>>> dev6 ~ # lspci | grep Giga >>>> 02:09.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 >>>> Gigabit Ethernet (rev 03) >>>> 02:09.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 >>>> Gigabit Ethernet (rev 03) >>>> >>>> datastore0 ~ # lspci | grep net >>>> 00:08.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a2) >>>> 00:09.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a2) >>>> >>>> datastore1 ~ # lspci | grep net >>>> 00:08.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a2) >>>> 00:09.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a2) >>>> >>>> datastore2 ~ # lspci | grep net >>>> 00:08.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a2) >>>> 00:09.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a2) >> I hope this is useful info for you. Let me know if I can assist you in >> any way. >> >> Cheers, >> John T. >> >> > > -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/options/samba
