Hoot, I forgot to mention this in my earlier email. Please take a look at the link below and tune the TCP settings at the end hosts accordingly: http://fasterdata.es.net/TCP-tuning/linux.html
Raj On Sep 14, 2010, at 2:30 PM, Raj Kettimuthu wrote: > > On Sep 14, 2010, at 8:10 AM, Hoot Thompson wrote: > >> My apologies, I sent you an earlier version of the diagram that showed only >> three servers (total) running. I actually had six going. However, my port >> assignments were wrong per your example and the syntax of my copy command >> was incorrect as well. Now, with your support, the basic transfer mechanism >> is running and data seems to be going down both wires. My objective now is >> to fill both 10GigE pipes on route to trying to move data in excess of >> 2000MB/sec. Any tuning hints at this point would be appreciated. Then >> looking ahead, we're putting the pieces in place for our 40GigE disk-to-disk >> test from GSFC (Greenbelt, MD) to the SC10 floor in New Orleans which leads >> to another question, "how is gridFTP impacted by increasing rtts?". > > By default, GridFTP uses TCP for transport but it supports parallel streams > to alleviate TCP problems with increasing RTTs. So for tuning, try varying > the number of parallel streams (-p in globus-url-copy). I would recommend > trying out various values between 4 and 32. > Another option is to have GridFTP use UDT instead of TCP. Information on how > to configure this is available at: > http://www.globus.org/toolkit/docs/latest-stable/data/gridftp/admin/#gridftp-config-udt > > >> >> Thanks again for all the help, >> >> Hoot >> >> '-----Original Message----- >> From: Raj Kettimuthu <[email protected]> >> To: Hoot Thompson <[email protected]> >> Cc: [email protected] >> Subject: Re: Call >> Date: Mon, 13 Sep 2010 17:27:07 -0500 >> >> Hoot, Right now, you have GridFTP backends running only on i7test4. You need >> 2 backends on i7test3 and another frontend process. You need a total of 6 >> server processes (4 backends and 2 frontends). >> >> Here is what you will run on i7test4 >> >> globus-gridftp-server -data-interface i7test4-10g1 -S -p 7000 -dn >> globus-gridftp-server -data-interface i7test4-10g2 -S -p 8000 -dn >> globus-gridftp-server -p 6000 -S -r i7test4:7000, i7test4:8000 >> >> Here is what you will run on i7test3 >> >> globus-gridftp-server -data-interface i7test3-10g1 -S -p 7000 -dn >> globus-gridftp-server -data-interface i7test3-10g2 -S -p 8000 -dn >> globus-gridftp-server -p 6000 -S -r i7test3:7000, i7test3:8000 >> >> Run the client as follows: globus-url-copy -vb -stripe >> gsiftp://i7test3:6000/i7raid/hoot/file_12GB >> gsiftp://i7test4:6000/i7raid/hoot/file_12GB >> >> Raj >> On Sep 13, 2010, at 5:05 PM, Hoot Thompson wrote: >>> Raj et al, >>> >>> I'm still missing something. Attached is a diagram of what I have set up - >>> two servers interconnected with two 10GigE links. I want to pass one file, >>> striped across both links from one server to the other, in this case from >>> i7test4 to i7test3 and/or i7test3 to i7test4. Shown on the attachment are >>> the three gridftp-servers I have running on each server. Are they correct? >>> Does the command line you gave below still apply? >>> >>> Hoot >>> >>> >>> -----Original Message----- >>> From: Raj Kettimuthu <[email protected]> >>> To: Hoot Thompson <[email protected]> >>> Subject: Re: Call >>> Date: Thu, 9 Sep 2010 17:04:56 -0500 >>> >>> Hoot, Sorry for the slow response. You have to use the following >>> command-line: $GLOBUS_LOCATION/bin/globus-url-copy -vb -stripe >>> gsiftp://i7test3-10g1:6000/i7raid/hoot/file_12GB >>> gsiftp://i7test3-10g2:6000/i7raid/hoot/file_12GB >>> >>> The data will flow between the 2 backends at each end and both the links >>> will be used. Striping also requires that the all source backends can read >>> the source file and all the destination backends can write to the >>> destination file i.e, backends at each end have a shared filesystem. >>> >>> The tutorial slides available at >>> http://www.globus.org/toolkit/data/gridftp/tutorials/ has more information >>> on striping. >>> >>> Please post your questions to gridftp-user list so that you do not block on >>> me. >>> >>> Raj >>> >>> >>> >>> On Sep 7, 2010, at 3:39 PM, Hoot Thompson wrote: >>>> ok, I'm getting close but admittedly I've reached a point of confusion. >>>> In trying to walk before I run, I have a triangle configured of sort where >>>> my frontend node calls out the two NIC cards in the other server using the >>>> -r option. >>>> >>>> Here's the frontend... >>>> sudo -E /home/globus/globus/sbin/globus-gridftp-server -S -p 6000 -r >>>> 192.168.1.13:7000,192.168.2.13:8000 -logfile /tmp/grid -log-level 'ALL' >>>> >>>> On the backend I have two servers started.... >>>> sudo -E /home/globus/globus/sbin/globus-gridftp-server -S -p 7000 -dn >>>> -logfile /tmp/grid -log-level 'ALL' >>>> >>>> sudo -E /home/globus/globus/sbin/globus-gridftp-server -S -p 8000 -dn >>>> -logfile /tmp/grid -log-level 'ALL' >>>> >>>> >>>> So now when I issue the following (logged into the frontend which is >>>> .1.12(10g1) and .2.12(10g2)) >>>> >>>> $GLOBUS_LOCATION/bin/globus-url-copy -vb -stripe >>>> gsiftp://i7test3-10g1:6000/i7raid/hoot/file_12GB /i7raid/hoot/file_12GB >>>> -----data travels over the .1.x subnet >>>> >>>> when I issue >>>> $GLOBUS_LOCATION/bin/globus-url-copy -vb -stripe >>>> gsiftp://i7test3-10g2:6000/i7raid/hoot/file_12GB /i7raid/hoot/file_12GB >>>> ------data travels over the .2.x subnet >>>> >>>> I assume this makes sense. My confusion is when I bring up a second >>>> gridftp server on the frontend, what does my globus-url-copy command look >>>> like to pull data over both links simultanesouly? >>>> >>>> BTW, performance is great for the one link, >>>> >>>> 945.64 MB/sec avg 1179.98 MB/sec inst >>>> >>>> >>>> -----Original Message----- >>>> From: Raj Kettimuthu <[email protected]> >>>> To: Hoot Thompson <[email protected]> >>>> Subject: Re: Call >>>> Date: Fri, 3 Sep 2010 16:15:24 -0500 >>>> >>>> You have to run 3 instances of gridftp server on each side. >>>> >>>> On the sending you run 2 data nodes (with -dn), one each on 1.12 and 2.12 >>>> and run 1 control node (with -r or remote_nodes 1.12, 2.12) on either 1.12 >>>> or 2.12 or you can designate another machine for this >>>> >>>> On the receiving you do the same - run 2 data nodes (with -dn), one each >>>> on 1.13 and 2.13 and run 1 control node (with -r or remote_nodes 1.13, >>>> 2.13) on either 1.13 or 2.13 or you can designate another machine for this >>>> >>>> Then you run 'globus-url-copy -stripe <source control node> <dest control >>>> node>' >>>> >>>> Hope this helps. >>>> >>>> Raj >>>> On Sep 3, 2010, at 4:06 PM, Hoot Thompson wrote: >>>>> So on the send side I can put something like the following in the .conf >>>>> file... >>>>> >>>>> remote_nodes 192.16.1.13:6000,192.168.2.13:7000 >>>>> >>>>> But on the receive side, how do you specify multiple data_node interfaces >>>>> on one machine? >>>>> >>>>> >>>>> Hoot >>>>> >>>>> >>>>> >>>>> >>>>> From: Raj Kettimuthu [mailto:[email protected]] >>>>> Sent: Friday, September 03, 2010 4:39 PM >>>>> To: Hoot Thompson >>>>> Subject: Re: Call >>>>> >>>>> >>>>> >>>>> Hoot, >>>>> The link below should have the information you are looking for: >>>>> http://www.globus.org/toolkit/docs/latest-stable/data/gridftp/admin/#gridftp-admin-striped >>>>> If something is not clear, please feel free to contact me. >>>>> >>>>> >>>>> Raj >>>>> >>>>> On Sep 3, 2010, at 3:33 PM, Hoot Thompson wrote: >>>>> >>>>>> Maybe you'd have a moment to offer a little advice and some of this will >>>>>> be repetitive. I currently have two servers, each with two 10GigE NICs. >>>>>> >>>>>> i7test3 >>>>>> 192.168.1.12 >>>>>> 192.168.2.12 >>>>>> >>>>>> i7test4 >>>>>> 192.168.1.13 >>>>>> 192.168.2.13 >>>>>> >>>>>> I want to stripe a single large file over the two links. What goes in >>>>>> the in the gridftp.conf files on both ends to make this happen? Do any >>>>>> command line options need to be invoked when I initiate the the url to >>>>>> url copy? >>>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> hoot >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> From: Raj Kettimuthu [mailto:[email protected]] >>>>>> Sent: Friday, September 03, 2010 4:12 PM >>>>>> To: Hoot Thompson >>>>>> Subject: Re: Call >>>>>> >>>>>> >>>>>> >>>>>> Good. I just started replying to your email on the server-to-server >>>>>> error. Have a good long weekend. >>>>>> >>>>>> >>>>>> Raj >>>>>> >>>>>> On Sep 3, 2010, at 3:09 PM, Hoot Thompson wrote: >>>>>> >>>>>>> Got server to server working :-) >>>>>>> >>>>>>> Have a good long weekend. >>>>>>> >>>>>>> Hoot >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> From: Raj Kettimuthu [mailto:[email protected]] >>>>>>> Sent: Thursday, September 02, 2010 4:20 PM >>>>>>> To: Hoot Thompson >>>>>>> Subject: Re: Call >>>>>>> >>>>>>> >>>>>>> >>>>>>> toll-free telecon is 877-950-6599 with passcode 6356785 >>>>>>> >>>>>>> On Sep 2, 2010, at 3:19 PM, Hoot Thompson wrote: >>>>>>> >>>>>>>> Yes but I didn't get the dial in number >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Raj Kettimuthu [mailto:[email protected]] >>>>>>>> Sent: Thursday, September 02, 2010 4:16 PM >>>>>>>> To: Hoot Thompson >>>>>>>> Subject: Call >>>>>>>> >>>>>>>> Pat and I are on the call. Are you going to join? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> >>> >>> >>> >>> <test_layout.pdf> >> >> >
