Thanks, I will verify the settings. -----Original Message----- From: Raj Kettimuthu <[email protected]> To: Raj Kettimuthu <[email protected]> Cc: Hoot Thompson <[email protected]>, [email protected] Subject: Re: Call Date: Wed, 15 Sep 2010 11:14:59 -0500
Hoot, I forgot to mention this in my earlier email. Please take a look at the link below and tune the TCP settings at the end hosts accordingly: http://fasterdata.es.net/TCP-tuning/linux.html Raj On Sep 14, 2010, at 2:30 PM, Raj Kettimuthu wrote: > > > On Sep 14, 2010, at 8:10 AM, Hoot Thompson wrote: > > > > > My apologies, I sent you an earlier version of the diagram that > > showed only three servers (total) running. I actually had six > > going. However, my port assignments were wrong per your example and > > the syntax of my copy command was incorrect as well. Now, with your > > support, the basic transfer mechanism is running and data seems to > > be going down both wires. My objective now is to fill both 10GigE > > pipes on route to trying to move data in excess of 2000MB/sec. Any > > tuning hints at this point would be appreciated. Then looking > > ahead, we're putting the pieces in place for our 40GigE disk-to-disk > > test from GSFC (Greenbelt, MD) to the SC10 floor in New Orleans > > which leads to another question, "how is gridFTP impacted by > > increasing rtts?". > > > > > By default, GridFTP uses TCP for transport but it supports parallel > streams to alleviate TCP problems with increasing RTTs. So for tuning, > try varying the number of parallel streams (-p in globus-url-copy). I > would recommend trying out various values between 4 and 32. > Another option is to have GridFTP use UDT instead of TCP. Information > on how to configure this is available at: > http://www.globus.org/toolkit/docs/latest-stable/data/gridftp/admin/#gridftp-config-udt > > > > > > > > > Thanks again for all the help, > > > > Hoot > > > > '-----Original Message----- > > From: Raj Kettimuthu <[email protected]> > > To: Hoot Thompson <[email protected]> > > Cc: [email protected] > > Subject: Re: Call > > Date: Mon, 13 Sep 2010 17:27:07 -0500 > > > > Hoot, Right now, you have GridFTP backends running only on i7test4. > > You need 2 backends on i7test3 and another frontend process. You > > need a total of 6 server processes (4 backends and 2 frontends). > > > > Here is what you will run on i7test4 > > > > globus-gridftp-server -data-interface i7test4-10g1 -S -p 7000 -dn > > globus-gridftp-server -data-interface i7test4-10g2 -S -p 8000 -dn > > globus-gridftp-server -p 6000 -S -r i7test4:7000, i7test4:8000 > > > > Here is what you will run on i7test3 > > > > globus-gridftp-server -data-interface i7test3-10g1 -S -p 7000 -dn > > globus-gridftp-server -data-interface i7test3-10g2 -S -p 8000 -dn > > globus-gridftp-server -p 6000 -S -r i7test3:7000, i7test3:8000 > > > > Run the client as follows: globus-url-copy -vb -stripe > > gsiftp://i7test3:6000/i7raid/hoot/file_12GB > > gsiftp://i7test4:6000/i7raid/hoot/file_12GB > > > > Raj > > On Sep 13, 2010, at 5:05 PM, Hoot Thompson wrote: > > > > > Raj et al, > > > > > > I'm still missing something. Attached is a diagram of what I have > > > set up - two servers interconnected with two 10GigE links. I want > > > to pass one file, striped across both links from one server to the > > > other, in this case from i7test4 to i7test3 and/or i7test3 to > > > i7test4. Shown on the attachment are the three gridftp-servers I > > > have running on each server. Are they correct? Does the command > > > line you gave below still apply? > > > > > > Hoot > > > > > > > > > -----Original Message----- > > > From: Raj Kettimuthu <[email protected]> > > > To: Hoot Thompson <[email protected]> > > > Subject: Re: Call > > > Date: Thu, 9 Sep 2010 17:04:56 -0500 > > > > > > Hoot, Sorry for the slow response. You have to use the following > > > command-line: $GLOBUS_LOCATION/bin/globus-url-copy -vb -stripe > > > gsiftp://i7test3-10g1:6000/i7raid/hoot/file_12GB > > > gsiftp://i7test3-10g2:6000/i7raid/hoot/file_12GB > > > > > > The data will flow between the 2 backends at each end and both the > > > links will be used. Striping also requires that the all source > > > backends can read the source file and all the destination backends > > > can write to the destination file i.e, backends at each end have a > > > shared filesystem. > > > > > > The tutorial slides available > > > at http://www.globus.org/toolkit/data/gridftp/tutorials/ has more > > > information on striping. > > > > > > Please post your questions to gridftp-user list so that you do not > > > block on me. > > > > > > Raj > > > > > > > > > > > > On Sep 7, 2010, at 3:39 PM, Hoot Thompson wrote: > > > > > > > ok, I'm getting close but admittedly I've reached a point of > > > > confusion. In trying to walk before I run, I have a triangle > > > > configured of sort where my frontend node calls out the two NIC > > > > cards in the other server using the -r option. > > > > > > > > Here's the frontend... > > > > sudo -E /home/globus/globus/sbin/globus-gridftp-server -S -p > > > > 6000 -r 192.168.1.13:7000,192.168.2.13:8000 -logfile /tmp/grid > > > > -log-level 'ALL' > > > > > > > > On the backend I have two servers started.... > > > > sudo -E /home/globus/globus/sbin/globus-gridftp-server -S -p > > > > 7000 -dn -logfile /tmp/grid -log-level 'ALL' > > > > > > > > sudo -E /home/globus/globus/sbin/globus-gridftp-server -S -p > > > > 8000 -dn -logfile /tmp/grid -log-level 'ALL' > > > > > > > > > > > > So now when I issue the following (logged into the frontend > > > > which is .1.12(10g1) and .2.12(10g2)) > > > > > > > > $GLOBUS_LOCATION/bin/globus-url-copy -vb -stripe > > > > gsiftp://i7test3-10g1:6000/i7raid/hoot/file_12GB /i7raid/hoot/file_12GB > > > > -----data travels over the .1.x subnet > > > > > > > > when I issue > > > > $GLOBUS_LOCATION/bin/globus-url-copy -vb -stripe > > > > gsiftp://i7test3-10g2:6000/i7raid/hoot/file_12GB /i7raid/hoot/file_12GB > > > > ------data travels over the .2.x subnet > > > > > > > > I assume this makes sense. My confusion is when I bring up a > > > > second gridftp server on the frontend, what does my > > > > globus-url-copy command look like to pull data over both links > > > > simultanesouly? > > > > > > > > BTW, performance is great for the one link, > > > > > > > > 945.64 MB/sec avg 1179.98 MB/sec inst > > > > > > > > > > > > -----Original Message----- > > > > From: Raj Kettimuthu <[email protected]> > > > > To: Hoot Thompson <[email protected]> > > > > Subject: Re: Call > > > > Date: Fri, 3 Sep 2010 16:15:24 -0500 > > > > > > > > You have to run 3 instances of gridftp server on each side. > > > > > > > > On the sending you run 2 data nodes (with -dn), one each on 1.12 > > > > and 2.12 and run 1 control node (with -r or remote_nodes 1.12, > > > > 2.12) on either 1.12 or 2.12 or you can designate another > > > > machine for this > > > > > > > > On the receiving you do the same - run 2 data nodes (with -dn), > > > > one each on 1.13 and 2.13 and run 1 control node (with -r or > > > > remote_nodes 1.13, 2.13) on either 1.13 or 2.13 or you can > > > > designate another machine for this > > > > > > > > Then you run 'globus-url-copy -stripe <source control node> > > > > <dest control node>' > > > > > > > > Hope this helps. > > > > > > > > Raj > > > > On Sep 3, 2010, at 4:06 PM, Hoot Thompson wrote: > > > > > > > > > So on the send side I can put something like the following in > > > > > the .conf file... > > > > > > > > > > remote_nodes 192.16.1.13:6000,192.168.2.13:7000 > > > > > > > > > > But on the receive side, how do you specify multiple data_node > > > > > interfaces on one machine? > > > > > > > > > > > > > > > Hoot > > > > > > > > > > > > > > > ______________________________________________________________ > > > > > > > > > > > > > > > > > > > > From: Raj Kettimuthu [mailto:[email protected]] > > > > > Sent: Friday, September 03, 2010 4:39 PM > > > > > To: Hoot Thompson > > > > > Subject: Re: Call > > > > > > > > > > > > > > > > > > > > Hoot, > > > > > The link below should have the information you are looking > > > > > for: > > > > > http://www.globus.org/toolkit/docs/latest-stable/data/gridftp/admin/#gridftp-admin-striped > > > > > If something is not clear, please feel free to contact me. > > > > > > > > > > > > > > > Raj > > > > > > > > > > On Sep 3, 2010, at 3:33 PM, Hoot Thompson wrote: > > > > > > > > > > > > > > > > Maybe you'd have a moment to offer a little advice and some > > > > > > of this will be repetitive. I currently have two servers, > > > > > > each with two 10GigE NICs. > > > > > > > > > > > > i7test3 > > > > > > 192.168.1.12 > > > > > > 192.168.2.12 > > > > > > > > > > > > i7test4 > > > > > > 192.168.1.13 > > > > > > 192.168.2.13 > > > > > > > > > > > > I want to stripe a single large file over the two links. > > > > > > What goes in the in the gridftp.conf files on both ends to > > > > > > make this happen? Do any command line options need to be > > > > > > invoked when I initiate the the url to url copy? > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > hoot > > > > > > > > > > > > > > > > > > ____________________________________________________________ > > > > > > > > > > > > > > > > > > > > > > > > From: Raj Kettimuthu [mailto:[email protected]] > > > > > > Sent: Friday, September 03, 2010 4:12 PM > > > > > > To: Hoot Thompson > > > > > > Subject: Re: Call > > > > > > > > > > > > > > > > > > > > > > > > Good. I just started replying to your email on the > > > > > > server-to-server error. Have a good long weekend. > > > > > > > > > > > > > > > > > > Raj > > > > > > > > > > > > On Sep 3, 2010, at 3:09 PM, Hoot Thompson wrote: > > > > > > > > > > > > > > > > > > > Got server to server working :-) > > > > > > > > > > > > > > Have a good long weekend. > > > > > > > > > > > > > > Hoot > > > > > > > > > > > > > > > > > > > > > __________________________________________________________ > > > > > > > > > > > > > > > > > > > > > > > > > > > > From: Raj Kettimuthu [mailto:[email protected]] > > > > > > > Sent: Thursday, September 02, 2010 4:20 PM > > > > > > > To: Hoot Thompson > > > > > > > Subject: Re: Call > > > > > > > > > > > > > > > > > > > > > > > > > > > > toll-free telecon is 877-950-6599 with passcode 6356785 > > > > > > > > > > > > > > On Sep 2, 2010, at 3:19 PM, Hoot Thompson wrote: > > > > > > > > > > > > > > > > > > > > > > Yes but I didn't get the dial in number > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > From: Raj Kettimuthu [mailto:[email protected]] > > > > > > > > Sent: Thursday, September 02, 2010 4:16 PM > > > > > > > > To: Hoot Thompson > > > > > > > > Subject: Call > > > > > > > > > > > > > > > > Pat and I are on the call. Are you going to join? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > <test_layout.pdf> > > > > > > > > > > >
