A general question, does gridFTP use direct or buffered IO when accessing disk? Hoot
_____ From: Raj Kettimuthu [mailto:[email protected]] Sent: Tuesday, September 14, 2010 3:31 PM To: Hoot Thompson Cc: [email protected] Subject: Re: Call On Sep 14, 2010, at 8:10 AM, Hoot Thompson wrote: My apologies, I sent you an earlier version of the diagram that showed only three servers (total) running. I actually had six going. However, my port assignments were wrong per your example and the syntax of my copy command was incorrect as well. Now, with your support, the basic transfer mechanism is running and data seems to be going down both wires. My objective now is to fill both 10GigE pipes on route to trying to move data in excess of 2000MB/sec. Any tuning hints at this point would be appreciated. Then looking ahead, we're putting the pieces in place for our 40GigE disk-to-disk test from GSFC (Greenbelt, MD) to the SC10 floor in New Orleans which leads to another question, "how is gridFTP impacted by increasing rtts?". By default, GridFTP uses TCP for transport but it supports parallel streams to alleviate TCP problems with increasing RTTs. So for tuning, try varying the number of parallel streams (-p in globus-url-copy). I would recommend trying out various values between 4 and 32. Another option is to have GridFTP use UDT instead of TCP. Information on how to configure this is available at: http://www.globus.org/toolkit/docs/latest-stable/data/gridftp/admin/#gridftp -config-udt Thanks again for all the help, Hoot '-----Original Message----- From: Raj Kettimuthu <[email protected] <mailto:raj%20kettimuthu%20%[email protected]%3e> > To: Hoot Thompson <[email protected] <mailto:hoot%20thompson%20%[email protected]%3e> > Cc: [email protected] Subject: Re: Call Date: Mon, 13 Sep 2010 17:27:07 -0500 Hoot, Right now, you have GridFTP backends running only on i7test4. You need 2 backends on i7test3 and another frontend process. You need a total of 6 server processes (4 backends and 2 frontends). Here is what you will run on i7test4 globus-gridftp-server -data-interface i7test4-10g1 -S -p 7000 -dn globus-gridftp-server -data-interface i7test4-10g2 -S -p 8000 -dn globus-gridftp-server -p 6000 -S -r i7test4:7000, i7test4:8000 Here is what you will run on i7test3 globus-gridftp-server -data-interface i7test3-10g1 -S -p 7000 -dn globus-gridftp-server -data-interface i7test3-10g2 -S -p 8000 -dn globus-gridftp-server -p 6000 -S -r i7test3:7000, i7test3:8000 Run the client as follows: globus-url-copy -vb -stripe gsiftp://i7test3:6000/i7raid/hoot/file_12GB gsiftp://i7test4:6000/i7raid/hoot/file_12GB Raj On Sep 13, 2010, at 5:05 PM, Hoot Thompson wrote: Raj et al, I'm still missing something. Attached is a diagram of what I have set up - two servers interconnected with two 10GigE links. I want to pass one file, striped across both links from one server to the other, in this case from i7test4 to i7test3 and/or i7test3 to i7test4. Shown on the attachment are the three gridftp-servers I have running on each server. Are they correct? Does the command line you gave below still apply? Hoot -----Original Message----- From: Raj Kettimuthu <[email protected] <mailto:raj%20kettimuthu%20%[email protected]%3e> > To: Hoot Thompson <[email protected] <mailto:hoot%20thompson%20%[email protected]%3e> > Subject: Re: Call Date: Thu, 9 Sep 2010 17:04:56 -0500 Hoot, Sorry for the slow response. You have to use the following command-line: $GLOBUS_LOCATION/bin/globus-url-copy -vb -stripe gsiftp://i7test3-10g1:6000/i7raid/hoot/file_12GB gsiftp://i7test3-10g2:6000/i7raid/hoot/file_12GB The data will flow between the 2 backends at each end and both the links will be used. Striping also requires that the all source backends can read the source file and all the destination backends can write to the destination file i.e, backends at each end have a shared filesystem. The tutorial slides available at http://www.globus.org/toolkit/data/gridftp/tutorials/ has more information on striping. Please post your questions to gridftp-user list so that you do not block on me. Raj On Sep 7, 2010, at 3:39 PM, Hoot Thompson wrote: ok, I'm getting close but admittedly I've reached a point of confusion. In trying to walk before I run, I have a triangle configured of sort where my frontend node calls out the two NIC cards in the other server using the -r option. Here's the frontend... sudo -E /home/globus/globus/sbin/globus-gridftp-server -S -p 6000 -r 192.168.1.13:7000,192.168.2.13:8000 -logfile /tmp/grid -log-level 'ALL' On the backend I have two servers started.... sudo -E /home/globus/globus/sbin/globus-gridftp-server -S -p 7000 -dn -logfile /tmp/grid -log-level 'ALL' sudo -E /home/globus/globus/sbin/globus-gridftp-server -S -p 8000 -dn -logfile /tmp/grid -log-level 'ALL' So now when I issue the following (logged into the frontend which is .1.12(10g1) and .2.12(10g2)) $GLOBUS_LOCATION/bin/globus-url-copy -vb -stripe gsiftp://i7test3-10g1:6000/i7raid/hoot/file_12GB /i7raid/hoot/file_12GB -----data travels over the .1.x subnet when I issue $GLOBUS_LOCATION/bin/globus-url-copy -vb -stripe gsiftp://i7test3-10g2:6000/i7raid/hoot/file_12GB /i7raid/hoot/file_12GB ------data travels over the .2.x subnet I assume this makes sense. My confusion is when I bring up a second gridftp server on the frontend, what does my globus-url-copy command look like to pull data over both links simultanesouly? BTW, performance is great for the one link, 945.64 MB/sec avg 1179.98 MB/sec inst -----Original Message----- From: Raj Kettimuthu <[email protected] <mailto:raj%20kettimuthu%20%[email protected]%3e> > To: Hoot Thompson <[email protected] <mailto:hoot%20thompson%20%[email protected]%3e> > Subject: Re: Call Date: Fri, 3 Sep 2010 16:15:24 -0500 You have to run 3 instances of gridftp server on each side. On the sending you run 2 data nodes (with -dn), one each on 1.12 and 2.12 and run 1 control node (with -r or remote_nodes 1.12, 2.12) on either 1.12 or 2.12 or you can designate another machine for this On the receiving you do the same - run 2 data nodes (with -dn), one each on 1.13 and 2.13 and run 1 control node (with -r or remote_nodes 1.13, 2.13) on either 1.13 or 2.13 or you can designate another machine for this Then you run 'globus-url-copy -stripe <source control node> <dest control node>' Hope this helps. Raj On Sep 3, 2010, at 4:06 PM, Hoot Thompson wrote: So on the send side I can put something like the following in the .conf file... remote_nodes 192.16.1.13:6000,192.168.2.13:7000 But on the receive side, how do you specify multiple data_node interfaces on one machine? Hoot _____ From: Raj Kettimuthu [mailto:[email protected]] Sent: Friday, September 03, 2010 4:39 PM To: Hoot Thompson Subject: Re: Call Hoot, The link below should have the information you are looking for: http://www.globus.org/toolkit/docs/latest-stable/data/gridftp/admin/#gridftp -admin-striped If something is not clear, please feel free to contact me. Raj On Sep 3, 2010, at 3:33 PM, Hoot Thompson wrote: Maybe you'd have a moment to offer a little advice and some of this will be repetitive. I currently have two servers, each with two 10GigE NICs. i7test3 192.168.1.12 192.168.2.12 i7test4 192.168.1.13 192.168.2.13 I want to stripe a single large file over the two links. What goes in the in the gridftp.conf files on both ends to make this happen? Do any command line options need to be invoked when I initiate the the url to url copy? Thanks, hoot _____ From: Raj Kettimuthu [mailto:[email protected]] Sent: Friday, September 03, 2010 4:12 PM To: Hoot Thompson Subject: Re: Call Good. I just started replying to your email on the server-to-server error. Have a good long weekend. Raj On Sep 3, 2010, at 3:09 PM, Hoot Thompson wrote: Got server to server working :-) Have a good long weekend. Hoot _____ From: Raj Kettimuthu [mailto:[email protected]] Sent: Thursday, September 02, 2010 4:20 PM To: Hoot Thompson Subject: Re: Call toll-free telecon is 877-950-6599 with passcode 6356785 On Sep 2, 2010, at 3:19 PM, Hoot Thompson wrote: Yes but I didn't get the dial in number -----Original Message----- From: Raj Kettimuthu [mailto:[email protected]] Sent: Thursday, September 02, 2010 4:16 PM To: Hoot Thompson Subject: Call Pat and I are on the call. Are you going to join? <test_layout.pdf>
