Re: Best utilizing fat long pipes and large file transfer
On Mon, 16 Jun 2008, Glen Turner wrote: Then there's the deliberate nobbling of the TCP implementation, such as the restriction to ten of connections to Windows Xp SP3. Apparently you're meant to buy Windows Server if you are running P2P applications :-) are you quite sure it is *10 tcp connections*? have you tested this? -Dan
RE: Best utilizing fat long pipes and large file transfer
It's 10 half-open (SYN_SENT) outbound TCP connections as I recall. - S -Original Message- From: [EMAIL PROTECTED] [EMAIL PROTECTED] Sent: Monday, June 16, 2008 12:26 To: Glen Turner [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] [EMAIL PROTECTED] Subject: Re: Best utilizing fat long pipes and large file transfer On Mon, 16 Jun 2008, Glen Turner wrote: Then there's the deliberate nobbling of the TCP implementation, such as the restriction to ten of connections to Windows Xp SP3. Apparently you're meant to buy Windows Server if you are running P2P applications :-) are you quite sure it is *10 tcp connections*? have you tested this? -Dan
Re: Best utilizing fat long pipes and large file transfer
Hi Sean, from thursday, we have copied some ~300 GB packages from Prague to San Diego (~200 ms delay, 10 GE flat ethernet end machines connected via 1GE) files using RBUDP which worked great. Each scenario needs some planning. You have to answer several questions: 1) What is the performance of storage subsystem (sometimes you need to connect external harddrives or tape robot) 2) How many files you need to transfer? 3) How big are these files? 4) What is the consistency scenarion (it is file consistency or package consistency)? In example, I've sent some film data. Lot (~30.000) of 10 MB DPXes. Consistency was package based. Harddrives have been at the beggining connected via iLink (arrived on this media), then moved to eSATA (went to shop, bought another drive and connected it into export machine). Main tuning for RBUDP has been to buy another harddrive and tar these files. Regards Michal Hi, I'm looking for input on the best practices for sending large files over a long fat pipe between facilities (gigabit private circuit, ~20ms RTT). I'd like to avoid modifying TCP windows and options on end hosts where possible (I have a lot of them). I've seen products that work as transfer stations using reliable UDP to get around the windowing problem. I'm thinking of setting up servers with optimized TCP settings to push big files around data centers but I'm curious to know how others deal with LFN+large transfers. thanks, Sean
Re: Best utilizing fat long pipes and large file transfer
Kevin Oberman [EMAIL PROTECTED] writes: From: Robert E. Seastrom [EMAIL PROTECTED] Date: Thu, 12 Jun 2008 21:15:49 -0400 Randy Bush [EMAIL PROTECTED] writes: and for those of us who are addicted to simple rsync, or whatever over ssh, you should be aware of the really bad openssh windowing issue. As a user of hpn-ssh for years, I have to wonder if there is any reason (aside from the sheer cussedness for which Theo is infamous) that the window improvements at least from hpn-ssh haven't been backported into mainline openssh? I suppose there might be portability concerns with the multithreaded ciphers, and there's certainly a good argument for not supporting NONE as a cipher type out of the box without a recompile, but there's not much excuse for the fixed size tiny buffers - I mean, it's 2008 already... Theo is known for his amazing stubbornness, but for area involving security and cryptography, I find it hard to say that his conservatism is excessive. Crypto is hard and often it is very non-intuitive. I remember the long discussions on entropy harvesting and seeding in FreeBSD which fortunately has cryptography professionals who could pick every nit and make sure FreeBSD did not end up with Debian-type egg all over its virtual face. Than again, the tiny buffers are silly and I can't imagine any possible security issue there. Many good reasons to not goof with the crypto. The window size was the main thing I was poking at. ---rob
Re: Best utilizing fat long pipes and large file transfer
Date: Thu, 12 Jun 2008 19:26:56 -0400 From: Robert Boyle [EMAIL PROTECTED] At 06:37 PM 6/12/2008, you wrote: I'm looking for input on the best practices for sending large files over a long fat pipe between facilities (gigabit private circuit, ~20ms RTT). I'd like to avoid modifying TCP windows and options on end hosts where possible (I have a lot of them). I've seen products that work as transfer stations using reliable UDP to get around the windowing problem. I'm thinking of setting up servers with optimized TCP settings to push big files around data centers but I'm curious to know how others deal with LFN+large transfers. In our experience, you can't get to line speed with over 20-30ms of latency using TCP regardless of how much you tweak it. We transfer files across the US with 60-70ms at line speeds with UDP based file transfer programs. There are a number of open source projects out there designed for this purpose. Clearly you have failed to try very hard or to check into what others have done. We routinely move data at MUCH higher rates over TCP at latencies over 50 ms. one way (100 ms. RTT). We find it fairly easy to move data at over 4 Gbps continuously. If you can't fill a GE to 80% (800 Mbps) at 30 ms, you really are not tying very hard. Note: I am talking about a single TCP stream running for over 5 minutes at a time on tuned systems. Tuning for most modern network stacks is pretty trivial. Some older stacks (e.g. FreeBSD V6) are hopeless. I can't speak to how Windows does as I make no use of it for high-speed bulk transfers. It's also fairly easy to run multiple parallel TCP streams to completely fill a 10 Gbps pipe at any distance. This is the preferred method for moving large volumes of data around the world in the RE community as it requires little or no system tuning and is available in several open-source tools. -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: [EMAIL PROTECTED] Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751 pgpJBc3JGmrbz.pgp Description: PGP signature
Re: Best utilizing fat long pipes and large file transfer
Many thanks for great replies on and off-list. The suggestions basically ranged from these options: 1. tune TCP on all hosts you wish to transfer between 2. create tuned TCP proxies and transfer through those hosts 3. setup a socat (netcat++) proxy and send through this host 4. use an alternative to plain netcat/scp for large file transfers My needs are pretty simple: occasionally I need to push large database files (300Gb+) around linux hosts. #4 seems like the best option for me. People suggested a slew of methods to do this: RBUDP, gridftp, bbcp, and many others, with programs either sending with reliable UDP or breaking large transfers into multiple streams. Because it was easy to use right away, I tried RBUDP and was able to copy a tarball at about 700Mb/s over a 20ms delay link, and when factoring in destination disk write speed, isn't too bad a starting point. GridFTP and bbcp look very useful too; I'll be exploring them as well. The presentation links Kevin O. sent were very interesting. I've looked at HPN-SSH before but haven't played with it much. I'll definitely try it out based on the feedback from this thread. Thanks again. sk Sean Knox wrote: Hi, I'm looking for input on the best practices for sending large files over a long fat pipe between facilities (gigabit private circuit, ~20ms RTT). I'd like to avoid modifying TCP windows and options on end hosts where possible (I have a lot of them). I've seen products that work as transfer stations using reliable UDP to get around the windowing problem. I'm thinking of setting up servers with optimized TCP settings to push big files around data centers but I'm curious to know how others deal with LFN+large transfers. thanks, Sean
Re: Best utilizing fat long pipes and large file transfer
Date: Fri, 13 Jun 2008 12:40:48 -0400 From: Robert Boyle [EMAIL PROTECTED] At 12:01 PM 6/13/2008, Kevin Oberman wrote: Clearly you have failed to try very hard or to check into what others have done. We routinely move data at MUCH higher rates over TCP at latencies over 50 ms. one way (100 ms. RTT). We find it fairly easy to move data at over 4 Gbps continuously. That's impressive. If you can't fill a GE to 80% (800 Mbps) at 30 ms, you really are not tying very hard. Note: I am talking about a single TCP stream running for over 5 minutes at a time on tuned systems. Tuning for most modern network stacks is pretty trivial. Some older stacks (e.g. FreeBSD V6) are hopeless. I can't speak to how Windows does as I make no use of it for high-speed bulk transfers. Let me refine my post then... In our experience, you can't get to line speed with over 20-30ms of latency using TCP on _Windows_ regardless of how much you tweak it. 99% of the servers in our facilities are Windows based. I should have been more specific. Sorry, but I don't do Windows, but my friends who do claim that this is not true. -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: [EMAIL PROTECTED] Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751 pgp8Wyb7nb3Xl.pgp Description: PGP signature
Re: Best utilizing fat long pipes and large file transfer
On 2008-06-12, Kevin Oberman [EMAIL PROTECTED] wrote: The idea is to use tuned proxies that are close to the source and destination and are optimized for the delay. OpenBSD has relayd(8), a versatile tool which can be used here. There is support for proxying TCP connections. These can be modified in a few ways - socket options (nodelay, sack, socket buffer) can be adjusted on the relayed connection, also SSL can be offloaded. It works with the firewall state table and can retain the original addresses. Parts of this are only in development snapshots at present but will be in the 4.4 release.
Re: Best utilizing fat long pipes and large file transfer
Date: Thu, 12 Jun 2008 15:37:47 -0700 From: Sean Knox [EMAIL PROTECTED] Hi, I'm looking for input on the best practices for sending large files over a long fat pipe between facilities (gigabit private circuit, ~20ms RTT). I'd like to avoid modifying TCP windows and options on end hosts where possible (I have a lot of them). I've seen products that work as transfer stations using reliable UDP to get around the windowing problem. I'm thinking of setting up servers with optimized TCP settings to push big files around data centers but I'm curious to know how others deal with LFN+large transfers. Not very fat or very long. I need to deal with 10GE over 200 ms (or more). These should be pretty easy, but as you realize, you will need large enough windows to keep the traffic in transit from filling the window and stalling the flow. The laws of physics (speed of light) are not forgiving. There is a project from Martin Swaney at U-Delaware (with Guy Almes and Aaron Brown) to do exactly what you are looking for. http://portal.acm.org/citation.cfm?id=1188455.1188714coll=GUIDEdl=GUIDE and http://www.internet2.edu/pubs/phoebus.pdf ESnet, Internet2 and Geant demonstrated it at last November's SuperComputing Conference in Reno. The idea is to use tuned proxies that are close to the source and destination and are optimized for the delay. Local systems can move data through them without dealing with the need to tune for the delay-bandwidth product. Note that this man in the middle may not play well with many security controls which deliberately try to prevent it, so you still may need some adjustments. -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: [EMAIL PROTECTED] Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751 pgpLXPMMiEqMd.pgp Description: PGP signature
Re: Best utilizing fat long pipes and large file transfer
At 06:37 PM 6/12/2008, you wrote: I'm looking for input on the best practices for sending large files over a long fat pipe between facilities (gigabit private circuit, ~20ms RTT). I'd like to avoid modifying TCP windows and options on end hosts where possible (I have a lot of them). I've seen products that work as transfer stations using reliable UDP to get around the windowing problem. I'm thinking of setting up servers with optimized TCP settings to push big files around data centers but I'm curious to know how others deal with LFN+large transfers. In our experience, you can't get to line speed with over 20-30ms of latency using TCP regardless of how much you tweak it. We transfer files across the US with 60-70ms at line speeds with UDP based file transfer programs. There are a number of open source projects out there designed for this purpose. -Robert Tellurian Networks - Global Hosting Solutions Since 1995 http://www.tellurian.com | 888-TELLURIAN | 973-300-9211 Well done is better than well said. - Benjamin Franklin
Re: Best utilizing fat long pipes and large file transfer
The idea is to use tuned proxies that are close to the source and destination and are optimized for the delay. Local systems can move data through them without dealing with the need to tune for the delay-bandwidth product. Note that this man in the middle may not play well with many security controls which deliberately try to prevent it, so you still may need some adjustments. and for those of us who are addicted to simple rsync, or whatever over ssh, you should be aware of the really bad openssh windowing issue. randy
Re: Best utilizing fat long pipes and large file transfer
Take a look at some of the stuff from Aspera. Mark On Thu, Jun 12, 2008 at 03:37:47PM -0700, Sean Knox wrote: Hi, I'm looking for input on the best practices for sending large files over a long fat pipe between facilities (gigabit private circuit, ~20ms RTT). I'd like to avoid modifying TCP windows and options on end hosts where possible (I have a lot of them). I've seen products that work as transfer stations using reliable UDP to get around the windowing problem. I'm thinking of setting up servers with optimized TCP settings to push big files around data centers but I'm curious to know how others deal with LFN+large transfers. thanks, Sean
Re: Best utilizing fat long pipes and large file transfer
Karl Auerbach wrote: Randy Bush wrote: and for those of us who are addicted to simple rsync, or whatever over ssh, you should be aware of the really bad openssh windowing issue. I was not aware of this. Do you have a pointer to a description? see the work by rapier and stevens at psc http://www.psc.edu/networking/projects/hpn-ssh/ this is why rsync starts off at a blazing pace and then takes a serious dive. randuy
Re: Best utilizing fat long pipes and large file transfer
And while I certainly like open source solutions, there are plenty of commercial products that do things to optimize this. Depending on the type of traffic the products do different things. Many of the serial-byte caching variety (e.g. Riverbed/F5) now also do connection/flow optimization and proxying, while many of the network optimizers now are doing serial-byte caching. I also for a while was looking for multicast based file transfer tools, but couldn't find any that were stable. I'd be interested in seeing the names of some of the projects Robert is talking about- perhaps I missed a few when I looked. One thing that is a simple solution? Split the file and then send all the parts at the same time. This helps a fair bit, and is easy to implement. Few things drive home the issues with TCP window scaling better than moving a file via ftp and then via ttcp. Sure, you don't always get all the file, but it does get there fast! --D On Thu, Jun 12, 2008 at 5:02 PM, Randy Bush [EMAIL PROTECTED] wrote: The idea is to use tuned proxies that are close to the source and destination and are optimized for the delay. Local systems can move data through them without dealing with the need to tune for the delay-bandwidth product. Note that this man in the middle may not play well with many security controls which deliberately try to prevent it, so you still may need some adjustments. and for those of us who are addicted to simple rsync, or whatever over ssh, you should be aware of the really bad openssh windowing issue. randy -- -- Darren Bolding -- -- [EMAIL PROTECTED] --
RE: Best utilizing fat long pipes and large file transfer
Hi, I'm looking for input on the best practices for sending large files There are both commercial products (fastcopy) and various free(*) products (bbcp, bbftp, gridftp) that will send large files. While they can take advantage of larger windows they also have the capability of using multiple streams (dealing with the inability to tune the tcp stack). There are, of course, competitors to these products which you should look into. As always, YMWV. Some references: http://www.softlink.com/fastcopy_techie.html (Some parts of NASA seem to like fastcopy) http://nccs.gov/user-support/general-support/data-transfer/bbcp/ (Full disclosure, bbcp was written by someone who sits about 3 meters from where I am sitting, but I cannot find a nice web reference from him about the product, so I am showing a different sites documentation) http://doc.in2p3.fr/bbftp/ (One of the first to use multistream for BaBar) http://www.globus.org/grid_software/data/gridftp.php (Part of the globus toolkit. Somewhat heavier weight if all you want is file transfer.) http://fasterdata.es.net/tools.html (A reference I am surprised Kevin did not point to :-) http://www.slac.stanford.edu/grp/scs/net/talk/ggf5_jul2002/NMWG_GGF5.pdf (A few year old performance evaluation) www.triumf.ca/canarie/amsterdam-test/References/010402-ftp.ppt (Another older performance evaluation) Gary (*) Some are GPL, and some (modified) BSD licenses. Which one is free enough depends on some strongly held beliefs of the validator.
RE: Best utilizing fat long pipes and large file transfer
I'm looking for input on the best practices for sending large files over a long fat pipe between facilities (gigabit private circuit, ~20ms RTT). providing you have RFC1323 type extensions enabled on a semi-decent OS, a 4MB TCP window should be more than sufficient to fill a GbE pipe over 30msec. with a modified TCP stack, that uses TCP window sizes up to 32MB, i've worked with numerous customers to achieve wire-rate GbE async replication for storage-arrays with FCIP. the modifications to TCP were mostly to adjust how it reacts to packet loss, e.g. don't halve the window. the intent of those modifications is that it doesn't use the greater internet but is more suited for private connections within an enterprise customer environment. that is used in production today on many Cisco MDS 9xxx FC switch environments. I'd like to avoid modifying TCP windows and options on end hosts where possible (I have a lot of them). I've seen products that work as transfer stations using reliable UDP to get around the windowing problem. given you don't want to modify all your hosts, you could 'proxy' said TCP connections via 'socat' or 'netcat++'. cheers, lincoln.
Re: Best utilizing fat long pipes and large file transfer
Date: Fri, 13 Jun 2008 09:02:31 +0900 From: Randy Bush [EMAIL PROTECTED] The idea is to use tuned proxies that are close to the source and destination and are optimized for the delay. Local systems can move data through them without dealing with the need to tune for the delay-bandwidth product. Note that this man in the middle may not play well with many security controls which deliberately try to prevent it, so you still may need some adjustments. and for those of us who are addicted to simple rsync, or whatever over ssh, you should be aware of the really bad openssh windowing issue. Actually, OpenSSH has a number of issues that restrict performance. Read about them at http://www.psc.edu/networking/projects/hpn-ssh/ Pittsburgh Supercomputer Center fixed these problems and on FreeBSD, you can get these by replacing the base OpenSSH with the openssh-portable port and select the HPN (High Performance Networking) and OVERWRITE_BASE options. I assume other OSes can deal with this or you can get the patches directly from: http://www.psc.edu/networking/projects/hpn-ssh/openssh-5.0p1-hpn13v3.diff.gz The port may also be built to support SmartCards which we require for authentication. -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: [EMAIL PROTECTED] Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751 pgpGIeZbnjikY.pgp Description: PGP signature
Re: Best utilizing fat long pipes and large file transfer
From: Robert E. Seastrom [EMAIL PROTECTED] Date: Thu, 12 Jun 2008 21:15:49 -0400 Randy Bush [EMAIL PROTECTED] writes: and for those of us who are addicted to simple rsync, or whatever over ssh, you should be aware of the really bad openssh windowing issue. As a user of hpn-ssh for years, I have to wonder if there is any reason (aside from the sheer cussedness for which Theo is infamous) that the window improvements at least from hpn-ssh haven't been backported into mainline openssh? I suppose there might be portability concerns with the multithreaded ciphers, and there's certainly a good argument for not supporting NONE as a cipher type out of the box without a recompile, but there's not much excuse for the fixed size tiny buffers - I mean, it's 2008 already... Theo is known for his amazing stubbornness, but for area involving security and cryptography, I find it hard to say that his conservatism is excessive. Crypto is hard and often it is very non-intuitive. I remember the long discussions on entropy harvesting and seeding in FreeBSD which fortunately has cryptography professionals who could pick every nit and make sure FreeBSD did not end up with Debian-type egg all over its virtual face. Than again, the tiny buffers are silly and I can't imagine any possible security issue there. -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: [EMAIL PROTECTED] Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751 pgpIfmjCKlkJK.pgp Description: PGP signature
