Re: [gentoo-user] Use split to break up a 10GB file binary?
On Mon, Jun 20, 2011 at 05:28:05PM -0700, Mark Knecht wrote: On Mon, Jun 20, 2011 at 2:46 PM, Indi thebeelzebubtrig...@gmail.com wrote: On Mon, Jun 20, 2011 at 08:25:57AM -0700, Mark Knecht wrote: Hi, Is split an appropriate program to use to break a single 10GB file into 100 100MB files to transfer over the net using rsync, and then use cat to reassemble? Is there some better way to do this? Just using rsync by itself would probably be a great deal faster, unless you have some undisclosed reason for wanting to split it up. Hi, Nothing technical that's undisclosed. My original reason was not knowing what rsync did in the case of errors I simply didn't want to start over on such a big file. I figured there was little to lose by stitching it back together are the other end and I could always figure out exactly what file had failed. That said I don't think there's much difference in the speed. In my case (and I think others will have a similar case) my uploads speeds are far lower than download. I get about 8MB/S download but only about 250KB/S upload. It's that low speed that's dominating everything else. When I first tried transferring the big file the intermediate speeds rsync was reporting were very similar. Of course I was referring to the time taken by the extra steps, not the transfer speed. :) You might check man rsync though, it does what you need without splitting and reassembling files.. -- caveat utilitor ♫ ❤ ♫ ❤ ♫ ❤ ♫
Re: [gentoo-user] Use split to break up a 10GB file binary?
On Tuesday 21 June 2011 01:28:05 Mark Knecht wrote: I get about 8MB/S download but only about 250KB/S upload. Should those B's be b's? In other words, bits instead of bytes? Or are you on something much faster than DSL? In either case the S's ought to be s's. On ADSL I get about 900KB/s down and 60KB/s up. -- Rgds Peter
Re: [gentoo-user] Use split to break up a 10GB file binary?
Peter Humphrey wrote: On Tuesday 21 June 2011 01:28:05 Mark Knecht wrote: I get about 8MB/S download but only about 250KB/S upload. Should those B's be b's? In other words, bits instead of bytes? Or are you on something much faster than DSL? In either case the S's ought to be s's. On ADSL I get about 900KB/s down and 60KB/s up. My neighbor has DSL and they get about that. I noticed it the other day when they needed help downloading and installing something. It was ~30Mbs or so and it took only a few seconds to get it. Made my lowly 768KBs look really bad. I just thought back to the old dial-up days and 3Kbs. lol Yea, OOo and KDE updates were lots of fun. I get those case sizes mixed up sometimes too. Maybe it is the pinky that hits the shift key that gets confused. sighs Dale :-) :-)
Re: [gentoo-user] Use split to break up a 10GB file binary?
On Tue, Jun 21, 2011 at 8:11 AM, Peter Humphrey pe...@humphrey.ukfsn.org wrote: On Tuesday 21 June 2011 01:28:05 Mark Knecht wrote: I get about 8MB/S download but only about 250KB/S upload. Should those B's be b's? In other words, bits instead of bytes? Or are you on something much faster than DSL? In either case the S's ought to be s's. On ADSL I get about 900KB/s down and 60KB/s up. -- Rgds Peter Bytes, not bits. I have two ISPs, Comcast Verizon. I needed the redundancy as Comcast for a long time was very unreliable. It's been better in the last year. That said Comcast speeds vary a lot based on what I suppose my neighbors are doing as cable modem is a shared interface. The Verizon 3Mbps service speeds are pretty constant. Measured this morning using Speakeasy's page with San Francisco as the other end: http://www.speakeasy.net/speedtest/ Comcast cable modem - 22.38Mbps download, 4.27Mbps upload Verizon DSL - 2.89Mbps download, 0.74Mbps upload In my home I'm 54Mbps wireless to the Comcast router which might effect things a bit but directly connected to the DSL router. If one goes down I just change a couple of Ethernet cables and start using the other. I suspect that if I went to my wife's machine right now I'd get close to 40Mbps. In reality I find that _real_ transfers to other Gentoo computers using rsync typically run around 50% of those numbers averaged over time. Point taken about s/S. I didn't know there was an accepted standard to use 's'. Cheers, Mark
Re: [gentoo-user] Use split to break up a 10GB file binary?
On Tuesday 21 June 2011 16:36:32 Mark Knecht wrote: Point taken about s/S. I didn't know there was an accepted standard to use 's'. Come to think of it, it may not apply to you (in which case apologies): it's an SI unit. -- Rgds Peter
Re: [gentoo-user] Use split to break up a 10GB file binary?
On Mon, Jun 20, 2011 at 10:25 AM, Mark Knecht markkne...@gmail.com wrote: Hi, Is split an appropriate program to use to break a single 10GB file into 100 100MB files to transfer over the net using rsync, and then use cat to reassemble? I think it should work just fine. I've split huge files into huge chunks and never had any issues. Is there some better way to do this? I wonder if splitting is even necessary; rsync will analyze the file and only transmit the differences, right?. So I'd think that even if the transfer fails, a retry would pick up where it left off (assuming rsync keeps the failed copy). Also check out net-misc/unison. It seems to be designed for just this sort of thing.
Re: [gentoo-user] Use split to break up a 10GB file binary?
On Mon, Jun 20, 2011 at 9:21 AM, Paul Hartman paul.hartman+gen...@gmail.com wrote: On Mon, Jun 20, 2011 at 10:25 AM, Mark Knecht markkne...@gmail.com wrote: Hi, Is split an appropriate program to use to break a single 10GB file into 100 100MB files to transfer over the net using rsync, and then use cat to reassemble? I think it should work just fine. I've split huge files into huge chunks and never had any issues. Is there some better way to do this? I wonder if splitting is even necessary; rsync will analyze the file and only transmit the differences, right?. So I'd think that even if the transfer fails, a retry would pick up where it left off (assuming rsync keeps the failed copy). Also check out net-misc/unison. It seems to be designed for just this sort of thing. I'll check them out Paul. Thanks for the extra ideas. I just tried it as an experiment between two machines here. Using split I broke the 10GB file into 100MB pieces, used rsync to get the pieces to my laptop, and then used cat to reassemble. The size of the results compares to the byte so that looks good. I was unsure whether rsync would restart where it left off or whether it would start over from the beginning. It's one huge file so it would be painful if it did the latter. This way I know my risk is at most 100MB, or maybe 20MB if I break the original up into smaller pieces. Cheers, Mark
Re: [gentoo-user] Use split to break up a 10GB file binary?
On Mon, Jun 20 2011, Paul Hartman wrote: On Mon, Jun 20, 2011 at 10:25 AM, Mark Knecht markkne...@gmail.com wrote: Hi, Is split an appropriate program to use to break a single 10GB file into 100 100MB files to transfer over the net using rsync, and then use cat to reassemble? I think it should work just fine. I've split huge files into huge chunks and never had any issues. Is there some better way to do this? I wonder if splitting is even necessary; rsync will analyze the file and only transmit the differences, right?. So I'd think that even if the transfer fails, a retry would pick up where it left off (assuming rsync keeps the failed copy). I believe that is the --partial option. allan
Re: [gentoo-user] Use split to break up a 10GB file binary?
I was unsure whether rsync would restart where it left off or whether it would start over from the beginning. It's one huge file so it would be painful if it did the latter. This way I know my risk is at most 100MB, or maybe 20MB if I break the original up into smaller pieces. check out the --partial option of rsync, this is what you need :)
Re: [gentoo-user] Use split to break up a 10GB file binary?
On Mon, Jun 20, 2011 at 10:48 AM, Allan Gottlieb gottl...@nyu.edu wrote: On Mon, Jun 20 2011, Paul Hartman wrote: On Mon, Jun 20, 2011 at 10:25 AM, Mark Knecht markkne...@gmail.com wrote: Hi, Is split an appropriate program to use to break a single 10GB file into 100 100MB files to transfer over the net using rsync, and then use cat to reassemble? I think it should work just fine. I've split huge files into huge chunks and never had any issues. Is there some better way to do this? I wonder if splitting is even necessary; rsync will analyze the file and only transmit the differences, right?. So I'd think that even if the transfer fails, a retry would pick up where it left off (assuming rsync keeps the failed copy). I believe that is the --partial option. allan Yes, that looks like what I want. Is there an option to have rsync keep trying if the other end goes down for a while or would I need to put the rsync command into a cron job so that it restarts every hour until it's completed the transfers? I don't see one scanning through the many, many options to rsync. Thanks, Mark
Re: [gentoo-user] Use split to break up a 10GB file binary?
On Monday 20 June 2011 08:25:57 Mark Knecht wrote: Hi, Is split an appropriate program to use to break a single 10GB file into 100 100MB files to transfer over the net using rsync, and then use cat to reassemble? Is there some better way to do this? Maybe not better, but you could use a torrent application that support magnet links to send files across the net. No need to send chop it up in chunks then.
Re: [gentoo-user] Use split to break up a 10GB file binary?
On Mon, Jun 20, 2011 at 12:17 PM, Nils Larsson n...@nerdshack.com wrote: On Monday 20 June 2011 08:25:57 Mark Knecht wrote: Hi, Is split an appropriate program to use to break a single 10GB file into 100 100MB files to transfer over the net using rsync, and then use cat to reassemble? Is there some better way to do this? Maybe not better, but you could use a torrent application that support magnet links to send files across the net. No need to send chop it up in chunks then. Interesting idea. I'll keep in in mind. I've already tested that chopping the file with split and reassembling it locally on another machine works. One advantage to chopping it up is that because I have two ISPs in my house I can send half over each ISP and speed things up considerably. The transfer to the remote machine is now in progress and I've transferred about 20 of the 402 files I need to send so things are progressing splendidly at this time. I loaded unison on one of my local machines and will play with that. Maybe I'll add some form of bittorrent into the mix also. Cheers, Mark
Re: [gentoo-user] Use split to break up a 10GB file binary?
On Mon, Jun 20, 2011 at 08:25:57AM -0700, Mark Knecht wrote Hi, Is split an appropriate program to use to break a single 10GB file into 100 100MB files to transfer over the net using rsync, and then use cat to reassemble? Is there some better way to do this? That's what split was written for. I can't think of anything better. BTW, what type of data is the 10 gig file? If it's text, then consider using zip or bzip2 on each of the fragments before transferring. If it's an already compressed binary format, then don't waste time attempting further compression. -- Walter Dnes waltd...@waltdnes.org
Re: [gentoo-user] Use split to break up a 10GB file binary?
On Mon, Jun 20, 2011 at 1:51 PM, Walter Dnes waltd...@waltdnes.org wrote: On Mon, Jun 20, 2011 at 08:25:57AM -0700, Mark Knecht wrote Hi, Is split an appropriate program to use to break a single 10GB file into 100 100MB files to transfer over the net using rsync, and then use cat to reassemble? Is there some better way to do this? That's what split was written for. I can't think of anything better. BTW, what type of data is the 10 gig file? If it's text, then consider using zip or bzip2 on each of the fragments before transferring. If it's an already compressed binary format, then don't waste time attempting further compression. -- Walter Dnes waltd...@waltdnes.org Hi Walter. It's a Virtualbox VM exported from virtualbox-4.0.8. It consumes about 40GB on disk, compressed by Virtualbox to about 10GB by their 'Export Appliance' feature. I actually did try compressing the exported 10GB file with gzip bzip2 before splitting it. None of those provided any compression. I didn't try the spilt outputs as I figured they are just binary chunks and wouldn't compress either. Thanks for the info. Cheers, Mark
Re: [gentoo-user] Use split to break up a 10GB file binary?
On Mon, Jun 20, 2011 at 08:25:57AM -0700, Mark Knecht wrote: Hi, Is split an appropriate program to use to break a single 10GB file into 100 100MB files to transfer over the net using rsync, and then use cat to reassemble? Is there some better way to do this? Just using rsync by itself would probably be a great deal faster, unless you have some undisclosed reason for wanting to split it up. -- caveat utilitor ♫ ❤ ♫ ❤ ♫ ❤ ♫
Re: [gentoo-user] Use split to break up a 10GB file binary?
On Monday 20 Jun 2011 22:06:59 Mark Knecht wrote: On Mon, Jun 20, 2011 at 1:51 PM, Walter Dnes waltd...@waltdnes.org wrote: On Mon, Jun 20, 2011 at 08:25:57AM -0700, Mark Knecht wrote Hi, Is split an appropriate program to use to break a single 10GB file into 100 100MB files to transfer over the net using rsync, and then use cat to reassemble? Is there some better way to do this? That's what split was written for. I can't think of anything better. BTW, what type of data is the 10 gig file? If it's text, then consider using zip or bzip2 on each of the fragments before transferring. If it's an already compressed binary format, then don't waste time attempting further compression. -- Walter Dnes waltd...@waltdnes.org Hi Walter. It's a Virtualbox VM exported from virtualbox-4.0.8. It consumes about 40GB on disk, compressed by Virtualbox to about 10GB by their 'Export Appliance' feature. I actually did try compressing the exported 10GB file with gzip bzip2 before splitting it. None of those provided any compression. I didn't try the spilt outputs as I figured they are just binary chunks and wouldn't compress either. Thanks for the info. You could try: wget -c -t 0 to retry (forever) should the transfer fail for some reason and not bother with splitting the file. -- Regards, Mick signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Use split to break up a 10GB file binary?
On Mon, Jun 20, 2011 at 2:46 PM, Indi thebeelzebubtrig...@gmail.com wrote: On Mon, Jun 20, 2011 at 08:25:57AM -0700, Mark Knecht wrote: Hi, Is split an appropriate program to use to break a single 10GB file into 100 100MB files to transfer over the net using rsync, and then use cat to reassemble? Is there some better way to do this? Just using rsync by itself would probably be a great deal faster, unless you have some undisclosed reason for wanting to split it up. Hi, Nothing technical that's undisclosed. My original reason was not knowing what rsync did in the case of errors I simply didn't want to start over on such a big file. I figured there was little to lose by stitching it back together are the other end and I could always figure out exactly what file had failed. That said I don't think there's much difference in the speed. In my case (and I think others will have a similar case) my uploads speeds are far lower than download. I get about 8MB/S download but only about 250KB/S upload. It's that low speed that's dominating everything else. When I first tried transferring the big file the intermediate speeds rsync was reporting were very similar. Cheers, Mark
