I'm not sure if it still does but I'm pretty sure net/sftp used to load the entire file in to memory on download. If that is the case perhaps something is watching and killing the process? Does net/sftp allow you to grab the file in chunks?
On Sat, May 11, 2013 at 5:12 PM, Chris McCann <[email protected]> wrote: > Thanks, Bensie, good to have another data point. > > > On May 11, 2013, at 4:55 PM, James Miller <[email protected]> wrote: > > While this doesn't help solve anything, I can confirm the same issue on > my 64-bit setup - files over 4GB just hang forever. Might need to shell > out... > > On Saturday, May 11, 2013, Chris McCann wrote: > >> The issue of unzipping it is separate -- I've confirmed rubyzip does not >> support Zip64 (> 4GB) files. >> >> The issue I'm contending with now is that Net::SFTP doesn't seem to >> transfer files > 4GB, and given the rubyzip issue, I wondered if they might >> have the same root cause, a 32-bit integer limitation. >> >> On Sat, May 11, 2013 at 3:53 PM, Neal Clark <[email protected]>wrote: >> >> so it sounds like it's not a Net::SFTP issue? have you tried shelling out >> to unzip the files instead of using rubyzip? >> >> On May 11, 2013, at 3:50 PM, Chris McCann <[email protected]> wrote: >> >> > I don't have control of the file format -- they're provided by a >> > third-party contractor. The normal sftp process is able to download the >> > files fine, so I'm suspecting it's a ruby limitation, and I really can't >> > ask them to change the compression format for that. >> > >> > On Sat, May 11, 2013 at 3:33 PM, Neal Clark <[email protected] >> >wrote: >> > >> >> having absolutely no experience doing any of the things your app is >> doing, >> >> my first thought is… try a compression format that lets you break >> things up >> >> into multiple files, e.g. rar. or try split(1). if it works for < 4gb, >> >> keeping it < 4gb seems like an easy "first try" solution. >> >> >> >> -n >> >> >> >> On May 11, 2013, at 3:28 PM, Chris McCann <[email protected]> >> wrote: >> >> >> >>> I've built a Rails app that retrieves zip files from a remote server >> and >> >>> processes them (loads the CSV files contained therein into a MySQL >> >>> database). >> >>> >> >>> The file retrieval is done using Ruby's Net::SFTP module running in >> the >> >>> background via DelayedJob. It works well, unless the zip file is over >> >> 4GB, >> >>> in which case it quits pretty much exactly after transferring 4GB of >> >> data. >> >>> There are no errors whatsoever in the log -- I just see that the file >> >>> download was started and never finished. >> >>> >> >>> The file is actually 8.5GB, and downloading it manually with sftp in >> the >> >>> terminal works just fine. >> >>> >> >>> I've searched the interwebs for any information about his problem >> without >> >>> any success. The only clue I have is that trying to unzip the files >> >> using >> >>> rubyzip also failed for large zip files because rubyzip doesn't handle >> >> the >> >>> Zip64 format, which any zip file over 4GB has. I'm wondering if >> rubyzip >> >>> has a similar 32-bit limit in the size of a file it can handle. >> Nothing >> >> in >> >>> the Net::SFTP docs say anything about 32-bit limitations. Thoughts? >> >>> >> >>> If anyone has a clue, I'm all ears. I'm not sure if it's a ruby >> >> problem, a >> >>> Rails problem, or perhaps a problem on the other end in which the FTP >> >>> client disconnects after 4GB has been transferred. Odd, to say the >> >> least. >> >>> >> >>> Thanks, >> >>> >> >>> Chris >> >>> >> >>> -- >> >>> -- >> >>> SD Ruby mailing list >> >>> [email protected] >> >>> http://groups.google.com/group/sdruby >> >>> --- >> >>> You received this message because you are subscribed to the Google >> >> Groups "SD Ruby" group. >> >>> To unsubscribe from this group and stop receiving emails from it, send >> >> an email to [email protected]. >> >>> For more options, visit https://groups.google.com/groups/opt_out. >> >>> >> >>> >> >> >> >> -- >> >> -- >> >> SD Ruby mailing list >> >> [email protected] >> >> http://groups.google.com/group/sdruby >> >> --- >> >> You received this message because you are subscribed to a topic in the >> >> Google Groups "SD Ruby" group. >> >> -- >> -- >> SD Ruby mailing list >> [email protected] >> http://groups.google.com/group/sdruby >> --- >> You received this message because you are subscribed to the Google Groups >> "SD Ruby" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> > -- > -- > SD Ruby mailing list > [email protected] > http://groups.google.com/group/sdruby > --- > You received this message because you are subscribed to a topic in the > Google Groups "SD Ruby" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/sdruby/4OhfZ4hklIA/unsubscribe?hl=en. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > > For more options, visit https://groups.google.com/groups/opt_out. > > > > -- > -- > SD Ruby mailing list > [email protected] > http://groups.google.com/group/sdruby > --- > You received this message because you are subscribed to the Google Groups > "SD Ruby" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- -- SD Ruby mailing list [email protected] http://groups.google.com/group/sdruby --- You received this message because you are subscribed to the Google Groups "SD Ruby" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
