While this doesn't help solve anything, I can confirm the same issue on my 64-bit setup - files over 4GB just hang forever. Might need to shell out...
On Saturday, May 11, 2013, Chris McCann wrote: > The issue of unzipping it is separate -- I've confirmed rubyzip does not > support Zip64 (> 4GB) files. > > The issue I'm contending with now is that Net::SFTP doesn't seem to > transfer files > 4GB, and given the rubyzip issue, I wondered if they might > have the same root cause, a 32-bit integer limitation. > > On Sat, May 11, 2013 at 3:53 PM, Neal Clark <[email protected]>wrote: > > so it sounds like it's not a Net::SFTP issue? have you tried shelling out > to unzip the files instead of using rubyzip? > > On May 11, 2013, at 3:50 PM, Chris McCann <[email protected]> wrote: > > > I don't have control of the file format -- they're provided by a > > third-party contractor. The normal sftp process is able to download the > > files fine, so I'm suspecting it's a ruby limitation, and I really can't > > ask them to change the compression format for that. > > > > On Sat, May 11, 2013 at 3:33 PM, Neal Clark <[email protected] > >wrote: > > > >> having absolutely no experience doing any of the things your app is > doing, > >> my first thought is… try a compression format that lets you break > things up > >> into multiple files, e.g. rar. or try split(1). if it works for < 4gb, > >> keeping it < 4gb seems like an easy "first try" solution. > >> > >> -n > >> > >> On May 11, 2013, at 3:28 PM, Chris McCann <[email protected]> > wrote: > >> > >>> I've built a Rails app that retrieves zip files from a remote server > and > >>> processes them (loads the CSV files contained therein into a MySQL > >>> database). > >>> > >>> The file retrieval is done using Ruby's Net::SFTP module running in the > >>> background via DelayedJob. It works well, unless the zip file is over > >> 4GB, > >>> in which case it quits pretty much exactly after transferring 4GB of > >> data. > >>> There are no errors whatsoever in the log -- I just see that the file > >>> download was started and never finished. > >>> > >>> The file is actually 8.5GB, and downloading it manually with sftp in > the > >>> terminal works just fine. > >>> > >>> I've searched the interwebs for any information about his problem > without > >>> any success. The only clue I have is that trying to unzip the files > >> using > >>> rubyzip also failed for large zip files because rubyzip doesn't handle > >> the > >>> Zip64 format, which any zip file over 4GB has. I'm wondering if > rubyzip > >>> has a similar 32-bit limit in the size of a file it can handle. > Nothing > >> in > >>> the Net::SFTP docs say anything about 32-bit limitations. Thoughts? > >>> > >>> If anyone has a clue, I'm all ears. I'm not sure if it's a ruby > >> problem, a > >>> Rails problem, or perhaps a problem on the other end in which the FTP > >>> client disconnects after 4GB has been transferred. Odd, to say the > >> least. > >>> > >>> Thanks, > >>> > >>> Chris > >>> > >>> -- > >>> -- > >>> SD Ruby mailing list > >>> [email protected] > >>> http://groups.google.com/group/sdruby > >>> --- > >>> You received this message because you are subscribed to the Google > >> Groups "SD Ruby" group. > >>> To unsubscribe from this group and stop receiving emails from it, send > >> an email to [email protected]. > >>> For more options, visit https://groups.google.com/groups/opt_out. > >>> > >>> > >> > >> -- > >> -- > >> SD Ruby mailing list > >> [email protected] > >> http://groups.google.com/group/sdruby > >> --- > >> You received this message because you are subscribed to a topic in the > >> Google Groups "SD Ruby" group. > > -- > -- > SD Ruby mailing list > [email protected] <javascript:_e({}, 'cvml', > '[email protected]');> > http://groups.google.com/group/sdruby > --- > You received this message because you are subscribed to the Google Groups > "SD Ruby" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:_e({}, 'cvml', > 'sdruby%[email protected]');>. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- -- SD Ruby mailing list [email protected] http://groups.google.com/group/sdruby --- You received this message because you are subscribed to the Google Groups "SD Ruby" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
