Definitely don't try testing this over the Internet - that will take forever to debug. If you're on a Mac just turn on SSH access and create test files of the desired size with the dd command. SFTP uses the SSH protocol so there is nothing else you need to setup.
On Sunday, May 12, 2013, Chris McCann wrote: > I haven't tried the #download method though I could easily do that. I was > contemplating setting up a local FTP server on my dev machine to try to > debug this, under the assumption that the file transfer speed would be very > fast running locally. > > I also haven't looked into Net::SSH but I sort of discounted it because it > seemed that a timeout or other issue there would cause the file transfer to > not always fail on a 4GB boundary. It fails at exactly the same number of > bytes each time. Still, there may be something to glean by looking there, > too. > > On Sun, May 12, 2013 at 9:05 PM, bradleyland > <[email protected]<javascript:_e({}, 'cvml', '[email protected]');> > > wrote: > >> Yeah, Ruby Fixnum normally automatically casts to Bignum if you exceed >> Fixnum boundaries, so it would be an odd case that a reference counter >> overflow would hang this up. This is really puzzling, and I'm wondering if >> the bug isn't further down the stack, like Net::SSH or something. >> >> I'm curious, have you tried Session#download instead of >> Session#download!. The former is non-blocking, but won't really have any >> effect on your operation unless you specifically need it to block until >> finished. >> >> This really sucks to debug! Waiting for a 4 GB file transfer is like >> waiting for paint to dry. >> >> >> On Sunday, May 12, 2013 11:43:04 PM UTC-4, Chris McCann wrote: >> >>> Thanks very much, Brad. I looked through the source to see if I could >>> find a 32-bit integer somewhere that might be the problem but didn't find >>> any smoking guns. Let me know if you do. >>> >>> On May 12, 2013, at 8:24 PM, bradleyland <[email protected]> wrote: >>> >>> A lot of libraries (Ruby and elsewhere) handle large files rather >>> naively. Fortunately, Net::SFTP looks like it has some smarts for this. The >>> docs for Session#download! say: "If local [the second argument] is omitted, >>> downloads the file to an in-memory buffer and returns the result as a >>> string; otherwise, returns the Net::SFTP::Operations::**Download >>> instance." The next place to look is at Download: http://net-ssh.** >>> github.io/net-sftp/classes/**Net/SFTP/Operations/Download.**html<http://net-ssh.github.io/net-sftp/classes/Net/SFTP/Operations/Download.html>. >>> It looks like you've already been there though, because you're using the >>> event handler. The file is written as the download progresses, so it >>> doesn't look like it's cramming the whole thing in memory. I also watched >>> memory usage while downloading a large file, and the ruby process stays >>> around 30 MB, so I don't think it's an issue related to in-memory buffering. >>> >>> I'm still digging to see if I can find a reason why it would hang at 4 >>> GB. >>> >>> -- >>> -- >>> SD Ruby mailing list >>> [email protected] >>> >>> http://groups.google.com/**group/sdruby<http://groups.google.com/group/sdruby> >>> --- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "SD Ruby" group. >>> To unsubscribe from this topic, visit https://groups.google.com/d/** >>> topic/sdruby/4OhfZ4hklIA/**unsubscribe?hl=en<https://groups.google.com/d/topic/sdruby/4OhfZ4hklIA/unsubscribe?hl=en> >>> . >>> To unsubscribe from this group and all its topics, send an email to >>> sdruby+un...@**googlegroups.com. >>> >>> For more options, visit >>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> >>> . >>> >>> >>> >>> -- >> -- >> SD Ruby mailing list >> [email protected] <javascript:_e({}, 'cvml', >> '[email protected]');> >> http://groups.google.com/group/sdruby >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "SD Ruby" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/sdruby/4OhfZ4hklIA/unsubscribe?hl=en. >> To unsubscribe from this group and all its topics, send an email to >> [email protected] <javascript:_e({}, 'cvml', >> 'sdruby%[email protected]');>. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> > > -- > -- > SD Ruby mailing list > [email protected] <javascript:_e({}, 'cvml', > '[email protected]');> > http://groups.google.com/group/sdruby > --- > You received this message because you are subscribed to the Google Groups > "SD Ruby" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:_e({}, 'cvml', > 'sdruby%[email protected]');>. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- -- SD Ruby mailing list [email protected] http://groups.google.com/group/sdruby --- You received this message because you are subscribed to the Google Groups "SD Ruby" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
