I haven't tried the #download method though I could easily do that. I was contemplating setting up a local FTP server on my dev machine to try to debug this, under the assumption that the file transfer speed would be very fast running locally.
I also haven't looked into Net::SSH but I sort of discounted it because it seemed that a timeout or other issue there would cause the file transfer to not always fail on a 4GB boundary. It fails at exactly the same number of bytes each time. Still, there may be something to glean by looking there, too. On Sun, May 12, 2013 at 9:05 PM, bradleyland <[email protected]> wrote: > Yeah, Ruby Fixnum normally automatically casts to Bignum if you exceed > Fixnum boundaries, so it would be an odd case that a reference counter > overflow would hang this up. This is really puzzling, and I'm wondering if > the bug isn't further down the stack, like Net::SSH or something. > > I'm curious, have you tried Session#download instead of Session#download!. > The former is non-blocking, but won't really have any effect on your > operation unless you specifically need it to block until finished. > > This really sucks to debug! Waiting for a 4 GB file transfer is like > waiting for paint to dry. > > > On Sunday, May 12, 2013 11:43:04 PM UTC-4, Chris McCann wrote: > >> Thanks very much, Brad. I looked through the source to see if I could >> find a 32-bit integer somewhere that might be the problem but didn't find >> any smoking guns. Let me know if you do. >> >> On May 12, 2013, at 8:24 PM, bradleyland <[email protected]> wrote: >> >> A lot of libraries (Ruby and elsewhere) handle large files rather >> naively. Fortunately, Net::SFTP looks like it has some smarts for this. The >> docs for Session#download! say: "If local [the second argument] is omitted, >> downloads the file to an in-memory buffer and returns the result as a >> string; otherwise, returns the Net::SFTP::Operations::**Download >> instance." The next place to look is at Download: http://net-ssh.** >> github.io/net-sftp/classes/**Net/SFTP/Operations/Download.**html<http://net-ssh.github.io/net-sftp/classes/Net/SFTP/Operations/Download.html>. >> It looks like you've already been there though, because you're using the >> event handler. The file is written as the download progresses, so it >> doesn't look like it's cramming the whole thing in memory. I also watched >> memory usage while downloading a large file, and the ruby process stays >> around 30 MB, so I don't think it's an issue related to in-memory buffering. >> >> I'm still digging to see if I can find a reason why it would hang at 4 GB. >> >> -- >> -- >> SD Ruby mailing list >> [email protected] >> >> http://groups.google.com/**group/sdruby<http://groups.google.com/group/sdruby> >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "SD Ruby" group. >> To unsubscribe from this topic, visit https://groups.google.com/d/** >> topic/sdruby/4OhfZ4hklIA/**unsubscribe?hl=en<https://groups.google.com/d/topic/sdruby/4OhfZ4hklIA/unsubscribe?hl=en> >> . >> To unsubscribe from this group and all its topics, send an email to >> sdruby+un...@**googlegroups.com. >> >> For more options, visit >> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> >> . >> >> >> >> -- > -- > SD Ruby mailing list > [email protected] > http://groups.google.com/group/sdruby > --- > You received this message because you are subscribed to a topic in the > Google Groups "SD Ruby" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/sdruby/4OhfZ4hklIA/unsubscribe?hl=en. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- -- SD Ruby mailing list [email protected] http://groups.google.com/group/sdruby --- You received this message because you are subscribed to the Google Groups "SD Ruby" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
