Have a look at this Gist: https://gist.github.com/bradland/5568217
That's the test script I've been using, along with stack traces after the interrupt. The `when :get` case statement writes progress, so we can see exactly how many bytes were transferred, plus we get some insight in to the read loop activity level. Observing the script, we can see the Net::SFTP read loop stop updating when trying to read 4294049792 bytes. Operations::Download#on_read updates progress every time a read finishes, so when it stops, we can assume that something has hung internally. In that same Gist, you can see the stack trace from the script when given SIGINT after it hangs. The resulting stack trace gives further clues that the problem is actually inside Net::SSH. I tried both Session#download! and Session#download. Both show the hang at net-ssh-2.6.7/lib/net/ssh/ruby_compat.rb:30 when given the interrupt. There's a call to IO#select here, which makes a system call to select(2) (we've arrived at the threshold of Ruby's C implementations) and returns an array. If we wanted to keep debugging, we'd want to attach a debugger to the process and have a look at the stack, focusing on arguments passed to IO#select. Given the file offset range we're in, a datatype boundary issue is likely. And at this point, you're really traveling down the rabbit hole. I'm an amateur Rubyist, so we're down to the point where my knowledge withers and I start grabbing at straws. This would make a good ticket for Net::SSH (and maybe Net::SFTP). I'm sure the authors of those libraries would be interested in seeing what we've discovered here. If you're not up for it, I'd be happy to take it to them. If you do open a ticket, be sure to post a link here so we can follow the progress. In the interim, it looks like your best bet for a workaround is shelling out. On Monday, May 13, 2013 12:18:04 AM UTC-4, Chris McCann wrote: > > I haven't tried the #download method though I could easily do that. I was > contemplating setting up a local FTP server on my dev machine to try to > debug this, under the assumption that the file transfer speed would be very > fast running locally. > > I also haven't looked into Net::SSH but I sort of discounted it because it > seemed that a timeout or other issue there would cause the file transfer to > not always fail on a 4GB boundary. It fails at exactly the same number of > bytes each time. Still, there may be something to glean by looking there, > too. > > On Sun, May 12, 2013 at 9:05 PM, bradleyland <[email protected]<javascript:> > > wrote: > >> Yeah, Ruby Fixnum normally automatically casts to Bignum if you exceed >> Fixnum boundaries, so it would be an odd case that a reference counter >> overflow would hang this up. This is really puzzling, and I'm wondering if >> the bug isn't further down the stack, like Net::SSH or something. >> >> I'm curious, have you tried Session#download instead of >> Session#download!. The former is non-blocking, but won't really have any >> effect on your operation unless you specifically need it to block until >> finished. >> >> This really sucks to debug! Waiting for a 4 GB file transfer is like >> waiting for paint to dry. >> >> >> On Sunday, May 12, 2013 11:43:04 PM UTC-4, Chris McCann wrote: >> >>> Thanks very much, Brad. I looked through the source to see if I could >>> find a 32-bit integer somewhere that might be the problem but didn't find >>> any smoking guns. Let me know if you do. >>> >>> On May 12, 2013, at 8:24 PM, bradleyland <[email protected]> wrote: >>> >>> A lot of libraries (Ruby and elsewhere) handle large files rather >>> naively. Fortunately, Net::SFTP looks like it has some smarts for this. The >>> docs for Session#download! say: "If local [the second argument] is omitted, >>> downloads the file to an in-memory buffer and returns the result as a >>> string; otherwise, returns the Net::SFTP::Operations::**Download >>> instance." The next place to look is at Download: http://net-ssh.** >>> github.io/net-sftp/classes/**Net/SFTP/Operations/Download.**html<http://net-ssh.github.io/net-sftp/classes/Net/SFTP/Operations/Download.html>. >>> >>> It looks like you've already been there though, because you're using the >>> event handler. The file is written as the download progresses, so it >>> doesn't look like it's cramming the whole thing in memory. I also watched >>> memory usage while downloading a large file, and the ruby process stays >>> around 30 MB, so I don't think it's an issue related to in-memory buffering. >>> >>> I'm still digging to see if I can find a reason why it would hang at 4 >>> GB. >>> >>> -- >>> -- >>> SD Ruby mailing list >>> [email protected] >>> >>> http://groups.google.com/**group/sdruby<http://groups.google.com/group/sdruby> >>> --- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "SD Ruby" group. >>> To unsubscribe from this topic, visit https://groups.google.com/d/** >>> topic/sdruby/4OhfZ4hklIA/**unsubscribe?hl=en<https://groups.google.com/d/topic/sdruby/4OhfZ4hklIA/unsubscribe?hl=en> >>> . >>> To unsubscribe from this group and all its topics, send an email to >>> sdruby+un...@**googlegroups.com. >>> >>> For more options, visit >>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> >>> . >>> >>> >>> >>> -- >> -- >> SD Ruby mailing list >> [email protected] <javascript:> >> http://groups.google.com/group/sdruby >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "SD Ruby" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/sdruby/4OhfZ4hklIA/unsubscribe?hl=en. >> To unsubscribe from this group and all its topics, send an email to >> [email protected] <javascript:>. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> > > -- -- SD Ruby mailing list [email protected] http://groups.google.com/group/sdruby --- You received this message because you are subscribed to the Google Groups "SD Ruby" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
