Good gouge, Bensie, thanks. On Sun, May 12, 2013 at 9:21 PM, James Miller <[email protected]> wrote:
> Definitely don't try testing this over the Internet - that will take > forever to debug. If you're on a Mac just turn on SSH access and create > test files of the desired size with the dd command. SFTP uses the SSH > protocol so there is nothing else you need to setup. > > > On Sunday, May 12, 2013, Chris McCann wrote: > >> I haven't tried the #download method though I could easily do that. I >> was contemplating setting up a local FTP server on my dev machine to try to >> debug this, under the assumption that the file transfer speed would be very >> fast running locally. >> >> I also haven't looked into Net::SSH but I sort of discounted it because >> it seemed that a timeout or other issue there would cause the file transfer >> to not always fail on a 4GB boundary. It fails at exactly the same number >> of bytes each time. Still, there may be something to glean by looking >> there, too. >> >> On Sun, May 12, 2013 at 9:05 PM, bradleyland <[email protected]>wrote: >> >>> Yeah, Ruby Fixnum normally automatically casts to Bignum if you exceed >>> Fixnum boundaries, so it would be an odd case that a reference counter >>> overflow would hang this up. This is really puzzling, and I'm wondering if >>> the bug isn't further down the stack, like Net::SSH or something. >>> >>> I'm curious, have you tried Session#download instead of >>> Session#download!. The former is non-blocking, but won't really have any >>> effect on your operation unless you specifically need it to block until >>> finished. >>> >>> This really sucks to debug! Waiting for a 4 GB file transfer is like >>> waiting for paint to dry. >>> >>> >>> On Sunday, May 12, 2013 11:43:04 PM UTC-4, Chris McCann wrote: >>> >>>> Thanks very much, Brad. I looked through the source to see if I could >>>> find a 32-bit integer somewhere that might be the problem but didn't find >>>> any smoking guns. Let me know if you do. >>>> >>>> On May 12, 2013, at 8:24 PM, bradleyland <[email protected]> wrote: >>>> >>>> A lot of libraries (Ruby and elsewhere) handle large files rather >>>> naively. Fortunately, Net::SFTP looks like it has some smarts for this. The >>>> docs for Session#download! say: "If local [the second argument] is omitted, >>>> downloads the file to an in-memory buffer and returns the result as a >>>> string; otherwise, returns the Net::SFTP::Operations::**Download >>>> instance." The next place to look is at Download: http://net-ssh.** >>>> github.io/net-sftp/classes/**Net/SFTP/Operations/Download.**html<http://net-ssh.github.io/net-sftp/classes/Net/SFTP/Operations/Download.html>. >>>> It looks like you've already been there though, because you're using the >>>> event handler. The file is written as the download progresses, so it >>>> doesn't look like it's cramming the whole thing in memory. I also watched >>>> memory usage while downloading a large file, and the ruby process stays >>>> around 30 MB, so I don't think it's an issue related to in-memory >>>> buffering. >>>> >>>> I'm still digging to see if I can find a reason why it would hang at 4 >>>> GB. >>>> >>>> -- >>>> -- >>>> SD Ruby mailing list >>>> [email protected] >>>> >>>> http://groups.google.com/**group/sdruby<http://groups.google.com/group/sdruby> >>>> --- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "SD Ruby" group. >>>> To unsubscribe from this topic, visit https://groups.google.com/d/** >>>> topic/sdruby/4OhfZ4hklIA/**unsubscribe?hl=en<https://groups.google.com/d/topic/sdruby/4OhfZ4hklIA/unsubscribe?hl=en> >>>> . >>>> To unsubscribe from this group and all its topics, send an email to >>>> sdruby+un...@**googlegroups.com. >>>> >>>> For more options, visit >>>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> >>>> . >>>> >>>> >>>> >>>> -- >>> -- >>> SD Ruby mailing list >>> [email protected] >>> http://groups.google.com/group/sdruby >>> --- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "SD Ruby" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/sdruby/4OhfZ4hklIA/unsubscribe?hl=en. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >>> >>> >> >> -- >> -- >> SD Ruby mailing list >> [email protected] >> http://groups.google.com/group/sdruby >> --- >> You received this message because you are subscribed to the Google Groups >> "SD Ruby" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> > -- > -- > SD Ruby mailing list > [email protected] > http://groups.google.com/group/sdruby > --- > You received this message because you are subscribed to a topic in the > Google Groups "SD Ruby" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/sdruby/4OhfZ4hklIA/unsubscribe?hl=en. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- -- SD Ruby mailing list [email protected] http://groups.google.com/group/sdruby --- You received this message because you are subscribed to the Google Groups "SD Ruby" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
