Bensie,
Here's the code:
def download_file!
SFTP.start(@host, @user, password: @password) do |sftp|
@remote_path = File.join(@remote_dir, @remote_file_name)
@extract_path = File.join(@tmp_dir, @remote_file_name)
sftp.download!(remote_path, extract_path, read_size: 64000) do
|event, downloader, *args|
case event
when :open then
# args[0] : file metadata
log_event "Starting download: #{@remote_path}"
when :finish then
log_event "Completed download of #{@remote_path}"
end
end
end
@extract_path
end
I have not been able to get the ruby script to download more than 4GB, and
I have also confirmed that rubyzip can't unzip a file > 4GB (it explicitly
does not support Zip64).
Cheers,
Chris
On Sat, May 11, 2013 at 3:52 PM, James Miller <[email protected]> wrote:
> Can you paste some code for the SFTP download? I'd definitely start by
> confirming whether or not the Ruby script can exceed a 4GB download by
> extracting that out to its own script, since it sounds like you're unsure
> whether the problem is in the download or the unarchive.
>
> Similarly, download the 8.5GB file manually and run it through rubyzip to
> see if it chokes on it.
>
> Once you nail down the culprit, then we can work on fixing it :)
>
>
> On Sat, May 11, 2013 at 3:33 PM, Neal Clark <[email protected]>wrote:
>
>> having absolutely no experience doing any of the things your app is
>> doing, my first thought is… try a compression format that lets you break
>> things up into multiple files, e.g. rar. or try split(1). if it works for <
>> 4gb, keeping it < 4gb seems like an easy "first try" solution.
>>
>> -n
>>
>> On May 11, 2013, at 3:28 PM, Chris McCann <[email protected]> wrote:
>>
>> > I've built a Rails app that retrieves zip files from a remote server and
>> > processes them (loads the CSV files contained therein into a MySQL
>> > database).
>> >
>> > The file retrieval is done using Ruby's Net::SFTP module running in the
>> > background via DelayedJob. It works well, unless the zip file is over
>> 4GB,
>> > in which case it quits pretty much exactly after transferring 4GB of
>> data.
>> > There are no errors whatsoever in the log -- I just see that the file
>> > download was started and never finished.
>> >
>> > The file is actually 8.5GB, and downloading it manually with sftp in the
>> > terminal works just fine.
>> >
>> > I've searched the interwebs for any information about his problem
>> without
>> > any success. The only clue I have is that trying to unzip the files
>> using
>> > rubyzip also failed for large zip files because rubyzip doesn't handle
>> the
>> > Zip64 format, which any zip file over 4GB has. I'm wondering if rubyzip
>> > has a similar 32-bit limit in the size of a file it can handle.
>> Nothing in
>> > the Net::SFTP docs say anything about 32-bit limitations. Thoughts?
>> >
>> > If anyone has a clue, I'm all ears. I'm not sure if it's a ruby
>> problem, a
>> > Rails problem, or perhaps a problem on the other end in which the FTP
>> > client disconnects after 4GB has been transferred. Odd, to say the
>> least.
>> >
>> > Thanks,
>> >
>> > Chris
>> >
>> > --
>> > --
>> > SD Ruby mailing list
>> > [email protected]
>> > http://groups.google.com/group/sdruby
>> > ---
>> > You received this message because you are subscribed to the Google
>> Groups "SD Ruby" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> an email to [email protected].
>> > For more options, visit https://groups.google.com/groups/opt_out.
>> >
>> >
>>
>> --
>> --
>> SD Ruby mailing list
>> [email protected]
>> http://groups.google.com/group/sdruby
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "SD Ruby" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>>
> --
> --
> SD Ruby mailing list
> [email protected]
> http://groups.google.com/group/sdruby
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "SD Ruby" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/sdruby/4OhfZ4hklIA/unsubscribe?hl=en.
> To unsubscribe from this group and all its topics, send an email to
> [email protected].
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>
--
--
SD Ruby mailing list
[email protected]
http://groups.google.com/group/sdruby
---
You received this message because you are subscribed to the Google Groups "SD
Ruby" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/groups/opt_out.
