Robin Sheat wrote: > On Tuesday 12 September 2006 08:17, Rob Dixon wrote: > >>One you have downloaded the remote file to memory or to local file storage >>you can then open either of them and access them through a filehandle. Will >>that do? >> >>Tell us a little more and we may be able to help better. > > Imagine I'm downloading a 10Gb file. I can't put it in memory, there's not > enough space. I don't want to write it to disk because I might fill the disk, > and it also slows the whole operation down. What I want to be be able to do > is pass it, line-by-line (or n-byte block by n-byte block is probably better > in case it's not text) as it's downloading to another program that is the one > that's processing it. There's no need, and perhaps no possibility, to have > all the content pre-downloaded in one place, but it can be processed on the > fly. > > However, to do this, I need it as a handle. It should be possible, as opening > a socket gives you a filehandle (well, something that behaves like one). > > For more concreteness, the test case I wrote to test the bit that the content > is being sent to is: > open(my $infh, "<t/svntest1.dump") or die "Can't open file \ > t/svntest1.dump: $!\n"; > $inst->restore($infh); > This works fine coming from a file, naturally. But say I want to stream a 10Gb > file into the restore method. Then I need a handle to pass, so that the > restore method doesn't have to have the whole content in memory. > > Hopefully this explains what I'm after.
I think so Robin, thanks. But I'm still not clear whether you really need to retrieve the data through a filehandle; surely any suitable means of IPC will do? What may help is that LWP allows for a callback to be specified in the get() call, so that the downloaded data can be passed in chunks to a user-written subroutine as it arrives. Use $agent->get($url, ':content_cb' => \&callback); and your callback() subroutine will be passed chunks of data as they arrive. You can also specify ':read_size_hint' => 1024 or similar to suggest how big you'f like your data blocks to be. Readl about all of this in the documentation for LWP::UserAgent under the description of the get() method. Now I start to get a little out of my depth here, as I'm essentially a Windows man, but I suspect that your program could write these chunks to a pipe which is then opened and read by the processing program through a filehandle. Another suggestion is to look at IO::All::LWP, which extends the IO::All module to handle LWP connections. However, I have no experience of this module and can help you no further with it. I hope some of this helps you. Rob -- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] <http://learn.perl.org/> <http://learn.perl.org/first-response>
