Re: Getting data from external URL
Hmmm Looking at _trivial_http_get: if ($code =~ /^30[1237]/ && $buf =~ /\012Location:\s*(\S+)/) { # redirect So it certainly seems like it's *trying to handle it. As I recall (it was a late night when I had an application that wasn't working), I had single stepped down into the guts of LWP::Simple and realized that it was returning a failure indicator when encountering a 302 status. I had assumed that this was intended behavior, but now that I look at the pod of what we've currently got installed (1.32), it sure seems like it should work. I'll look into this a bit to see if I can recreate it, but for now, let's chalk it up to either (1) something that's been fixed since the version that I was using at the time, or (2) I'm just out of my head. More likely the latter... Sorry for the confusion. On 29 Aug 2000, Gisle Aas wrote: > Steve Reppucci <[EMAIL PROTECTED]> writes: > > > Just a word of warning: LWP::Simple doesn't follow redirects (at least, > > the last I checked, not sure if it's been changed in the 3 or 4 > > months since I've last used it...), > > If it does not follow redirects then that is a bug. Do you have a > test case? > > Not much has changed in the last 3 or 4 months either. > > Regards, > Gisle > =-=-=-=-=-=-=-=-=-=- My God! What have I done? -=-=-=-=-=-=-=-=-=-= Steve Reppucci [EMAIL PROTECTED] | Logical Choice Software http://logsoft.com/ | 508/958-0183 Be Open |
Re: Getting data from external URL
Steve Reppucci <[EMAIL PROTECTED]> writes: > Just a word of warning: LWP::Simple doesn't follow redirects (at least, > the last I checked, not sure if it's been changed in the 3 or 4 > months since I've last used it...), If it does not follow redirects then that is a bug. Do you have a test case? Not much has changed in the last 3 or 4 months either. Regards, Gisle
Re: Getting data from external URL
On Mon, 28 Aug 2000, Jeff Beard wrote: > I just wrote a function yesterday that uses IO::Socket to interact with an > outside CGI program. It appears to work the way I want but this is my first > foray into writing TCP client code. > > It's basically this: > > sub tcp_client { > my ( $rhost, $rport, $query_string ) = @_; > > my $socket = IO::Socket::INET->new( PerrAddr => $rhost, > PeerPort => $rport, > Proto=> "tcp", > Type => SOCK_STREAM) > or die "Couldn't connect to $rhost:$rport : $!\n"; > > print $socket "GET /programname?$query_string\n"; At the very least send a HTTP/1.x part and a Host: header. Otherwise virtual hosts are hosed. -- Fastnet Software Ltd. High Performance Web Specialists Providing mod_perl, XML, Sybase and Oracle solutions Email for training and consultancy availability. http://sergeant.org | AxKit: http://axkit.org
Re: Getting data from external URL
I just wrote a function yesterday that uses IO::Socket to interact with an outside CGI program. It appears to work the way I want but this is my first foray into writing TCP client code. It's basically this: sub tcp_client { my ( $rhost, $rport, $query_string ) = @_; my $socket = IO::Socket::INET->new( PerrAddr => $rhost, PeerPort => $rport, Proto=> "tcp", Type => SOCK_STREAM) or die "Couldn't connect to $rhost:$rport : $!\n"; print $socket "GET /programname?$query_string\n"; my $response; while ( <$socket> ) { $response .= $_ } close($socket); return \$answer; } --Jeff At 06:58 AM 8/28/00 -0400, Steve Reppucci wrote: >Just a word of warning: LWP::Simple doesn't follow redirects (at least, >the last I checked, not sure if it's been changed in the 3 or 4 >months since I've last used it...), so you need to be certain that you're >using it in a context where you're fetching something that won't return a >redirect. > >HTH... > >On Sat, 26 Aug 2000, Stas Bekman wrote: > > > On Sat, 26 Aug 2000, Rodney Broom wrote: > > > > > OK, lots of banter... > > > > > > Hey V, if you are on a *NIX system, then this is a fast way: > > > > > > open U, "lynx -source www.some.url.dom |"; > > > $data = join '', ; > > > > > > There, you're finished. Admittedly, this isn't terribly efficiant, > but it works > > > just fine and has short devel time. > > > > This one is much more efficient and requires even less coding: > > > > use LWP::Simple; > > $content = get("http://www.sn.no/") > > > > And it doesn't require you to be on any particular OS, as far as I know. > > > > see perldoc LWP::Simple and as advised by many others LWP::UserAgent for > > more advanced uses. > > >=-=-=-=-=-=-=-=-=-=- My God! What have I done? -=-=-=-=-=-=-=-=-=-= >Steve Reppucci [EMAIL PROTECTED] | >Logical Choice Software http://logsoft.com/ | >508/958-0183 Be Open | Jeff Beard __ Web:www.cyberxape.com Email: jeff at cyberxape dot com Location: Boulder, Colorado, USA
Re: Getting data from external URL
On Mon, 28 Aug 2000, Steve Reppucci wrote: > > Just a word of warning: LWP::Simple doesn't follow redirects (at least, > the last I checked, not sure if it's been changed in the 3 or 4 > months since I've last used it...), so you need to be certain that you're > using it in a context where you're fetching something that won't return a > redirect. That's why I've mentioned this :) LWP::UserAgent for more advanced uses. > > HTH... > > On Sat, 26 Aug 2000, Stas Bekman wrote: > > > On Sat, 26 Aug 2000, Rodney Broom wrote: > > > > > OK, lots of banter... > > > > > > Hey V, if you are on a *NIX system, then this is a fast way: > > > > > > open U, "lynx -source www.some.url.dom |"; > > > $data = join '', ; > > > > > > There, you're finished. Admittedly, this isn't terribly efficiant, but it works > > > just fine and has short devel time. > > > > This one is much more efficient and requires even less coding: > > > > use LWP::Simple; > > $content = get("http://www.sn.no/") > > > > And it doesn't require you to be on any particular OS, as far as I know. > > > > see perldoc LWP::Simple and as advised by many others LWP::UserAgent for > > more advanced uses. > > > =-=-=-=-=-=-=-=-=-=- My God! What have I done? -=-=-=-=-=-=-=-=-=-= > Steve Reppucci [EMAIL PROTECTED] | > Logical Choice Software http://logsoft.com/ | > 508/958-0183 Be Open | > > _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://apachetoday.com http://jazzvalley.com http://singlesheaven.com http://perlmonth.com perl.org apache.org
Re: Getting data from external URL
Just a word of warning: LWP::Simple doesn't follow redirects (at least, the last I checked, not sure if it's been changed in the 3 or 4 months since I've last used it...), so you need to be certain that you're using it in a context where you're fetching something that won't return a redirect. HTH... On Sat, 26 Aug 2000, Stas Bekman wrote: > On Sat, 26 Aug 2000, Rodney Broom wrote: > > > OK, lots of banter... > > > > Hey V, if you are on a *NIX system, then this is a fast way: > > > > open U, "lynx -source www.some.url.dom |"; > > $data = join '', ; > > > > There, you're finished. Admittedly, this isn't terribly efficiant, but it works > > just fine and has short devel time. > > This one is much more efficient and requires even less coding: > > use LWP::Simple; > $content = get("http://www.sn.no/") > > And it doesn't require you to be on any particular OS, as far as I know. > > see perldoc LWP::Simple and as advised by many others LWP::UserAgent for > more advanced uses. =-=-=-=-=-=-=-=-=-=- My God! What have I done? -=-=-=-=-=-=-=-=-=-= Steve Reppucci [EMAIL PROTECTED] | Logical Choice Software http://logsoft.com/ | 508/958-0183 Be Open |
Re: Getting data from external URL
Hello, Thanks everyone for such a simple solutions. I think I will use followig method as it seems much simpler. Thanks Vijay - Original Message - From: "Rodney Broom" <[EMAIL PROTECTED]> To: "Stas Bekman" <[EMAIL PROTECTED]> Cc: "Vijay" <[EMAIL PROTECTED]>; "mod_perl Maillinglist" <[EMAIL PROTECTED]> Sent: Saturday, August 26, 2000 10:24 AM Subject: Re: Getting data from external URL > SB> This one is much more efficient and requires even less coding: > SB> use LWP::Simple; > SB> $content = get("http://www.sn.no/") > > Even better, thanks Stas. > > >
Re: Getting data from external URL
SB> This one is much more efficient and requires even less coding: SB> use LWP::Simple; SB> $content = get("http://www.sn.no/") Even better, thanks Stas.
Re: Getting data from external URL
On Sat, 26 Aug 2000, Rodney Broom wrote: > OK, lots of banter... > > Hey V, if you are on a *NIX system, then this is a fast way: > > open U, "lynx -source www.some.url.dom |"; > $data = join '', ; > > There, you're finished. Admittedly, this isn't terribly efficiant, but it works > just fine and has short devel time. This one is much more efficient and requires even less coding: use LWP::Simple; $content = get("http://www.sn.no/") And it doesn't require you to be on any particular OS, as far as I know. see perldoc LWP::Simple and as advised by many others LWP::UserAgent for more advanced uses. _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://apachetoday.com http://jazzvalley.com http://singlesheaven.com http://perlmonth.com perl.org apache.org
Re: Getting data from external URL
OK, lots of banter... Hey V, if you are on a *NIX system, then this is a fast way: open U, "lynx -source www.some.url.dom |"; $data = join '', ; There, you're finished. Admittedly, this isn't terribly efficiant, but it works just fine and has short devel time. Rodney Broom
RE: Getting data from external URL
Hello Vijay, You can make simple HTTP GET or POST requests using the LWP and HTTP modules. I recommend that you read the ActivePerl help files. I have attached the one you really need but it probably won't make sense unless you read up on LWP. # A small sample follows: use LWP::UserAgent; # use HTTP::Request::Common qw(POST); $ua = new LWP::UserAgent; $ua->agent("Netscape 3.0"); $page_url = 'http://www.1freestuff.com/cgi-bin/topsites/topsites.cgi?puzzler'; $response = &get($page_url); # Do a GET. # print $response->as_string; # Get the NEW form number: $content = $response->content; $content =~ m/(value=")(\d.*)(")/ig; $number = $2; print "The magic number is: " . $number; sub get { my $page_url = $_[0]; my $request = new HTTP::Request(GET, "$page_url"); my $response = $ua->request($request); return $response; }; -Original Message- From: Vijay [mailto:[EMAIL PROTECTED]] Sent: Friday, August 25, 2000 11:15 PM To: [EMAIL PROTECTED]; mod_perl Maillinglist Subject: Getting data from external URL Hello, I want to get data from an external url in my perl program (either thru Embperl Execute or directly from perl). What I need is like this. There is a URL which gives some information in text format. I want to get that into a variable or file using perl and using my own html templates, I want to show that data. If anyone has done something like this, please let me know. Thanks Vijay Title: HTTP::Request::Common - Construct common HTTP::Request objects HTTP::Request::Common - Construct common HTTP::Request objects NAME SUPPORTED PLATFORMS SYNOPSIS DESCRIPTION SEE ALSO COPYRIGHT NAME HTTP::Request::Common - Construct common HTTP::Request objects SUPPORTED PLATFORMS Linux Solaris Windows SYNOPSIS use HTTP::Request::Common; $ua = LWP::UserAgent->new; $ua->request(GET 'http://www.sn.no/'); $ua->request(POST 'http://somewhere/foo', [foo => bar, bar => foo]); DESCRIPTION This module provide functions that return newly created HTTP::Request objects. These functions are usually more convenient to use than the standard HTTP::Request constructor for these common requests. The following functions are provided. GET $url, Header => Value,... The GET() function returns a HTTP::Request object initialized with the GET method and the specified URL. Without additional arguments it is exactly equivalent to the following call HTTP::Request->new(GET => $url) but is less cluttered. It also reads better when used together with the LWP::UserAgent->request() method: my $ua = new LWP::UserAgent; my $res = $ua->request(GET 'http://www.sn.no/') if ($res->is_success) { ... You can also initialize header values in the request by specifying some key/value pairs as optional arguments. For instance: $ua->request(GET 'http://www.sn.no/', If_Match => 'foo', From => '[EMAIL PROTECTED]', ); A header key called 'Content' is special and when seen the value will initialize the content part of the request instead of setting a header. HEAD $url, [Header => Value,...] Like GET() but the method in the request is HEAD. PUT $url, [Header => Value,...] Like GET() but the method in the request is PUT. POST $url, [$form_ref], [Header => Value,...] This works mostly like GET() with POST as the method, but this function also takes a second optional array or hash reference parameter ($form_ref). This argument can be used to pass key/value pairs for the form content. By default we will initialize a request using the application/x-www-form-urlencoded content type. This means that you can emulate a HTML POSTing like this: POST 'http://www.perl.org/survey.cgi', [ name => 'Gisle Aas', email => '[EMAIL PROTECTED]', gender => 'M', born => '1964', perc => '3%', ]; This will create a HTTP::Request object that looks like this: POST http://www.perl.org/survey.cgi Content-Length: 66 Content-Type: application/x-www-form-urlencoded name=Gisle%20Aas&email=gisle%40aas.no&gender=M&born=1964&perc=3%25 The POST method also supports the multipart/form-data content used for Form-based File Upload as specified in RFC 1867. You trigger this content format by specifying a content type of 'form-data' as one of the request headers. If one of the values in the $form_ref is an array reference, then it is treated as a file part specification with the following interpretation: [ $file, $filename, Header => Value... ] The first value in the array ($file) is the name of a file to open. This file will be read and its content placed in the request. The routine will croak if the file can't be opened. Use an undef as $file value if you want to specify the content directly. The $filename is the filename to report in the request. If this
Re: Getting data from external URL
Hello, > > I want to get data from an external url in my perl program (either thru > Embperl Execute or directly from perl). What I need is like this. > > There is a URL which gives some information in text format. I want to get > that into a variable or file using perl and using my own html templates, I > want to show that data. > > If anyone has done something like this, please let me know. > HTML::Embperl::ProxyInput maybe (ab)used to fetch the data from the remote host Gerald - Gerald Richterecos electronic communication services gmbh Internetconnect * Webserver/-design/-datenbanken * Consulting Post: Tulpenstrasse 5 D-55276 Dienheim b. Mainz E-Mail: [EMAIL PROTECTED] Voice:+49 6133 925151 WWW:http://www.ecos.de Fax: +49 6133 925152 -
Re: Getting data from external URL
Apache::Request simply maps the Apache object into an Apache::Request object (I know that sounds like double-speak, but it's late), and adds some extra methods. I don't think that's what Vijay's original question is all about. What's needed is an HTTP::Request object. Look at LWP::UserAgent which brings together HTTP::Request and HTTP::Response objects and you also should grab HTTP::Headers. Like I said, it's late. I think that whole collection ball of wax is in libwww-perl available on CPAN. Other than the parsing and redisplaying that Vijay asked about, that's pretty much what a proxy server does, and there's a pretty detailed example or two in the Eagle book. As far as the parsing that response data, a mixture of good ole perl technique and HTML::Parser should be sufficient. -- Rob --On 08/25/00 21:33:14 -0700 Yann Ramin <[EMAIL PROTECTED]> wrote: > If you're in a modperl enviroment, and don't use CGI.pm, try using > Apache::Request (I'm not a fan of using Apache.pm for too much). Has > anyone ever considered making a wrapper for all the modules which get > back data from the request (roll Apache::Request, Apache::Cookie, etc, > into one)? Seems trivial. > > Yann > > > Vijay wrote: >> >> Hello, >> >> I want to get data from an external url in my perl program (either >> thru Embperl Execute or directly from perl). What I need is like >> this. >> >> There is a URL which gives some information in text format. I want >> to get that into a variable or file using perl and using my own html >> templates, I want to show that data. >> >> If anyone has done something like this, please let me know. >> >> Thanks >> >> Vijay > > -- > > > Yann Ramin[EMAIL PROTECTED] > Atrus Trivalie Productionswww.redshift.com/~yramin > Monterey High IT www.montereyhigh.com > ICQ 46805627 > AIM oddatrus > Marina, CA > > IRM Developer Network Toaster Developer > SNTS Developer KLevel Developer > > (yes, this .signature is way too big) > > "All cats die. Socrates is dead. Therefore Socrates is a cat." >- The Logician > >THE STORY OF CREATION > > In the beginning there was data. The data was without form and null, > and darkness was upon the face of the console; and the Spirit of IBM > was moving over the face of the market. And DEC said, "Let there be > registers"; and there were registers. And DEC saw that they carried; > and DEC seperated the data from the instructions. DEC called the data > Stack, and the instructions they called Code. And there was evening > and there was a maorning, one interrupt... >-- Rico Tudor > > William Safire's Rules for Writers: > > Remembe _ _ _ _ __ _ _ _ _ /\_\_\_\_\/\_\ /\_\_\_\_\_\ /\/_/_/_/_/ /\/_/ \/_/_/_/_/_/ QUIDQUID LATINE DICTUM SIT, /\/_/__\/_/ __/\/_//\/_/ PROFUNDUM VIDITUR /\/_/_/_/_/ /\_\ /\/_//\/_/ /\/_/ \/_/ /\/_/_/\/_//\/_/ (Whatever is said in Latin \/_/ \/_/ \/_/_/_/_/ \/_/ appears profound) Rob Tanner McMinnville, Oregon [EMAIL PROTECTED]
Re: Getting data from external URL
If you're in a modperl enviroment, and don't use CGI.pm, try using Apache::Request (I'm not a fan of using Apache.pm for too much). Has anyone ever considered making a wrapper for all the modules which get back data from the request (roll Apache::Request, Apache::Cookie, etc, into one)? Seems trivial. Yann Vijay wrote: > > Hello, > > I want to get data from an external url in my perl program (either thru > Embperl Execute or directly from perl). What I need is like this. > > There is a URL which gives some information in text format. I want to get > that into a variable or file using perl and using my own html templates, I > want to show that data. > > If anyone has done something like this, please let me know. > > Thanks > > Vijay -- Yann Ramin [EMAIL PROTECTED] Atrus Trivalie Productions www.redshift.com/~yramin Monterey High ITwww.montereyhigh.com ICQ 46805627 AIM oddatrus Marina, CA IRM Developer Network Toaster Developer SNTS Developer KLevel Developer (yes, this .signature is way too big) "All cats die. Socrates is dead. Therefore Socrates is a cat." - The Logician THE STORY OF CREATION In the beginning there was data. The data was without form and null, and darkness was upon the face of the console; and the Spirit of IBM was moving over the face of the market. And DEC said, "Let there be registers"; and there were registers. And DEC saw that they carried; and DEC seperated the data from the instructions. DEC called the data Stack, and the instructions they called Code. And there was evening and there was a maorning, one interrupt... -- Rico Tudor William Safire's Rules for Writers: Remembe