On Tue, Mar 05, 2002 at 06:54:41AM -0800, Randal L. Schwartz wrote: > >>>>> "Reinier" == Reinier Post <[EMAIL PROTECTED]> writes: > > Reinier> On Mon, Mar 04, 2002 at 04:33:37PM +0530, kavitha malar wrote: > >> I want to search a text in a website how to do that through perl. > > Reinier> perl -MLWP::Simple -e \ > Reinier> 'getprint http://www.google.com/search?q=$word+site:$site' > > Reinier> I'm serious. (This is what I use to find my own pages.) > > Except now, Google has gotten fairly upset about "automated" page > fetches. There's a thread on use.perl.org about it.
Thanks for the pointer. http://www.google.com/terms_of_service.html is pretty vague about it. As someone in the thread remarked, we are talking about a single query here, for personal use, without even any reformatting of the results. > And last time I checked, Google *specifically* blocks the default > agent type that LWP uses, so you'll get no response. You have > to change the agent type to something with "Mozilla" in it. :) Mmm, I should have checked that. I actually feed the Google query URL to lynx or links. > Gisle - would it be unfair to have a special useragent string > when LWP detects that it is visiting Google? :) Nice idea :) But hidden magic in code is always bad. -- Reinier
