I am scraping a website that provides a list. To get to the next page, I
increment the page number in the URL. No problems with this. But I notice
that when I navigate to the page in my browser, there is a slight delay
while the page goes to the appropriate anchor tag. When I scrape the page
programmatically, I keep getting the same page over and over again. I
suspect that this is because Mojolicious is retrieving the page before the
javascript (or whatever) has a chance to move to the "next" page.
Is there a way to tell Mojolicious to wait until after the page finishes
rendering/moving/whatever to the desired anchor tag?
My GET request:
for(my $i = 1; $i < 20; $i++) {
my $url = '
http://www.forbes.com/global2000/list/#page:'.$pageNumber.'_sort:0_direction:asc_search:_filter:All%20industries_filter:All%20countries_filter:All%20states
';
my $tx = $ua->get($url);
if($tx->success) {
# ... do some stuff
}
}
Any suggestions?
--
You received this message because you are subscribed to the Google Groups
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
