Looks like I was unclear. I do not need only extract all links, I must continue fetching and parsing them, not more than N at a time and if possible not less. I need some king of mirroring crawler.
On Sat, Nov 12, 2016 at 6:25 PM, Scott Wiersdorf <[email protected]> wrote: > Here is a complete working example you can run: > > #!/usr/bin/env perl > use Mojolicious::Lite; > > get '/random-urls' => sub { > my $c = shift; > $c->render_later; > > $c->delay( > sub { ## first step > my $delay = shift; > > $c->ua->get('https://www.random.org/bytes/', > $delay->begin); > $c->ua->get('https://www.random.org/integer-sets/', > $delay->begin); > $c->ua->get('https://www.random.org/strings/', > $delay->begin); > $c->ua->get('https://www.random.org/audio-noise/', > $delay->begin); > }, > > sub { ## second step > my $delay = shift; > > for my $dom (map { $_->res->dom } @_) { > say STDERR $dom->at('title')->text; > } > > $c->render(text => "Got all the links"); > } > ); > }; > > app->start; > > > When the results finally all come back from the first step, the second > step will print out the page titles to STDERR, and return to the client > "Got all the links". > > Scott > > On Saturday, November 12, 2016 at 2:47:57 AM UTC-7, Alex Povolotsky wrote: >> >> Hello >> >> Examples of Mojo::UserAgent are limited to fetching a set of files. >> >> But I need a bit more complex thing: I need to parse a site, reading >> pages, parsing it and reading links, using non-blocking UA with, say, 4 >> downloads at a time, no more and if possible no less. >> >> Can someone give me a good example? >> >> Alex >> > -- > You received this message because you are subscribed to a topic in the > Google Groups "Mojolicious" group. > To unsubscribe from this topic, visit https://groups.google.com/d/ > topic/mojolicious/wdp_pgd4e0k/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/mojolicious. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Mojolicious" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/mojolicious. For more options, visit https://groups.google.com/d/optout.
