I'm not an expert, but what you're asking for looks like a job for
Minion: https://metacpan.org/pod/Minion
On 14/11/16 14:41, Alex Povolotsky wrote:
Looks like I was unclear. I do not need only extract all links, I must
continue fetching and parsing them, not more than N at a time and if
possible not less. I need some king of mirroring crawler.
On Sat, Nov 12, 2016 at 6:25 PM, Scott Wiersdorf
<[email protected] <mailto:[email protected]>> wrote:
Here is a complete working example you can run:
#!/usr/bin/env perl
use Mojolicious::Lite;
get '/random-urls' => sub {
my $c = shift;
$c->render_later;
$c->delay(
sub { ## first step
my $delay = shift;
$c->ua->get('https://www.random.org/bytes/
<https://www.random.org/bytes/>', $delay->begin);
$c->ua->get('https://www.random.org/integer-sets/
<https://www.random.org/integer-sets/>', $delay->begin);
$c->ua->get('https://www.random.org/strings/
<https://www.random.org/strings/>', $delay->begin);
$c->ua->get('https://www.random.org/audio-noise/
<https://www.random.org/audio-noise/>', $delay->begin);
},
sub { ## second step
my $delay = shift;
for my $dom (map { $_->res->dom } @_) {
say STDERR $dom->at('title')->text;
}
$c->render(text => "Got all the links");
}
);
};
app->start;
When the results finally all come back from the first step, the
second step will print out the page titles to STDERR, and return
to the client "Got all the links".
Scott
On Saturday, November 12, 2016 at 2:47:57 AM UTC-7, Alex
Povolotsky wrote:
Hello
Examples of Mojo::UserAgent are limited to fetching a set of
files.
But I need a bit more complex thing: I need to parse a site,
reading pages, parsing it and reading links, using
non-blocking UA with, say, 4 downloads at a time, no more and
if possible no less.
Can someone give me a good example?
Alex
--
You received this message because you are subscribed to a topic in
the Google Groups "Mojolicious" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/mojolicious/wdp_pgd4e0k/unsubscribe
<https://groups.google.com/d/topic/mojolicious/wdp_pgd4e0k/unsubscribe>.
To unsubscribe from this group and all its topics, send an email
to [email protected]
<mailto:[email protected]>.
To post to this group, send email to [email protected]
<mailto:[email protected]>.
Visit this group at https://groups.google.com/group/mojolicious
<https://groups.google.com/group/mojolicious>.
For more options, visit https://groups.google.com/d/optout
<https://groups.google.com/d/optout>.
--
You received this message because you are subscribed to the Google
Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to [email protected]
<mailto:[email protected]>.
To post to this group, send email to [email protected]
<mailto:[email protected]>.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
