I do not need an extra job manager with forks, I need to employ Mojo's event-based nonblocking I/O
On Mon, Nov 14, 2016 at 3:46 PM, Alexander Karelas <[email protected]> wrote: > I'm not an expert, but what you're asking for looks like a job for Minion: > https://metacpan.org/pod/Minion > > On 14/11/16 14:41, Alex Povolotsky wrote: > > Looks like I was unclear. I do not need only extract all links, I must > continue fetching and parsing them, not more than N at a time and if > possible not less. I need some king of mirroring crawler. > > On Sat, Nov 12, 2016 at 6:25 PM, Scott Wiersdorf < > [email protected]> wrote: > >> Here is a complete working example you can run: >> >> #!/usr/bin/env perl >> use Mojolicious::Lite; >> >> get '/random-urls' => sub { >> my $c = shift; >> $c->render_later; >> >> $c->delay( >> sub { ## first step >> my $delay = shift; >> >> $c->ua->get('https://www.random.org/bytes/', >> $delay->begin); >> $c->ua->get('https://www.random.org/integer-sets/', >> $delay->begin); >> $c->ua->get('https://www.random.org/strings/', >> $delay->begin); >> $c->ua->get('https://www.random.org/audio-noise/', >> $delay->begin); >> }, >> >> sub { ## second step >> my $delay = shift; >> >> for my $dom (map { $_->res->dom } @_) { >> say STDERR $dom->at('title')->text; >> } >> >> $c->render(text => "Got all the links"); >> } >> ); >> }; >> >> app->start; >> >> >> When the results finally all come back from the first step, the second >> step will print out the page titles to STDERR, and return to the client >> "Got all the links". >> >> Scott >> >> On Saturday, November 12, 2016 at 2:47:57 AM UTC-7, Alex Povolotsky >> wrote: >>> >>> Hello >>> >>> Examples of Mojo::UserAgent are limited to fetching a set of files. >>> >>> But I need a bit more complex thing: I need to parse a site, reading >>> pages, parsing it and reading links, using non-blocking UA with, say, 4 >>> downloads at a time, no more and if possible no less. >>> >>> Can someone give me a good example? >>> >>> Alex >>> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Mojolicious" group. >> To unsubscribe from this topic, visit https://groups.google.com/d/to >> pic/mojolicious/wdp_pgd4e0k/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at https://groups.google.com/group/mojolicious. >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to the Google Groups > "Mojolicious" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/mojolicious. > For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Mojolicious" group. > To unsubscribe from this topic, visit https://groups.google.com/d/ > topic/mojolicious/wdp_pgd4e0k/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/mojolicious. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Mojolicious" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/mojolicious. For more options, visit https://groups.google.com/d/optout.
