I agree with Alexander Karelas: your question is a textbook job queue application.
It is possible to solve this using only non-blocking user agents, but in either case you'll have to write some kind of controller that keeps track of "in-flight" requests, a queue of pending requests, and something to manage the queue. It's not a trivial application to do it well—you're essentially implementing Minion. This is likely not the right forum to ask for someone to write this for you. Scott On Monday, November 14, 2016 at 5:53:50 AM UTC-7, Alex Povolotsky wrote: > > I do not need an extra job manager with forks, I need to employ Mojo's > event-based nonblocking I/O > > On Mon, Nov 14, 2016 at 3:46 PM, Alexander Karelas <[email protected] > <javascript:>> wrote: > >> I'm not an expert, but what you're asking for looks like a job for >> Minion: https://metacpan.org/pod/Minion >> >> On 14/11/16 14:41, Alex Povolotsky wrote: >> >> Looks like I was unclear. I do not need only extract all links, I must >> continue fetching and parsing them, not more than N at a time and if >> possible not less. I need some king of mirroring crawler. >> >> On Sat, Nov 12, 2016 at 6:25 PM, Scott Wiersdorf <[email protected] >> <javascript:>> wrote: >> >>> Here is a complete working example you can run: >>> >>> #!/usr/bin/env perl >>> use Mojolicious::Lite; >>> >>> get '/random-urls' => sub { >>> my $c = shift; >>> $c->render_later; >>> >>> $c->delay( >>> sub { ## first step >>> my $delay = shift; >>> >>> $c->ua->get('https://www.random.org/bytes/', >>> $delay->begin); >>> $c->ua->get('https://www.random.org/integer-sets/', >>> $delay->begin); >>> $c->ua->get('https://www.random.org/strings/', >>> $delay->begin); >>> $c->ua->get('https://www.random.org/audio-noise/', >>> $delay->begin); >>> }, >>> >>> sub { ## second step >>> my $delay = shift; >>> >>> for my $dom (map { $_->res->dom } @_) { >>> say STDERR $dom->at('title')->text; >>> } >>> >>> $c->render(text => "Got all the links"); >>> } >>> ); >>> }; >>> >>> app->start; >>> >>> >>> When the results finally all come back from the first step, the second >>> step will print out the page titles to STDERR, and return to the client >>> "Got all the links". >>> >>> Scott >>> >>> On Saturday, November 12, 2016 at 2:47:57 AM UTC-7, Alex Povolotsky >>> wrote: >>>> >>>> Hello >>>> >>>> Examples of Mojo::UserAgent are limited to fetching a set of files. >>>> >>>> But I need a bit more complex thing: I need to parse a site, reading >>>> pages, parsing it and reading links, using non-blocking UA with, say, 4 >>>> downloads at a time, no more and if possible no less. >>>> >>>> Can someone give me a good example? >>>> >>>> Alex >>>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "Mojolicious" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/mojolicious/wdp_pgd4e0k/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected] <javascript:>. >>> To post to this group, send email to [email protected] >>> <javascript:>. >>> Visit this group at https://groups.google.com/group/mojolicious. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Mojolicious" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected] >> <javascript:>. >> Visit this group at https://groups.google.com/group/mojolicious. >> For more options, visit https://groups.google.com/d/optout. >> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Mojolicious" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/mojolicious/wdp_pgd4e0k/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected] <javascript:>. >> To post to this group, send email to [email protected] >> <javascript:>. >> Visit this group at https://groups.google.com/group/mojolicious. >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "Mojolicious" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/mojolicious. For more options, visit https://groups.google.com/d/optout.
