Hello, I have written this exact use-case example in a gist. I don't typically share it (and haven't put it onto CPAN) because I worry that people might abuse it. While it has parallelism limiting it does not have rate limiting. It doesn't check robots.txt. You need to comply with user agreements. That said, here it is, use it RESPONSIBLY.
https://gist.github.com/jberger/5153008 On Saturday, November 12, 2016 at 3:47:57 AM UTC-6, Alex Povolotsky wrote: > > Hello > > Examples of Mojo::UserAgent are limited to fetching a set of files. > > But I need a bit more complex thing: I need to parse a site, reading > pages, parsing it and reading links, using non-blocking UA with, say, 4 > downloads at a time, no more and if possible no less. > > Can someone give me a good example? > > Alex > -- You received this message because you are subscribed to the Google Groups "Mojolicious" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/mojolicious. For more options, visit https://groups.google.com/d/optout.
