I think the discoverability is pretty low. I would suggest adding a section, as a start, to the Scrapy docs, called "Things that use Scrapy where source code is available and it's kind of a cool bit of code so you might want to check it out" and then have a list of links with a brief description of each.
Presumably you can come up with a better title than I proposed. On Fri, Jan 30, 2015 at 1:11 AM, Travis Leleu <[email protected]> wrote: > Asheesh, that looks awesome! Thank you for sharing. > > How do other people feel about the availability/discoverability of 3rd > party addons to scrapy? I suspect there is a lot of neat things like > Asheesh's project, but I don't know where to find them (other than github > search, but that's more effort :P ). > > On Wed, Jan 28, 2015 at 11:33 PM, Asheesh Laroia <[email protected]> > wrote: > >> Hi all, >> >> In https://github.com/openhatch/oh-bugimporters (see also docs at >> https://oh-bugimporters.readthedocs.org/en/latest/ ) we do something >> similar with Scrapy. Feel free to use our example as inspiration on how to >> do it, or how *not* to do it! >> >> I'm excited that you're interested in doing something like this. I hope >> other framework maintainers might be able to chime in. I'm "just" another >> user of Scrapy like yourself. >> >> On Wed, Jan 28, 2015 at 11:00 PM, user12345 <[email protected]> >> wrote: >> >>> Hi, >>> >>> Thanks for the interest. I'm looking for specifically for replies from >>> project maintainers/contributors to see whether or this is feasible in the >>> first place - don't want to waste anyone's time, including mine. If it is, >>> and I can do it on my own, I'd love to tackle it with some design guidance, >>> to make sure I'm working with the framework instead of against it. >>> >>> >>> On Wednesday, January 28, 2015 at 6:15:16 PM UTC-8, user12345 wrote: >>>> >>>> I'm working on a scrapy project where a "rabbit client" and "crawl >>>> worker" work together to consume scrape requests from a queue. These >>>> requests have more configuration than a start_url - it could be something >>>> like url and a set of xpaths, or a domain-specific configuration, like >>>> site-specific product ID (from which we programmatically build the url) and >>>> optional identifiers like color, style, and size to further specify the >>>> item one wants to scrape. >>>> >>>> I'm wondering if it would be desirable to have built-in support for >>>> more specific "crawl configurations" like this within the framework? If >>>> that's the case, I'd be more than happy to have a design discussion and >>>> hash out the details. >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "scrapy-users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/scrapy-users. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "scrapy-users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/scrapy-users. >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to the Google Groups > "scrapy-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/scrapy-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
