Hi Yan! Thanks for joining the scrapy users list, and it's lovely to see someone interested in helping the project out.
Let me try to answer your questions one by one: On Sat, Jan 3, 2015 at 12:24 AM, Yan Yi <[email protected]> wrote: > > 1. How do I run scrapy from source after forking the project? > The "INSTALL" text file simply points me to the online documentation for > standard installation. Makefile.buildbot has a section saying python > extras/makedeb.py build. Tried to run makedeb.py build but I got an error > saying no module named scrapy. Not sure why the scrapy source has scrapy as > a dependency. > > Make a virtualenv, and do "pip install -e .", is my usual strategy. I made a ASCII art screencast here of me doing that, but it is accidentally super long. You can watch me try a bunch of things and fail a lot, which will probably be instructive. Here's the link: https://asciinema.org/a/15161 That took me way more work than I expected! I think probably some docs may have become out of date. Some things to note: * I created the virtualenv in "." (the current directory) and therefore invoke pip from the ./bin/ directory. * https://oh-bugimporters.readthedocs.org/en/latest/intro.html is a separate project I work on that depends on Scrapy, so its development environment setup tips might prove useful to you, too. * Sometimes I seem to do nothing for 10 seconds at a time or longer; this is because I'm installing things, in a different terminal. Please feel free to use the timing bar at the bottom to skip past the parts where I'm doing nothing. > 2. I could not find process_spider_exception() method in > master/scrapy/middleware.py. > Am I looking in the wrong place? > My usual way to find code like this is to use the GitHub web search, or this command line tool: $ git grep "def process_spider_exception" | cat scrapy/contrib/spidermiddleware/httperror.py: def process_spider_exception(self, response, exception, spider): scrapy/core/spidermw.py: def process_spider_exception(_failure): So one of those seems to be the answer! > > 3. How do I begin fixing this bug? A few tips on the direction I'd need to > take would be great. > I haven't contributed to Scrapy, but from my experience with similar projects, the answer is usually: * Write a test case that indicates that the bug is real -- so the test should *fail* when you run it * Hack up the code until it passes I realize you probably wanted more help than that! I don't have a huge amount of experience with the Scrapy codebase, so that's the help I can provide. Make sure to read through http://doc.scrapy.org/en/latest/contributing.html and other docs in the doc.scrapy.org site. Cheers, and welcome aboard! -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
