Sir, My exams just finished yesterday so I can finally get back to work on scrapy. I have submitted my GSoC proposal. I know I'm late but I will surely cover the lost time. I have created a blog where I will be posting my work with scrapy (http://ahhda.blogspot.in/). The proposal however requires the link of a contribution which I don't have as I was busy with my college. Although I have contributed to sympy (https://github.com/sympy/sympy/pull/9121). I have given this link in the proposal. I hope this is acceptable.
I have also created a copy at (https://docs.google.com/document/d/1FUg1fhdIWS5HLh8zjbPTpR6kXwsG60pRdJ6QsF4m3u0/edit). Do tell me if you find something missing or wrong with in the proposal. The results will be announced on 27th April. Till then I will continue to work on scrapy and fix some bugs. Looking towards a great summer :) Regards, Anuj On Thursday, March 19, 2015 at 1:13:46 AM UTC+5:30, Mikhail Korobov wrote: > > Hi, > > среда, 18 марта 2015 г., 23:52:19 UTC+5 пользователь Anuj Bansal написал: >> >> Sir, >> >> I have learned the differences between Python 2 and Python 3. I have >> created a google doc ( >> https://docs.google.com/document/d/1xf7OtuyB5b6npCOLalZ-yjPZEcoKNb19iimfElyDino/edit) >> >> in which I have written the common porting errors which I could find after >> going through various blogs and projects and there corresponding syntax >> corrections. You can add your valuable suggestions or anything that I have >> missed out to it by directly going to the link and editing it. Do tell me >> if you find something wrong with the approach. >> >> >>> The recommended way is to use "six" Python module. Some parts of Scrapy >>> are already ported to Python 3 - see e.g. >>> https://travis-ci.org/scrapy/scrapy/jobs/54761340 - 235 tests pass in >>> Python 3.3. To get started try cloning Scrapy and running some tests using >>> tox (as described in docs). >>> >> >> I got some errors while setting up scrapy and found out that I had to >> install libssl-dev, libffi-dev, python-dev and libxml2-dev. As mentioned on >> ( >> http://stackoverflow.com/questions/17611324/error-when-installing-scrapy-on-ubuntu-13-04 >> ). >> Shouldn't these be added to the scrapy requirements ? Should I create an >> issue relating to this ? I'm currently working on Ubuntu 14.04. >> > > Scrapy requirements.txt lists Python packages (not system packages). There > are some install notes here: > http://doc.scrapy.org/en/latest/intro/install.html > libffi-dev is a dependency of PyOpenSSL; libxml2-dev is a dependency of > lxml. I'm not sure - maybe we can document this all. It would be > documenting the requirements of our requirements though. > > >> >> >>> You can also check >>> https://github.com/scrapy/scrapy/blob/master/tests/py3-ignores.txt file >>> - try uncommenting something and run tests again to see what's not ported. >>> We can't rely only on tests when porting, but they are a good start. >>> >> >> This is great ! Would really help me in planning my strategy. >> >> >>> This URL encoding thing is where we stopped. Without having a solid >>> solution we can't port scrapy.Request, and without scrapy.Request most >>> other Scrapy components don't work. >>> >> >> Handling binary data is the most trickiest issue that people face in >> supporting Python 2 and Python 3. So the first thing to do would be to find >> the best solution for URL encoding. Only then we would be able to port >> other scrapy components. >> So I should first take a look at the w3lib project. >> >> As quoted in the book ( >> http://python3porting.com/strategies.html#python-2-and-python-3-without-conversion >> ): >> >> "My recommendation for the development workflow if you want to support >> Python 3 without using 2to3 is to run 2to3 on the code once and then fix >> it up until it works on Python 3. Only then introduce Python 2 support into >> the Python 3 code, using six where needed. Add support for Python 2.7 >> first, and then Python 2.6. Doing it this way can sometimes result in a >> very quick and painless process." >> >> Is this the recommended method ? >> > > Usually I just start with the existing code and add Python 3 support to it > using "six" package and a common sense :) The metod from the book sounds > OK, but you need to be very careful not to break existing Python 2.x code. > __future__ imports can be also helpful (2to3 doesn't add them). We don't > need Python 2.6 support. > > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
