Hi sir, I've started working on my own code to make something similar to the proof-of-concept demo that was provided. However, it'll be a bit more basic than that, I'll not be working with threading to keep it simple. Meanwhile I'm drafting an application. I would try to explain what I have in my mind about the project and API, so that we are on the same page.
something like.. import Scraper sc = Scraper() sc.initialize() ========== | | | | ========== .. It will create a UI on the output cell of the notebook, which would have many options in the widgets(IPython 2.0). We can then bind the response from crawlers to some python objects, which can be used for later processing. I would love to hear what you think of this. On Thursday, 13 March 2014 00:28:19 UTC+5:30, Mikhail Korobov wrote: > > Hi Vatsal, > > Nice to meet you! > > I don't think implementing *both* projects in 3 months is a good idea. It > is better to have a solid proposal for one of these projects. > > Making a prior contribution is a good idea - in fact, we were told it is a > requirement (see > https://wiki.python.org/moin/SummerOfCode/ApplicationTemplate2014). So > you're in a right direction. > > As a mentor of "Scrapy IDE" project I'll comment on it. To complete this > project you will need to develop an understanding of Scrapy internals, > IPython internals, JS/CSS/HTML, browser security and UI design. You said > you've used Scrapy for personal projects (that's great!); are you familiar > with IPython and its "notebook" feature? Try to come up with ideas about > how to use them together. Think about how did you debug and develop your > spiders, and what visual tools may have helped you. Maybe ipython-scrapy > demo (linked in wiki - http://nbviewer.ipython.org/gist/kmike/9001574) > would be helpful - note that it is just a quick hack, and the > implementation should be changed. There are many possible options and ideas > to implement; proposal should include a plan - what exactly do you want to > implement? Architecture overview and maybe some UI mockups would be nice. > > среда, 12 марта 2014 г., 9:25:48 UTC+6 пользователь Vatsal Juneja написал: >> >> Hi, >> >> Looks like there is a pull request already pending for the web UI >> project. I'd be glad if anyone can tell me if it's still open to work on. I >> have some enhancements that I'd like to propose. >> >> On Tuesday, 11 March 2014 23:35:47 UTC+5:30, Vatsal Juneja wrote: >>> >>> Hello! >>> >>> I'm a pre-final year Mathematics and Computing student from Indian >>> Institute of Technology(BHU), Varanasi. >>> >>> There are two projects that have caught my attention: *IPython IDE for >>> Scrapy* and *Scrapyd web UI* and I'd like to know if I can take both >>> the projects over the 12 weeks of GSoC. I've been working on HTML, JS and >>> Python stack for around 4 years now and would love to contribute to scrapy >>> as it has been quite helpful in some of my personal projects as well. >>> >>> Please feel free to ask any questions. >>> >>> Hoping to get a reply soon. >>> >>> Thanks and Regards, >>> >>> Vatsal >>> >> -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
