More questions: * Why can embedding a webpage in IPython notebook be insecure? How can it be fixed? * What would be the architecture? How many processes or threads will you use, and what will they be used for? How will they communicate?
воскресенье, 16 марта 2014 г., 20:13:59 UTC+6 пользователь Mikhail Korobov написал: > > Hi Vatsal, > > In the proposal we're looking for 2 things: > > 1) a demonstrated ability to contribute to Python open-source projects; > 2) understanding of the project you want to take. > > "Scrapy IDE" is listed as "Advanced" project partially because (2) is not > easy. This project has a lot of freedom in what one can do. That's why the > challenge is to decide what exactly you want to achieve and convince other > people that (a) the ideas are good and (b) you can deliver. > > Your idea that "sc.initialize()" will create an UI in an IPython notebook, > and that UI will use IPython 2.0 widgets makes sense; I think it is a good > start. But a proposal should be much more detailed. Some initial points to > think about: > > * What will be this "Scrapy IDE" for? How will it help developing spiders? > What would be the "new" workflow? What's wrong with developing spiders > using existing tools (write code using text editor, run spider from command > line, maybe insert a call to debugger to inspect the response)? > * What will be possible to do using this UI? > * What would this UI look like? > * Will it be possible to run existing spiders using IPython notebook? If > so, how can this be acheived? If not, why it is not feasible? > * Will it be possible to write spiders in IPython cells and develop them > interactively? If so, how can this be acheived? If not, why it is not > feasible? > * What about Scrapy settings? How are IPython notebooks related to Scrapy > projects? > * My demo allows to execute XPath selectors over pages fetched using > Scrapy downloader and show the results visually (by highlighting matched > text). What other helpers can we provide? > * Can creating IPython extension reduce the boilerplate (your example has > 3 lines, maybe we can bring it down to 1)? > > пятница, 14 марта 2014 г., 15:16:17 UTC+6 пользователь Vatsal Juneja > написал: >> >> Hi sir, >> >> Since the time is very limited, I've started working on my proposal as >> well. I'd be very glad if you can also tell me what all do you expect to >> see in the application proposal. >> >> On Thursday, 13 March 2014 00:28:19 UTC+5:30, Mikhail Korobov wrote: >>> >>> Hi Vatsal, >>> >>> Nice to meet you! >>> >>> I don't think implementing *both* projects in 3 months is a good idea. >>> It is better to have a solid proposal for one of these projects. >>> >>> Making a prior contribution is a good idea - in fact, we were told it is >>> a requirement (see >>> https://wiki.python.org/moin/SummerOfCode/ApplicationTemplate2014). So >>> you're in a right direction. >>> >>> As a mentor of "Scrapy IDE" project I'll comment on it. To complete this >>> project you will need to develop an understanding of Scrapy internals, >>> IPython internals, JS/CSS/HTML, browser security and UI design. You said >>> you've used Scrapy for personal projects (that's great!); are you familiar >>> with IPython and its "notebook" feature? Try to come up with ideas about >>> how to use them together. Think about how did you debug and develop your >>> spiders, and what visual tools may have helped you. Maybe ipython-scrapy >>> demo (linked in wiki - http://nbviewer.ipython.org/gist/kmike/9001574) >>> would be helpful - note that it is just a quick hack, and the >>> implementation should be changed. There are many possible options and ideas >>> to implement; proposal should include a plan - what exactly do you want to >>> implement? Architecture overview and maybe some UI mockups would be nice. >>> >>> среда, 12 марта 2014 г., 9:25:48 UTC+6 пользователь Vatsal Juneja >>> написал: >>>> >>>> Hi, >>>> >>>> Looks like there is a pull request already pending for the web UI >>>> project. I'd be glad if anyone can tell me if it's still open to work on. >>>> I >>>> have some enhancements that I'd like to propose. >>>> >>>> On Tuesday, 11 March 2014 23:35:47 UTC+5:30, Vatsal Juneja wrote: >>>>> >>>>> Hello! >>>>> >>>>> I'm a pre-final year Mathematics and Computing student from Indian >>>>> Institute of Technology(BHU), Varanasi. >>>>> >>>>> There are two projects that have caught my attention: *IPython IDE >>>>> for Scrapy* and *Scrapyd web UI* and I'd like to know if I can take >>>>> both the projects over the 12 weeks of GSoC. I've been working on HTML, >>>>> JS >>>>> and Python stack for around 4 years now and would love to contribute to >>>>> scrapy as it has been quite helpful in some of my personal projects as >>>>> well. >>>>> >>>>> Please feel free to ask any questions. >>>>> >>>>> Hoping to get a reply soon. >>>>> >>>>> Thanks and Regards, >>>>> >>>>> Vatsal >>>>> >>>> -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
