You can take a look at Yahoo Query Language (YQL) for fetching such data in a structured format, as opposed to crawling everything and then looking for structure.
On Sep 12, 3:45 am, Richard <[email protected]> wrote: > Yes definitely - have a look for articles about web scraping. I do > that kind of stuff with Python full time. > > Probably you would make a command line script to crawl and scrape that > data into a structured form and then insert it into your web2py app > database. > > On Sep 12, 7:09 am, bally boy <[email protected]> wrote: > > > > > I was just contemplating on this idea to develop some sort of app which > > could crawl through the various social networking sites, look at the > > comments ppl have posted on various things rangin from music to songs to > > products to movies , and then categorize them into individual sections and > > grade each product based on certain criterias etc. > > For eg: there might be many ppl who might have said something on a certain > > movie ' X' or a certain gadget 'Y', on sites like twitter or facebook.Could > > a tool be made to crawl across and fish those comments or tweets and try to > > make something intelligent out of that. I am sure there should be a lot of > > intelligence hidden up there in those social networking sites which can be > > utilized in making something comprehensible. > > > Would love it if someone could please give a thought to it, and yeah would > > such a application be feasible using web2py.(although I have not yet even > > laid out the path ahead)
