I have put all headers but now there is an error in terminal 2015-05-22 14:50:57+0100 [reviewedbase] DEBUG: Retrying <POST http://cerebro-production.herokuapp.com/products_production/_search> (failed 1 times): 503 Service Unavailable2015-05-22 14:50:57+0100 [reviewedbase] DEBUG: Retrying <POST http://cerebro-production.herokuapp.com/products_production/_search> (failed 2 times): [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]2015-05-22 14:51:01+0100 [reviewedbase] INFO: Crawled 1 pages (at 1 pages/min), scraped 0 items (at 0 items/min)2015-05-22 14:51:53+0100 [reviewedbase] DEBUG: Gave up retrying <POST http://cerebro-production.herokuapp.com/products_production/_search> (failed 3 times): 503 Service Unavailable2015-05-22 14:51:53+0100 [reviewedbase] DEBUG: Crawled (503) <POST http://cerebro-production.herokuapp.com/products_production/_search> (referer: http://www.reviewed.com/search/products?sort=rating,desc)2015-05-22 14:51:53+0100 [reviewedbase] DEBUG: Ignoring response <503 http://cerebro-production.herokuapp.com/products_production/_search>: HTTP status code is not handled or not allowed
:/ Em sexta-feira, 22 de maio de 2015 14:13:12 UTC+1, Inês Martins escreveu: > > I am quite new to scrapy. But I have made a few spiders. If you want can > see my doubt in form of question in SO here > http://stackoverflow.com/questions/30396154/scrapy-json-response-body-different-from-json-response-in-website > I am trying to scrapy reviews from this website > http://www.reviewed.com/search/products?sort=rating,desc . Through > firebug I can see that there is a POST request that sends out a json > response with the products. > > What is really confusing me is that my response.body is not the same > response from website, that I see in firebug separator "response". In > scrapy spider, there are only 10 hits and in response from website there > are 20 hits, and really there are 20 products in the firt page (start_url). > What am I doing wrong? Am I passing a wrong Request? > > I have also tested in RESTclient with headers and that but there is also > giving me a json response with only 10 hits... > > Other question, sorry if it is stupid, how can I tell if I need a Request > or a FormRequest? > > Thanks so much for your time and help. > > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
