I have this question on SO, but no answers unfortunately. Figured Id try my luck here.
https://stackoverflow.com/questions/37770678/scrapy-not-parsing-response-in-make-requests-from-url-loop I'm trying to get scrapy to grab a URL from a message queue, and then scrape that URL. I have the loop going just fine and grabbing the URL from the queue, but it never enters the parse() method once it has a url, it just continues to loop (and sometimes the url comes back around even though I've deleted it from the queue...) While it's running in terminal, if I CTRL+C and force it to end, it enters the parse() method and crawls the page, then ends. I'm not sure what's wrong here. Scrapy needs to be running at all times to catch a url as it enters the queue. Anyone have ideas or have done something like this? class my_Spider(Spider): name = "my_spider" allowed_domains = ['domain.com'] def __init__(self): super(my_Spider, self).__init__() self.url = None def start_requests(self): while True: # Crawl the url from queue yield self.make_requests_from_url(self._pop_queue()) def _pop_queue(self): # Grab the url from queue return self.queue() def queue(self): url = None while url is None: conf = { "sqs-access-key": "", "sqs-secret-key": "", "sqs-queue-name": "crawler", "sqs-region": "us-east-1", "sqs-path": "sqssend" } # Connect to AWS conn = boto.sqs.connect_to_region( conf.get('sqs-region'), aws_access_key_id=conf.get('sqs-access-key'), aws_secret_access_key=conf.get('sqs-secret-key') ) q = conn.get_queue(conf.get('sqs-queue-name')) message = conn.receive_message(q) # Didn't get a message back, wait. if not message: time.sleep(10) url = None else: url = message if url is not None: message = url[0] message_body = str(message.get_body()) message.delete() self.url = message_body return self.url def parse(self, response): ... yield item -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users+unsubscr...@googlegroups.com. To post to this group, send email to scrapy-users@googlegroups.com. Visit this group at https://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
