Ok I patched *sqlite.py* module and SqlitePriorityQueue class and it seems to work fine with Postgre. But i have some questions if smone can answer. 1. SQLite use parameter "check_same_thread=False" for connection. But I think Postgre con is thread-safe so i didnt use smthing like this. 2. Whats is the use of remove and clear methods of SqlitePriorityQueue class? Queue worked fine before patch them as i couldnt find a call for them.
On Monday, December 26, 2016 at 10:45:25 AM UTC+2, Nikolaos-Digenis Karagiannis wrote: > > Hi, > > You probably looked into some old documentation. > This setting was introduced before scrapy separated from scrapyd. > Job queues are now implemented in scrapyd. > A pickaxe search confirms this: > > https://github.com/scrapy/scrapy/commit/75e2c3eb338ea03e487907fa8c99bb12317e9435 > This was a point were many release notes are missing, > notes clarifying what was removed from scrapy, > because the "separation of scrapyd" alone doesn't say much. > > Do you use scrapyd? > Unfortunately, the job queue class is no longer configurable. > It shouldn't be hard however to patch scrapyd, > either to make it configurable or to use your own fork. > > Check out scrapyd's repository https://github.com/scrapy/scrapyd/ > and if you come up with something > don't hesitate to open a PR or an issue with suggestions. > We'll be glad to help, > scrapyd does need its components to become less tightly coupled > and making the job queue configurable can contribute to this. > > > > On Monday, 26 December 2016 01:49:50 UTC+2, k bez wrote: >> >> I have started to implement a custom job queue so i ll move from default >> SQLite to Postgre. >> I use this setting SPIDER_QUEUE_CLASS = 'mysite.scraper.PostgreSQLQueue' >> but scrapyd seems to ignore it and create/use default SQLite DB. >> I want to ask if SPIDER_QUEUE_CLASS dont work anymore? >> Thanks in advance. >> > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users+unsubscr...@googlegroups.com. To post to this group, send email to scrapy-users@googlegroups.com. Visit this group at https://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
