Re: Standalone script with django's ORM and multiprocessing
prabhu S pisze: > Solution appears like a hack to me. Why do you close the connection in > every process? Can you not just close once in parent? Execute commits > alone in each process. > I've checked that. Unfortunately, in my case, it won't work. I have a loop looking like this: for job in joblist: p = Process(target=self.run_job) p.start() When i close connection before this loop i get errors like at the beggining. When i close connection just before creating a new process i get: 'lost connection while performing a query' (or something like that :)). Finally when i close connection just right after this loop it starts to behave weirdly. One time it works fine, but next time i launch it i get mysql has gone away errors. I suppose this happens because sometimes subprocesses fire up queries after parent finishes this loop, but sometimes they're faster and still use parent's connection. Closing connections in subprocesses might look a bit hacky. I could try to somehow reinitialize db connection, but simply closing it, and leaving the rest to django is a lot easier :) Thank you for your replies. I appreciate it. Regards Spajderix --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Standalone script with django's ORM and multiprocessing
On Tue, 2009-08-11 at 03:12 -0700, prabhu S wrote: [...] > Why do you close the connection in > every process? Can you not just close once in parent? That would also work. Regards, Malcolm --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Standalone script with django's ORM and multiprocessing
Solution appears like a hack to me. Why do you close the connection in every process? Can you not just close once in parent? Execute commits alone in each process. On Aug 11, 10:50 am, Spajderix wrote: > Malcolm Tredinnick pisze: > > > > > On Tue, 2009-08-11 at 11:06 +0200, Spajderix wrote: > > >> Hi! > > >> I've written a standalone script, which looks throught a table in db for > >> tasks to perform. It then starts subprocesses for each task it founds > >> using multiprocessing.Process class from python. Everything's fine when > >> there is one task, but when i try to start more subprocesses at once i get: > > >> OperationalError: (2013, 'Lost connection to MySQL server during query') > >> raise errorclass, errorvalue > >> OperationalError: (2006, 'MySQL server has gone away') > > >> I guess that this happens because all subprocesses share one connection, > >> and when one of them closes this connection, rest of subprocesses raises > >> an error. > > > That certainly sounds believable. We call close() explicitly, too, so > > that is why ongoing operations are interrupted in the middle. > > >> Do you know a way to go round this problem? > > > If you close the database connection, Django will open a new one the > > next time it needs it. So I suspect you can work around this by > > explicitly closing the connection immediately after you start a new > > process (in the new process). Then the process will get its own > > connection when you try to do something. > > > Regards, > > Malcolm > > Thank you! That solved the problem:) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Standalone script with django's ORM and multiprocessing
Malcolm Tredinnick pisze: > On Tue, 2009-08-11 at 11:06 +0200, Spajderix wrote: > >> Hi! >> >> I've written a standalone script, which looks throught a table in db for >> tasks to perform. It then starts subprocesses for each task it founds >> using multiprocessing.Process class from python. Everything's fine when >> there is one task, but when i try to start more subprocesses at once i get: >> >> OperationalError: (2013, 'Lost connection to MySQL server during query') >> raise errorclass, errorvalue >> OperationalError: (2006, 'MySQL server has gone away') >> >> I guess that this happens because all subprocesses share one connection, >> and when one of them closes this connection, rest of subprocesses raises >> an error. >> > > That certainly sounds believable. We call close() explicitly, too, so > that is why ongoing operations are interrupted in the middle. > > >> Do you know a way to go round this problem? >> > > If you close the database connection, Django will open a new one the > next time it needs it. So I suspect you can work around this by > explicitly closing the connection immediately after you start a new > process (in the new process). Then the process will get its own > connection when you try to do something. > > Regards, > Malcolm > > Thank you! That solved the problem:) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Standalone script with django's ORM and multiprocessing
On Tue, 2009-08-11 at 11:06 +0200, Spajderix wrote: > Hi! > > I've written a standalone script, which looks throught a table in db for > tasks to perform. It then starts subprocesses for each task it founds > using multiprocessing.Process class from python. Everything's fine when > there is one task, but when i try to start more subprocesses at once i get: > > OperationalError: (2013, 'Lost connection to MySQL server during query') > raise errorclass, errorvalue > OperationalError: (2006, 'MySQL server has gone away') > > I guess that this happens because all subprocesses share one connection, > and when one of them closes this connection, rest of subprocesses raises > an error. That certainly sounds believable. We call close() explicitly, too, so that is why ongoing operations are interrupted in the middle. > Do you know a way to go round this problem? If you close the database connection, Django will open a new one the next time it needs it. So I suspect you can work around this by explicitly closing the connection immediately after you start a new process (in the new process). Then the process will get its own connection when you try to do something. Regards, Malcolm --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Standalone script with django's ORM and multiprocessing
Hi! I've written a standalone script, which looks throught a table in db for tasks to perform. It then starts subprocesses for each task it founds using multiprocessing.Process class from python. Everything's fine when there is one task, but when i try to start more subprocesses at once i get: OperationalError: (2013, 'Lost connection to MySQL server during query') raise errorclass, errorvalue OperationalError: (2006, 'MySQL server has gone away') I guess that this happens because all subprocesses share one connection, and when one of them closes this connection, rest of subprocesses raises an error. Do you know a way to go round this problem? Regards Spajderix --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
