Re: [ZODB-Dev] Temporary files not closed
Izak Burger wrote: Before I dive into the code and attempt to find the problem, I thought I'd post first, partly to report the problem, but at the same time to find out if anyone has seen this and if perhaps a fix is already available. Never experienced this myself, what storage(s) are you using? Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Temporary files not closed
Jim Fulton wrote: I haven't seen this and, looking at the code on the trunk, I see no reason why many savepoint temporary storages or ZEO transaction buffers would be kept open. Savepoint temporary storages are closed after use I wonder if Izak inadvertently has lots of savepoints lying around? What happens to a savepoint if the transaction it's associated with goes away? (by abort, for example...) Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Temporary files not closed
On Thu, Jun 25, 2009 at 4:46 AM, Izak Burgeri...@upfrontsystems.co.za wrote: Hi all, This is a zope issue that I think is probably zodb related. I've been seeing this behaviour for a couple of years now. When I inspect /proc/PID/fd (on a linux machine, this lists the open file descriptors held by the process with process id PID), I see many descriptors that point to open files, eg: $ ls -go /proc/15431/fd | grep deleted lrwx-- 1 64 2009-06-25 10:09 13 - /var/tmp/tmptBEWCx.tbuf (deleted) lrwx-- 1 64 2009-06-25 10:09 14 - /var/tmp/tmpKSuzMN (deleted) lrwx-- 1 64 2009-06-25 10:09 6 - /var/tmp/tmp2HUAnn.tbuf (deleted) lrwx-- 1 64 2009-06-25 10:09 7 - /var/tmp/tmpex5QmJ (deleted) Have you looked at the contents of the files? Once you have an idea of what is being written you might be able to determine what is doing the writing. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Temporary files not closed
Benji York wrote: Have you looked at the contents of the files? Once you have an idea of what is being written you might be able to determine what is doing the writing. The big 6.3GB one contains pickles. Lots and lots of pickles. I suspect we're simply dealing with a process that creates a LOT of savepoints before finally committing. The developer of the application is out at the moment, when he gets back from his lunch-break/touch-rugby game I shall ask him about this. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Temporary files not closed
Izak Burger wrote: I suspect we're simply dealing with a process that creates a LOT of savepoints before finally committing. The developer of the application is out at the moment, when he gets back from his lunch-break/touch-rugby game I shall ask him about this. The developer confirmed that the process imports a lot of data, and commits every now and then. By comitting more often we can keep these files smaller. I've been watching it for a while now, and I see it grows to about 7.3 GB and then it starts over again. It seems everything works as it should then, the important thing is simply to have enough space on /tmp. Thanks for the pointers. regards, Izak ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Temporary files not closed
On Thu, Jun 25, 2009 at 7:19 AM, Izak Burgeri...@upfrontsystems.co.za wrote: Benji York wrote: Have you looked at the contents of the files? Once you have an idea of what is being written you might be able to determine what is doing the writing. The big 6.3GB one contains pickles. Lots and lots of pickles. I suspect we're simply dealing with a process that creates a LOT of savepoints before finally committing. The developer of the application is out at the moment, when he gets back from his lunch-break/touch-rugby game I shall ask him about this. We have been seeing something similar on Windows. Where there are lots of zeo-cache files + .tbufs lying around. This does not happen all the time. I was presuming it had something to do with signal handling/windows/tempfile. We have not found a way to reproduce the problem but have seen it at several customer environments. -alan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Temporary files not closed
On Jun 25, 2009, at 6:56 AM, Chris Withers wrote: Jim Fulton wrote: I haven't seen this and, looking at the code on the trunk, I see no reason why many savepoint temporary storages or ZEO transaction buffers would be kept open. Savepoint temporary storages are closed after use I wonder if Izak inadvertently has lots of savepoints lying around? What happens to a savepoint if the transaction it's associated with goes away? (by abort, for example...) The savepoint storage and temporary file is closed. Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
