Re: [ZODB-Dev] Temporary files not closed
On Jun 25, 2009, at 6:56 AM, Chris Withers wrote: > Jim Fulton wrote: >> I haven't seen this and, looking at the code on the trunk, I see >> no reason why many savepoint temporary storages or ZEO >> transaction buffers would be kept open. Savepoint temporary >> storages are closed after use > > I wonder if Izak inadvertently has lots of savepoints lying around? > > What happens to a savepoint if the transaction it's associated with > goes away? (by abort, for example...) The savepoint storage and temporary file is closed. Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Temporary files not closed
On Thu, Jun 25, 2009 at 7:19 AM, Izak Burger wrote: > Benji York wrote: >> Have you looked at the contents of the files? Once you have an idea of >> what is being written you might be able to determine what is doing the >> writing. > > The big 6.3GB one contains pickles. Lots and lots of pickles. > > I suspect we're simply dealing with a process that creates a LOT of > savepoints before finally committing. The developer of the application > is out at the moment, when he gets back from his lunch-break/touch-rugby > game I shall ask him about this. We have been seeing something similar on Windows. Where there are lots of zeo-cache files + .tbufs lying around. This does not happen all the time. I was presuming it had something to do with signal handling/windows/tempfile. We have not found a way to reproduce the problem but have seen it at several customer environments. -alan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Temporary files not closed
Izak Burger wrote: > I suspect we're simply dealing with a process that creates a LOT of > savepoints before finally committing. The developer of the application > is out at the moment, when he gets back from his lunch-break/touch-rugby > game I shall ask him about this. The developer confirmed that the process imports a lot of data, and commits every now and then. By comitting more often we can keep these files smaller. I've been watching it for a while now, and I see it grows to about 7.3 GB and then it starts over again. It seems everything works as it should then, the important thing is simply to have enough space on /tmp. Thanks for the pointers. regards, Izak ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Temporary files not closed
Benji York wrote: > Have you looked at the contents of the files? Once you have an idea of > what is being written you might be able to determine what is doing the > writing. The big 6.3GB one contains pickles. Lots and lots of pickles. I suspect we're simply dealing with a process that creates a LOT of savepoints before finally committing. The developer of the application is out at the moment, when he gets back from his lunch-break/touch-rugby game I shall ask him about this. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Temporary files not closed
On Thu, Jun 25, 2009 at 4:46 AM, Izak Burger wrote: > Hi all, > > This is a zope issue that I think is probably zodb related. I've been > seeing this behaviour for a couple of years now. When I inspect > /proc/PID/fd (on a linux machine, this lists the open file descriptors > held by the process with process id PID), I see many descriptors that > point to open files, eg: > > $ ls -go /proc/15431/fd | grep deleted > lrwx-- 1 64 2009-06-25 10:09 13 -> /var/tmp/tmptBEWCx.tbuf (deleted) > lrwx-- 1 64 2009-06-25 10:09 14 -> /var/tmp/tmpKSuzMN (deleted) > lrwx-- 1 64 2009-06-25 10:09 6 -> /var/tmp/tmp2HUAnn.tbuf (deleted) > lrwx-- 1 64 2009-06-25 10:09 7 -> /var/tmp/tmpex5QmJ (deleted) Have you looked at the contents of the files? Once you have an idea of what is being written you might be able to determine what is doing the writing. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Temporary files not closed
Chris Withers wrote: > I wonder if Izak inadvertently has lots of savepoints lying around? I looked at the code and I have to agree that I cannot see anything obviously wrong there. I just checked a number of zope processes and I see they all have 4 or 5 of these "deleted" files open. These files were created using tempfile.TemporaryFile (which on posix system unlinks the file directly after opening it). If this was caused by a missing close() call, I would expect the number of open files to grow. This does not happen. These checkpoint files do grow though, and eventually they fill up the filesystem. Jim's comment about the file being kept open for the life of the client makes sense. Tell me, does the size of the file ever decrease, or does it continue to grow? For example: $ ls -lHh /proc/23770/fd/21 -rw--- 0 zope zope 6.3G 2009-06-25 13:58 /proc/23770/fd/21 I very seriously doubt we have a 6.3GB sized transaction anywhere. regards, Izak ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Temporary files not closed
Jim Fulton wrote: > I haven't seen this and, looking at the code on the trunk, I see no > reason why many savepoint temporary storages or ZEO transaction > buffers would be kept open. Savepoint temporary storages are closed > after use I wonder if Izak inadvertently has lots of savepoints lying around? What happens to a savepoint if the transaction it's associated with goes away? (by abort, for example...) Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Temporary files not closed
On Jun 25, 2009, at 4:46 AM, Izak Burger wrote: > Hi all, > > This is a zope issue that I think is probably zodb related. I've been > seeing this behaviour for a couple of years now. When I inspect > /proc/PID/fd (on a linux machine, this lists the open file descriptors > held by the process with process id PID), I see many descriptors that > point to open files, eg: > > $ ls -go /proc/15431/fd | grep deleted > lrwx-- 1 64 2009-06-25 10:09 13 -> /var/tmp/tmptBEWCx.tbuf > (deleted) > lrwx-- 1 64 2009-06-25 10:09 14 -> /var/tmp/tmpKSuzMN (deleted) > lrwx-- 1 64 2009-06-25 10:09 6 -> /var/tmp/tmp2HUAnn.tbuf > (deleted) > lrwx-- 1 64 2009-06-25 10:09 7 -> /var/tmp/tmpex5QmJ (deleted) > > This gets worse over time until we run out of diskspace, with a trace > back like this (somewhat sanitised to protect the guilty): > > ".../parts/zope2/lib/python/transaction/_manager.py",line 110, in > savepoint return self.get().savepoint(optimistic) File > ".../parts/zope2/lib/python/transaction/_transaction.py", line 312, in > savepoint self._saveAndRaiseCommitishError() # reraises! > ".../parts/zope2/lib/python/transaction/_transaction.py", line 422, in > _saveAndRaiseCommitishError t, v, tb = > self._saveAndGetCommitishError() File > ".../parts/zope2/lib/python/transaction/_transaction.py", line 309, in > savepoint savepoint = Savepoint(self, optimistic, *self._resources) > File > ".../parts/zope2/lib/python/transaction/_transaction.py", line 737, in > __init__ savepoint = savepoint() > ".../parts/zope2/lib/python/ZODB/Connection.py", line 1046, in > savepoint > self._commit(None) > ".../parts/zope2/lib/python/ZODB/Connection.py",line 555, in _commit > self._store_objects(ObjectWriter(obj), transaction) > ".../parts/zope2/lib/python/ZODB/Connection.py",line 583, in > _store_objects s = self._storage.store(oid, serial, p, self._version, > transaction) > ".../parts/zope2/lib/python/ZODB/Connection.py", line 1180, in store > self._file.write(data) IOError: [Errno 28] No space left on device > > Naturally a restart solves the problem immediately, but this is > still a bug. > > Before I dive into the code and attempt to find the problem, I thought > I'd post first, partly to report the problem, but at the same time to > find out if anyone has seen this and if perhaps a fix is already > available. I haven't seen this and, looking at the code on the trunk, I see no reason why many savepoint temporary storages or ZEO transaction buffers would be kept open. Savepoint temporary storages are closed after use and a single transaction buffer if created per ZEO client. It is kept optn for the life of the client. Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Temporary files not closed
Chris Withers wrote: > Never experienced this myself, what storage(s) are you using? We have a Zeo server with two zope clients, with a single FileStorage based store. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Temporary files not closed
Izak Burger wrote: > Before I dive into the code and attempt to find the problem, I thought > I'd post first, partly to report the problem, but at the same time to > find out if anyone has seen this and if perhaps a fix is already available. Never experienced this myself, what storage(s) are you using? Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Temporary files not closed
Hi all, This is a zope issue that I think is probably zodb related. I've been seeing this behaviour for a couple of years now. When I inspect /proc/PID/fd (on a linux machine, this lists the open file descriptors held by the process with process id PID), I see many descriptors that point to open files, eg: $ ls -go /proc/15431/fd | grep deleted lrwx-- 1 64 2009-06-25 10:09 13 -> /var/tmp/tmptBEWCx.tbuf (deleted) lrwx-- 1 64 2009-06-25 10:09 14 -> /var/tmp/tmpKSuzMN (deleted) lrwx-- 1 64 2009-06-25 10:09 6 -> /var/tmp/tmp2HUAnn.tbuf (deleted) lrwx-- 1 64 2009-06-25 10:09 7 -> /var/tmp/tmpex5QmJ (deleted) This gets worse over time until we run out of diskspace, with a trace back like this (somewhat sanitised to protect the guilty): ".../parts/zope2/lib/python/transaction/_manager.py",line 110, in savepoint return self.get().savepoint(optimistic) File ".../parts/zope2/lib/python/transaction/_transaction.py", line 312, in savepoint self._saveAndRaiseCommitishError() # reraises! ".../parts/zope2/lib/python/transaction/_transaction.py", line 422, in _saveAndRaiseCommitishError t, v, tb = self._saveAndGetCommitishError() File ".../parts/zope2/lib/python/transaction/_transaction.py", line 309, in savepoint savepoint = Savepoint(self, optimistic, *self._resources) File ".../parts/zope2/lib/python/transaction/_transaction.py", line 737, in __init__ savepoint = savepoint() ".../parts/zope2/lib/python/ZODB/Connection.py", line 1046, in savepoint self._commit(None) ".../parts/zope2/lib/python/ZODB/Connection.py",line 555, in _commit self._store_objects(ObjectWriter(obj), transaction) ".../parts/zope2/lib/python/ZODB/Connection.py",line 583, in _store_objects s = self._storage.store(oid, serial, p, self._version, transaction) ".../parts/zope2/lib/python/ZODB/Connection.py", line 1180, in store self._file.write(data) IOError: [Errno 28] No space left on device Naturally a restart solves the problem immediately, but this is still a bug. Before I dive into the code and attempt to find the problem, I thought I'd post first, partly to report the problem, but at the same time to find out if anyone has seen this and if perhaps a fix is already available. regards, Izak ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
